boost intrusive - wonderful!

I wrote a while ago about my disappointment with boost serialize, so it's only fair to redress the balance and say how delighted I am with boost intrusive. The STL list, map and set classes are extremely handy, but to those of us old enough to have cut our teeth on assembler, there's something rather unsatisfactory about all that behind-the-scenes memory allocation going on.

Back in the good old days (like in VMS or RSX), pretty much every data structure began with a pair of forward/backward pointers which could be used to link it into a list. All the key lists in the system (active processes, paged memory allocations, devices, ...) were held like this, with a listhead having the same structure. Insertion and removal are dead easy, and the relatively small size of the systems meant that linear searches weren't really a performance problem.

When I first started writing C++, I wrote a generic linked-list class like this, and an AVL tree class as well. I reused them for a while but then started using STL. But there was always this nagging feeling about those little memory blocks being manipulated behind the curtain.

Recently I've been working on something where memory usage is a problem. I thought those days were behind us, but this one (correlating information about network flows gathered through Netflow) was using up all the memory (8 gbytes) on the machine. Something had to be done. I put all the data structures on a diet, figured out how to get rid of stale information, and so on. But those 32 bytes being wasted for every list entry suddenly seemed important. I'd seen boost intrusive in passing but it had always looked a bit complicated. Time to read it seriously... and discover just how simple it all is.

Those old doubly-linked list structures from RSX... easy:

class my_structure : boost::intrusive::list_base_hook<>
{...

and then:

typedef boost::intrusive::list<my_structure> my_structure_list;

And that's it! You can have sets, too, with lots of nice facilities for lookups, and space/performance tradeoffs, and many other things. For example, I like to keep sets of things indexed by something in the structure. I have a much-reused class for doing this, built on std::set. But lookups require the construction of a whole data structure just to carry the key. I wrote a hack to make this reasonably efficient, but I don't like it.

They thought of that. The advanced lookup and insert functions just let you lookup using the key, without constructing an instance of the structure. The necessary C++ in the structure definition looks a little odd first time, but once you're used to it, you'd never do it any other way.

I'm gradually migrating all my recent code to use boost intrusive. It is just so much better than the STL classes.

Yeah for boost! More thoughts about it here.

The French and American Medical Systems

While I was in Paris the other day I needed to see a doctor. This reminded me why I like the French medical system so much, and why I dislike the US system so much.

A typical visit to the doctor in the US goes something like this.

1. Make an appointment. If you're lucky, you can generally get one the same week.
2. At the due time, drive to medical centre and find a parking space. The medical centres are the size of shopping malls with similar parking problems.
3. Enter the vast, overwhelming medical centre, and try to find the right reception desk, out of hundreds.
4. Give your file number to the receptionist. Don't bother with your name, the computer won't know what to do with it.
5. Wait. Probably 10-20 minutes.
6. Nurse #1 shows up and leads you through a maze of corridors to a tiny room, where she tells you wait.
7. Wait another 20 minutes.
8. Nurse #2 shows up and takes your blood pressure, temperature, height, inside leg measurement and a dozen other things utterly unrelated to the reason for your visit. You also explain your symptoms to the nurse. Heaven forfend that an actual doctor should have to waste his valuable time listening to a patient.
9. She leads you to another room, where you wait another 20 minutes.
10. Doctor shows up. He maybe asks a question or two, gives his diagnosis, and disappears. I don't believe I've ever spent more than 60 seconds in the room with the doctor. His time is just too precious to waste with patients!
    
11. Wait some more.
    
12. Nurse #n shows up with a prescription, if you're going to get one. Though actually my experience is that they will do all they can to avoid giving you any medication.
13. Finally you're done. Nurse #n leads you back through the maze to the reception desk, where you pay $50 or so "co-pay". Your insurer will pay most of the cost, which for a simple consultation like this is typically $200-300.

At the end of all this, you've spent an hour or more, seen a doctor for 30 seconds, and in all probability don't have a correct diagnosis or medication, though you are considerably poorer.

Here's how it typically works in France.

1. Make an appointment, usually the same day. In this particular case, I called, spoke to the doctor himself, who said, "Can you be here in ten minutes?".
   
2. Go to doctor's office, which is typically an apartment in a stylish building (in Paris) or a little rented office with parking just in front of the door, outside Paris. Not a terrifyingly huge medical centre.
   
3. The doctor opens the door for you - not a receptionist or some one from a vast nursing staff, but the actual doctor.
4. You sit in the doctor's office, explain your symptoms, and he asks lots of questions, examines you if necessary. He treats you like an intelligent person rather than as a numbered and rather stupid case file. You spend maybe 10-15 minutes with him.
5. He gives you a prescription - which he writes with his very own fountain pen. You pay, typically $30-50 for a consultation like this.
6. You're done. You've spent 15 minutes with the doctor, and little or no time waiting. You haven't had to deal with an army of nurses. You have a diagnosis and, if appropriate, suitable medication.

The US is a wonderful country, and I appreciate it more and more as I live there. But the medical system is something I'm afraid I'll never get used to or really even understand. What do doctors do when they're not seeing patients, i.e. 99% of the time? The whole system has one primary objective, which is to pour money into the insurance companies. Yet Americans - especially poor ones who are the most disadvantaged by the system - will fight to the death to prevent it changing, as the "Obamacare" debate and rhetoric shows.

Living Life Backwards on the Romance Car Express

Ever since the very first time I visited Japan, nearly three decades ago, I've wanted to travel on the Odakyu Line's Romance Car Express. They have these really cool trains with the driver's cab 747-like on the roof, so passengers in the first car have a panoramic view of the track ahead. And how could you resist the name? I've even contemplated travelling the length of the line just for the sake of it. On a recent trip I finally had a reason. We were travelling to Hakone, for a stay at a traditional Japanese inn (ryokan) and to do some autumn leaf spotting, and the Romance Car is the ideal way to get there, in fact the only direct service from Tokyo.

So we went out to Shinjuku station, supposedly the biggest station in the world if Tokyo hasn't overtaken it by now. Tickets are only available from machines, and while I suppose I could have figured it out in the end, I was very relieved when a helpful member of staff showed up and guided me through the whole thing, including buying the special all-inclusive Hakone Tour ticket. This kind of extraordinary personal service only happens in Japan. I chose our seats in the very end part of the Romance Car, and off we went to catch our train.

All passenger trains in Japan have the locomotive built in, from the Shinkansen down to the humblest one-car rural train. That means they're symmetrical - the front is the same as the back. And in the case of the Romance Car, that means there is an observation car at the back as well as at the front. And guess where our seats were... the booking system doesn't warn you that your view will be of the track receding behind you.

The view of course is exactly the same. But - which I hadn't expected - there's something very surreal about seeing the world zooming away backwards behind you. After a while it begins to feel like your whole life is running backwards. The journey only lasts about 90 minutes. If it lasted for several hours, I'm sure you'd start to get younger, eventually becoming the little boy in short trousers whose Dad used to hold him on the bridge parapet to watch the trains go by.

The most surprising thing is the level (grade) crossings, of which there are a great many. They open within seconds of the train passing - whereas they close much earlier, for safety. So as soon as you pass, the barriers open and - Japan being a crowded place - cars, bicycles and people flood across the road.

On the way back we bought regular tickets for a car in the middle of the train. We weren't sure we could take the Living Life Backwards experience again.

Business Meetings in Japan

If you're used to doing business in the US, business meetings between Japanese companies seem very strange at first. In the US, a meeting generally lasts several hours and is planned well in advance. The discussion is focussed, with an agenda and a deliberate progression towards a conclusion. At the end, though there are generally no formal minutes, there are action items and agreements. You know what you've achieved, whether it was a lot or a little.

When I first started working closely with my Japanese colleagues, I just couldn't understand what was happening. Meetings, even at a high level, can happen at short notice. They're rarely longer than an hour. And at the end, as a Westerner you have little idea what has been discussed, and no idea at all what has been concluded. It's just like a cosy chat amongst old friends. Very often, it's not at a formal conference table, but sitting in armchairs around a low table. For this reason I call this meeting style the "Fireside Chat".

An OL ("office lady", a kind of general administrative assistant, invariably female and fairly young) brings in drinks - hot tea in winter, cold barley tea in summer. The head people from the respective teams start a conversation, others join in occasionally, showing due respect. At the end one of them looks at their watch and says, "ah, I think it's time to finish" (actually what they say is a lot less explicit than that). Everyone stands up and files out to the elevator, performs the ritual elevator bow, and the visitors depart. "What was that all about?" you're left thinking to yourself.

Since business does get done in Japan, just as effectively as anywhere else, you can't help wondering how this works. Of course the well-known Japanese dislike of conflict plays a part. But there are as many reasons for business partners to disagree in Japan as there are anywhere else - price, specification, availability, all that stuff - yet they evidently do get resolved somehow.

In the end I realised that geography has a lot to do with it. In the US, such meetings typically involve a plane journey, often an overnight stay. They're expensive and it's important to get the most out of them. In Japan, most large companies have their offices in Tokyo. It's rare that it takes more than a half-hour taxi or subway ride to get there. So meetings are cheap, and you can afford to do them quite often. They don't need to be so focussed, it's OK to let things kind of evolve naturally over several of these apparently rather loose discussions. That coupled with the natural Japanese way of doing things just leads to this completely different style.

Of course, as with everything in Japan, it may be that I've missed the point completely.

boost serialize - not such a good idea!

I'm a big fan of the boost libraries for C++. Mostly, they are a huge productivity gain - things like regex, function and bind can reduce the work involved in a complex program by half or more. So when I needed to pickle a very large (gigabytes) and complex data structure between restarts, boost was the obvious place to look.

Boost serialize certainly looked like the answer. Just add a few lines to each class saying what you what to save and restore, then with a single function call, you can pickle the whole structure and later reload it. It takes care of loops, diamonds and all the other things that happen in real-life data structures. And if you don't like the default way of saving something, it's easy to write your own. How good can it get!?

Well, that's the theory, and it could have been the practice too. But it isn't. For some reason known only to themselves, serialize's authors decided that the C++ inheritance mechanism wasn't for them. They invented their own completely parallel mechanism for dealing with polymorphism and subclasses. The effect is that trying to save my structure results in an exception thrown from somewhere in an enormous depth of function calls. I persevered, and tracked down what was happening. For some reason - and it's just impossible to plough through all the code and figure out the details - it silently ignores some of the calls to the subclass registration process. After several days of trying to figure out the details - on a live application because that's the only place the data can be collected - I have finally given up. I'll live without this capability in my program.

Unfortunately this is a common problem with boost. It seems to be de rigeur to invent new ways of doing things even though they are more complex and don't work especially well. The build system is another case in point - instead of using make, known and hated by generations of programmers, they invented their own, bjam, which is unknown and incomprehensible. If it works for you, great. If not, forget it.

I'll carry on using boost, but you do have to be selective. Unfortunately.

More thoughts on Boost here.

Divisumma, the Star!

A few weeks ago I got an email fom someone saying, in essence, "Can I come round and get some pictures of your Divisumma please?"

I was surprised, especially when I discovered he was coming from Detroit - I would have thought there would be more around than that. He is making a video about architecture for Yale, and had been specifically asked to include some footage of the Divisumma. He'd evidently had trouble finding one in the US but - thanks to this blog - he found mine.

He came by today and spent half an hour or so in the sunshine, shooting very close-up footage of the keyboard and the case. And, when he's finished putting it together, Divisumma will be a movie star!

Halfway to Upside Down

When I started this blog it was with the intention of writing regularly about aerobatics - hence the title. But you know how things are... I got involved with learning to fly the heli, which dramatically reduced the amount of acro I fly.

But today I flew the Pitts, and I took my camera with me. It's hard to take pictures of aerobatics while flying solo, I can tell you - trying to fly a tricky manouver left-handed while holding the camera with the right and trying to get it to focus on the right thing.

I did however manage to get a nice picture of a knife-edge. That's when you hold the airplane in a 90 degree bank, which means that the wings aren't providing any lift. So the only way to maintain altitude is to hold a very steep fuselage angle and let the fuselage provide the lift, which means LOTS of top rudder - in the Pitts S2C, full top rudder. Of course the drag is huge, so you slowly lose airspeed and eventually you will fall out of the sky. But you have long enough to take a nice pic, like this one.

Full size photo, and others from the same flight, here.