Pete Aguereberry's Car

One of the great characters in the history of Death Valley is Pete Aguereberry. His story is truly a dream come true. Born in the Basque region of southern France (as you can tell from his name), he was fascinated by the stories of the Californian gold rush. At 16, he emigrated to California. By 1907 he was installed in what's now the Death Valley National Park, with his very own gold mine. It hadn't been easy - there were several years of disputed claims and lawsuits.

He stayed there until he died in 1945, working the mine by himself. It must have been a very lonely and hard existence. He lived in a small house a short way from the mine, a long way from the nearest town even by car. Even another other human presence was an hour or more away, and much more before cars were available.

His name lives on today. Aguereberry Point is a 6500 foot peak overlooking Furnace Creek and the northern end of the Badwater valley, with magnificent views in all directions. He loved this view so much that he built the road to it, still in use today, so others could appreciate it too.

Not far from the house is the rusted shell of an old car. There are many pictures on the web, generally entitled "Pete Aguereberry's Car". But to me it looked like a late 1940s model, too late for it to have belonged to him. I was intrigued to know whether it could in fact have been his daily drive.

The wreck is in a sorry state. Everything that could possibly be taken, has been - the wheels, the seats, the instruments, nearly all the engine parts. Even the cylinder head has been taken. The body has been used for target practice and is full of holes. Vandals have evidently jumped on the roof, partly collapsing it. There is nothing that positively identifies the make or model.

However there is one strong clue. The engine is a very unusual type, with eight cylinders arranged in one long straight line. Nearly all eight-cylinder engines use the V8 configuration. "Straight eights" have always been a rarity. They take up a lot of space and the length of the crankshaft means that the front and back of the engine don't necessarily agree with each other, or with the camshaft, about exactly where they are in the four-stroke cycle. They've only ever been used in very high-end luxury models.

A quick Google search showed that in this period, there were only a handful of suitable models. Another helpful feature of the wreck is that it has only two doors - unusual in a luxury sedan. It was fairly easy to narrow it down to a Buick Sport from 1947 or 1948. All of the features compare, including things like the detailed line of the front wings flowing to the rear of the car and the remaining "fangs" of the grille. As further confirmation, the headless engine is clearly the overhead-valve type, like the Buick. Most engines in the late 40s were still of the "flathead" type, with the valves in the cylinder block.

So it couldn't have been Pete's - he'd been dead and buried in Lone Pine for at least two years before this car was built. Whose was it then? The story relates that towards the end of his life, he was helped with the mine by a nephew. Could this have been the nephew's car? In any case the mine must have produced well to finance a car like this, the equivalent of a high-end Mercedes Benz today.

Raw Sockets, Raw Brain

I've been working for a while on a networking app that involves moving packets at great speed from one interface to another, while doing a little processing on them. Key to the performance we need is to avoid copying data or making kernel calls for each packet. We already did some testing a while back with Netmap, which provides direct access from user-mode code to hardware buffers and buffer rings - perfect for our application. Intel now has something similar called DPDK, but we chose Netmap because it was there when we needed it and also because, being open source, it avoids having to do a secret handshake with Intel.

However Netmap has some limitations so we have to modify it before we can fully integrate with it. In the meantime I wanted to do some functional testing, where ultimate performance isn't essential. So it seemed obvious to code a variation which uses sockets, accepting the performance penalty for now.

Little did I know. Getting raw sockets to work, and then getting my code to work correctly with them, was really a major pain. Since I did eventually get it to work, I hope these notes may help anyone else who runs into the same requirement.

What I wanted was a direct interface to the ethernet device driver for a specific interface. Thus I'd get and send raw ethernet packets directly to or from that specific interface. But sockets are really designed to sit on top of the internal IP network stack in the Linux kernel (or Unix generally) - for which they work very nicely. It's pretty much an unnatural act to get them to deal only with one specific interface. It's also, I guess, a pretty uncommon use case. The documentation is scrappy and incomplete, and the common open source universal solution of "just Google it" didn't come up with much either.

Just getting hold of raw packets is quite well documented. You create a socket using:

    my_socket = socket(AF_PACKET, SOCK_RAW, htons(ETH_P_ALL))

and you will get, and be able to send, raw packets, with everything from the ethernet header upwards created by you. (You'll need to run this as root to get it to work).

Since my application runs as "bump in the wire", picking up all traffic regardless of the MAC address, I also needed to listen promiscuously. This also is reasonably well documented - it's done by reading the interface flags, setting the requisite bit, and setting them again:

    ifreq ifr;
    memset(&ifr, 0, sizeof(ifr));
    ioctl(my_socket, SIOCGIFFLAGS, &ifr);
    ifr.ifr_flags |= IFF_PROMISC;
    ioctl(my_socket, SIOCSIFFLAGS, &ifr);

Production code should check for errors, of course.

I got my code running like this, but it was still getting packets destined for all interfaces. This is where it got tricky and had me tearing my hair out. Some documentation suggests using bind for this, while others suggest using an IOCTL whose name I forget. It's all very unclear. Eventually, bind did the trick. First you have to translate the interface name to an interface id, then use that for the call to bind:

    sockaddr_ll sa;
    ifr.ifr_addr.sa_family = AF_INET;
    strncpy(ifr.ifr_name, sys_name.c_str(), sys_name.size()+1);
    ioctl(my_socket, SIOCGIFINDEX, &ifr);
    sa.sll_ifindex = ifr.ifr_ifindex;
    sa.sll_family = AF_PACKET;
    sa.sll_halen = ETH_ALEN;
    my_mac_address.to_msg(sa.sll_addr);
    bind(my_socket, (sockaddr*)&sa, sizeof(sockaddr_ll));

A couple of wrinkles here. It may seem intuitively obvious that sockaddr_ll is in effect a subclass of sockaddr, but it isn't documented that way anywhere that I could find. And finding the header files that defines these things, and then the header files they depend upon, and so (almost) ad infinitum, is a nightmare. In the end the best solution I could come up with was to run just the preprocessor on my source files, and look at the resulting C code. And note the ugly cast in the call to bind, because in the world of C there is no such thing as inheritance - the common superclass is actually a macro, and as far as the compiler is concerned sockaddr and sockaddr_ll are completely unrelated.

Another wrinkle is the bind function itself. I use boost::bind all the time, far too much to want to type or read the full qualified name, so my common header file contains "using boost::bind". That absolutely wipes out any attempt to use the socket function of the same name. The only way round it is to define a trivial function  called socket_bind (or whatever you prefer), whose definition in its .cpp file studiously avoids using the common header file. It's only a nuisance, but it did take a little thought to come up with a reasonable workaround when I first ran into the problem.

So, with all this done, I was receiving raw ethernet frames, doing my thing with them, and sending them on through the paired egress interface. Wonderful.

Except actually, not. The frames I was receiving were way longer than ethernet frames. Since I'm using jumbo-frame sized buffers (9000 bytes), I'd receive them OK but not be able to send them. But sometimes, they were even too large for that, and I wouldn't receive anything at all. And this was where things got really frustrating.

The first move, of course, was to check the MTUs (maximum permitted frame size under TCP and IP) on all the relevant interfaces. They were fine. Then I found a suggestion that TCP will use the MTU of the loopback interface, relying on the driver to straighten things out. So I set that down to 1400 too. It still made no difference.

At that point, my code didn't send ICMP messages for too-large packets, which a router or host is supposed to do. I spent a whole Saturday in a distress-coding binge writing my ICMP implementation, and changing my super-slick multi-threaded lock-free infrastructure to accommodate it. It did make a very small difference. Instead of just blasting away with giant frames, it would send each packet initially as a giant frame, then retransmit it later in smaller, frame-sized chunks. The data did get through, but at a pititful rate with all those retransmissions and timeouts.

Finally, after much Googling, I discovered the "tcp segmentation offload" parameter. That made no difference. With more Googling, I also discovered "generic segmentation offload". That made things better, though still far from good. I had Wireshark running on all four interfaces - the two test systems running iPerf, and both interfaces on my system in the middle. (All this is running as VMs under VMware, by the way - see earlier rant reasoned discourse about the problems I had trying to get Xen to work). Wireshark clearly showed packets leaving the first system as correctly sized ethernet frames, yet when they showed up at the second system they'd magically coalesced into jumbo frames.

After much cursing I found the third thing I had to turn off, "generic receive offload". The design assumption here is that practically all network traffic is TCP - which after all is true. So the hardware (emulated in my case) combines smaller TCP packets into huge ones, to reduce the amount of work done in the network stack. It's an excellent idea, since much of the overhead of network processing is in handling each packet rather than the actual data bytes. But of course it completely broke my application.

This is not one of the better documented bits of Linux. There is - of course - a utility to manage all this stuff, but it's so obscure that it's not part of the standard Ubuntu distribution. You have to explicitly install it. So a summary of what is required to solve this problem is:

    sudo -s

    apt-get install ethtool

    ethtool -K ethn tso off

    ethtool -K ethn gso off

    ethtool -K ethn gro off

All of this requires root privileges. Whoever wrote ethtool had a sense of humor - the '-K' option sets parameters, the '-k' option (lower case) shows them. It would have been too hard, I suppose, to think of a different letter for such fundamentally different operations.

With that done, my code sees the packets at their normal (no greater than MTU) size. Finally, I could get on with debugging my own code.

Wakkanai - Tales from the Frozen North

A few years ago we visited Hokkaido, taking the train across the island to Abashiri and then driving our rented car all around the southern coast. We spent an especially memorable night in the wind capital of Japan, Erimo Misaki. When I wrote about that trip, I ended with "we're looking forward to returning in winter when everything is covered in snow". And so when Isabelle had a meeting planned in Sapporo in December, we just had to make an adventure out of it.

Since we'd already seen a lot of the southern half of Hokkaido, we decided to go north. The truth is that there isn't much to visit in northern Hokkaido - it's pretty, but extremely empty. We aimed for the northernmost town in Japan, Wakkanai, which is just a few miles from the northernmost accessible place in the whole country, Soya Misaki. It's also the basis of a pun in Japanese, since it sounds very similar to "I don't know" - "Where is this train going?", "Wakkanai". "But you're the driver, surely you must know". Actually, that's probably about the most interesting thing about Wakkanai.

First, though, we had to get there. We flew with JAL from San Francisco to Haneda. This is just so much better than arriving in Narita. First, it's a short subway or monorail ride from the city, or an almost-affordable taxi ride - compared to a long and infrequent train ride or the interminable limo bus (forget a taxi, which will cost at least $250). Second, it just works amazingly well. There are never any lines - for immigration, for check-in, for baggage. If US airports worked even a tenth as well it would be a huge improvement.

The next day we had free time in Tokyo. We had lunch at our favourite kaiten Sushi restaurant, then went for a walk in the neighbourhood of our hotel in Ebisu,  and discovered Shirokanedai Park. This is a nature wilderness smack in the middle of Tokyo - we could hear the Yamanote line trains as we wandered around the gorgeous autumn foliage. Cats and dogs aren't allowed, but nobody had told the ginger cat I caught on camera racing through the undergrowth. Later we had a very pleasant evening with old friends from the days when I used to visit Japan several times a year.

Next morning, return to Haneda. It's the fifth-busiest airport in the world - in the US, only Chicago and Atlanta handle more passengers. Yet it always seems deserted, and once again there were no lines, at all. On takeoff there were fantastic views of Fuji - it was a very clear winter morning. Sapporo had a generous cover of snow, with a lot more falling during the afternoon. The view from our twelfth-floor hotel room of the constantly changing snowfall was magnificent. I ventured outside only to buy the Japan Railways timetable - as on every trip - which turned out later to be a very wise purchase.

Next day Isabelle was busy. I decided to take the train somewhere just for the pleasure of it, and ended up in Noboribetsu on the coast. The name means "special climb", except it doesn't because Kanji are just used for their phonetic value in Hokkaido place names. "Betsu" is actually the Ainu word for river, and appears everywhere - always written as 別 meaning "special".

Noboribetsu is a typical small rural town, which is to say there's absolutely nothing going on there and it has a faintly derelict, mouldy feeling to it. I walked down to the harbour, which was of no interest at all, then found the only place that was open for lunch. It was no surprise to see practically everyone from the restaurant waiting on the platform for the return train to Sapporo.

The next day our train, the Super Soya, departed at 7.48 precisely - Japanese trains always run to time. Well, almost always, as we found later. The train chugs along for five hours, mostly spent on a single track winding through snow-covered fields in narrow valleys as it heads generally northwards towards Wakkanai. The views were magnificent. Finally, after a mountain pass and tiny villages, we saw the sea, and soon after we arrived at Wakkanai station.

Here we had to hurry. Not having a car, the only way to Soya Misaki was by bus. There's exactly one in the afternoon, and it leaves 33 minutes after the train arrives. In that time, we had to get our bags to the hotel, and get back to the station. We made it, with a few minutes to spare - just as well, because the only alternative was a taxi at absolutely astronomical cost (around $150 for the round trip).

The bus trundled along through Wakkanai, all grey and cold and snowy. At the new shopping mall - which seems to be the happening place in town - a group of old ladies, their weekly (monthly?) trip to the big city over, boarded the bus to return to their tiny fishing villages.

Finally we arrived at Soya Misaki, several other people getting off with us. This is the northernmost point in Japan that you can visit - there's a tiny uninhabited island that's further north but you'd need your own boat to get there. The southern tip of Sakhalin, in Russia, is about 40km away. On a clear day you really can see it, or so they say. This day was cold - below freezing, and with a strong breeze. There's very little to see, just a couple of monuments. We looked for the monument to the victims of KAL007 - the plane that in 1983 inadvertently overflew the USSR and got shot down. But we couldn't find it. Only later did we discover that there are several more monuments on top of the hill, a short walk from the shore but not possible in the 25 minutes before the return bus. And you really don't want to miss that, because the next one is another 3 hours.

Once you've scanned the horizon for Sakhalin, and taken each other's pictures in front of the various monuments, the only thing left is to visit the northernmost tacky tourist junk shop in Japan. Every attraction in Japan has these (though there were none in Noboribetsu), selling much the same variety of made-in-China local handicrafts. This one was special, though, as definitively the furthest north in Japan. We bought some postcards and some dried squid, of which more later.

The only thing left now was to wait for the bus. There's a shelter, but it's set back from the road. And we really didn't want to miss it. It was very cold and windy, and the idea of spending another three hours in the gift shop was enough to make me happy to brave the cold. The sight of the bus, when it arrived (precisely punctual of course), was truly the best part of the excursion.

We trundled back to Wakkanai, sharing our dried squid with a couple of Taiwanese girls who were on the same pilgirmage as us. It tasted funny. Later we discovered that this wasn't the classic Japanese snack, which is enjoyable in a chewy and getting-stuck-between-your-teeth kind of way, but an "improved" version made from New Zealand squid soaked in corn syrup. Yuck.

Once back in town, there really wasn't much to do. From our room we had a view over the generously-sized but completely empty harbour. In summer, Wakkanai is the departure point for the much-visited islands of Rebun and Rishiri, but in winter they're covered in snow and nobody goes there. There are also - maybe - boats that visit from Russia.

For dinner, we chose the most famous restaurant in town. Oddly, it's called kuruma-ya, written 車屋, which means "car dealer". We were almost the only people there, but we had an excellent meal including gigantic Hokkaido crab legs and much other seafood.

Our flight next day was at 1.30pm, so to occupy the morning we took a taxi out to Cape Noshabu, the northernmost point in the town itself. There's not much there, so after the obligatory snowy picture, we continued along the coast road. It is very bleak. There are a few houses in the snow, facing out over the cold, grey ocean, surrounded by rickety wooden structures for drying seaweed (kombu). You can see the two islands, grey mountains looming out of the mist and chill.

We still had time when we returned. We discovered the Russian shopping street. Even though the hotel clerk said there are no more tourist boats from Russia, there is this whole street of tourist-type stores, with everything labelled in Russian. Compared to Khabarovsk in Sakhalin, Wakkanai is the sunny south. But the street was almost deserted and looked very sad and run-down. Maybe they run in summer.

Finally it was time to go to the airport. It was sunny when we left, but when we arrived just 20 minutes later there was a blizzard and you could barely see across the runway. We waited, and waited - the display said something about "investigating the weather conditions" which wasn't very encouraging. Finally Isabelle had exhausted the possibilities of the tiny gift shop and we started to go through the security. Just then there was an announcement, and the guard held up his hands in a big "X". What had we done wrong?

It took only a moment to realise that the flight had finally been cancelled. It was very lucky that we hadn't gone airside, because it meant we were in the first few people in the queue at the desk. But what on earth to do now? There's exactly one flight a day to Tokyo, and even if we got on the next day's flight it could just as well be cancelled too. That would be a big problem, since our flight back home was that night. There are a couple of flights to Sapporo, but they're on little commuter turboprops - there were more than enough people waiting to fill the next two flights.

The alternative was the train, again. Luckily Isabelle had retrieved our bags, and with them the JR timetable (moral: never let it out of your hands). I realised we might just make the midday train, which left for Sapporo in half an hour. But how to get back to the town? The taxi rank, outside in the blizzard, was deserted.

Isabelle rushed outside and talked to a guy who had exactly the same reflex as us. Somehow, despite not having a word in common, she communicated with him and agreed to share a taxi, should one arrive. Which it did, in the nick of time. Constantly encouraged by our new friend, the driver went as quickly as he could on the snow and ice covered roads. We arrived at the station seven minutes before the departure time, enough to buy our tickets and see the blessed train arrive.

We had no idea what we'd do once the train took us back to Sapporo, but it was much better to be stuck in Sapporo than Wakkanai! The Tokyo-Sapporo air route is the busiest in the world, so getting back in time for our flight home would not be a problem. JR timetable to the rescue again - it showed that we should easily be in time for the last couple of flights.

This train journey was painful, unlike the previous day. The train was much older, and stunk of diesel fumes. There was no food service, so we survived on a few nuts that we had with us. It was overheated and stuffy, and for much of the trip it was dark outside. But still, we weren't stranded in Wakkanai! And for part of the way, we sat at the front of the train getting a driver's-eye view which is much better than looking sideways. Tiny country stations flashed by, no more than a short platform and a lamppost - yet surely a lifeline to some remote community. The track was completely buried in the snow, only the shiny tops of the rails showing. Visibility was terrible, no more than a couple of hundred yards in the swirling blizzard, and you can certainly appreciate the driver's task, having to slow down for invisible bends, known only through his minute knowledge of every twist and turn in the mountainous track.

One thing we noticed is how polite Japanese trains are. American trains have incredibly loud horns that you can hear from miles away. Japanese trains just have a little high-pitched whistle that goes "pheeeep" - you can barely hear it from inside the train. It's as though the train is saying "shitsurei itashimasu" (the super-polite version of "excuse me") in a squeaky voice just like the shop ladies.

Eventually we arrived in Sapporo. And we were late! This is unknown in Japan, a full quarter hour behind schedule, most likely due to the terrible conditions in the mountains.

Finally we got to the ANA desk at the airport - no queue of course. The lady was very helpful, took all our details, told us there was room on the last flight - and then showed us the price. It was over $1000! I'd been kind of expecting that - I'd managed to call ANA from the train and they weren't very helpful, with our "Explore Japan" tickets which are much cheaper than the published fare. Without much hope, I explained again that we hadn't chosen to travel this way, that the flight had been cancelled and all the rest. There was much "sooouu desu nee" and typing and phone calls. And then - mirabile dictu - she handed us two boarding passes. With my few words of Japanese I'd just saved over $1000!

The plane and then the subway delivered us to Shinagawa station at exactly midnight. We couldn't believe the crowds. Trains departing outbound were so full that people were left standing on the platform. It needed the white-gloved crowd-pushers from the Yamanote line. It was almost impossible to get through all the people to the taxi rank outside. It was the second-last Friday before the Christmas holiday, and evidently Tokyo's salary-men (and women) were partying to the full.

We'd barely eaten all day, so dinner was a priority. When we arrived from the US we'd discovered an Italian wine bar on the wrong side of the tracks at Ebisu station, practically in the catacombs down at street level. We rushed back there. Every few minutes one of us would say, with a sense of wonder, "We're not in Wakkanai!". It was such an incredible relief to be back in Tokyo.

The next day, Saturday, was our final day, but the JAL flight from Haneda doesn't leave until midnight. We went shopping in Ginza, which I haven't done for a very long time. It's an amazing place, in such a Japanese way. The quality of everything on sale is just astounding, whether it's food, household stuff, porcelain... anything. I could have spent the whole day just revelling in the place, admiring things that normally I'd never even pay attention to. I found a little etched brass model of the Toyota FJ, which I just couldn't resist (although it is still sitting wrapped, accusingly, on my desk, waiting to be built). It comes from a lovely series of architectural models, allowing you to build things like a typical Tokyo street scene. And of course I just had to visit Tenshodo, one of the world's greatest model train shops, three tiny floors crammed full of every make in the world (even Hornby from Britain, I wonder who on earth buys that in Japan), and all within a few paces of the Ginza Crossing.

And so, finally, to Haneda, and our flight home. Wakkanakatta.

Ubuntu + Xen = Disaster!

I'm writing this as I reinstall Ubuntu 12.10 on my new test machine. My idea - to test some networking software I'm working on - was to buy a new high-power desktop machine (done), install Ubuntu on it (done), then run Xen to create a simulated network.

Xen is really the open source software from hell. Nothing works the way it's apparently supposed to, and actually in the end nothing works at all. You search around on Google looking for people who've had the same problem, you try the fixes they suggest. Sometimes they work, taking you a few milliseconds further towards failure, other times they have no effect. It's just incredibly frustrating.

So now I'm reinstalling the system and I'll run VMware instead. It's a shame, I liked the idea of Xen, being free and open source and all that. But unless you just have way too much time on your hands, it's hopeless. For me, this whole business of building systems is a distraction from the important matter at hand, which is writing code for my own software. The less time I spend on it, the better.

The first problem was the book I bought. I was on a business trip for a week, so I thought the ten hours or so on the plane, and lonely nights in a hotel, would be the perfect occasion to get myself up to speed. I bought "Running Xen" by Matthews et al. The "et al" turns out to be extremely important. The book is terribly written, by Matthews' entire class of students, none of whom can write very well, and for sure can't write consistently. It never really quite tells you how to do anything, constantly distracting itself with dire warnings about all the bad things that can go wrong. Actually, given my experience with Xen, maybe that's not completely inappropriate.

I bought another book, "The Book of Xen". This is at least well written, and it probably isn't the authors' fault that none of the recipes they give for how to do things actually turn out to work.

Anyway, let's go back to when I didn't know how bad this all was. I installed Xen, made the necessary configuration changes, and rebooted. Woohoo! There I was running a hypervisor. I could type "xm list" and see it, complete with my one and only virtual machine (Dom0 in Xen-speak). So now, all I had to do was follow the recipe from the book, and I'd have my software - which was already running on the bare machine - running on a VM.

Everybody tells you that if you want your VM to use a file as its virtual disk, you absolutely shouldn't use the "loopback driver", you should use something called blktap. So naturally that's what I tried to do. I copied magic incantations into various configuration files, created my virtual disk, and tried to mount it.

Nothing worked. I tried all sorts of combinations of things, and of course I Googled all the error messages and the like. I found lots of suggestions, but nothing that actually helped. That was when the curtain started to open on the fire and brimstone of open source hell. There were loads of responses along the lines of "ah, but you need to install xxx then edit /etc/yyy". But xxx doesn't work on Ubuntu. Google. "xxx doesn't work on Ubuntu, but you can do xxxzzz instead". Well yes, but "xxxzzz" doesn't actually do what "xxx" does. And so on. One article even says "the great thing about Xen is that there are so many different ways to do the same thing." Maybe they had their tongue in their cheek. In any case they were wrong, because the truth is that there are so many ways to fail to do the same thing.

I was about to give up when I suddenly thought to try the dreaded loopback driver, which is supposed not to have good performance. (I've heard that Butler Lampson, one of the great minds in computer science, once said "Performance is a characteristic of a correctly functioning system" - in other words, it doesn't matter what the performance is, if it doesn't actually work. When I had the opportunity to ask him he denied having said it, but it's true anyway).

And... it worked! Replacing "tap:aio:" by "file:" suddenly had everything working correctly. I could mount my virtual device, treat it just like a disk, copy my host system onto it. Now I was ready for the next step, boot the VM with an exact copy of the Ubuntu 12.10 system that was running as the host.

I even tried going back to blktap for the guest system, figuring that maybe there was some kind of conflict with running it in the host. Well, that of course got nowhere. But I edited the config file and... it got another five whole lines further in the console log before failing in a different way.

I Googled the error messages, as you do with open source (open source would just never work at all without Google). The problem was that the root file system wasn't getting mounted. This may (or may not) have something to do with the way Xen deals with booting guests. Instead of doing what the hardware does, and executing files from the guest's disk image, it boots using the host kernel, then somehow flips over to the guest in mid-boot. Why it does this I don't know, but it creates whole chapters in the textbooks explaining how to deal with kernel incompatibilities and the like. And evidently I'd just been bitten by one of these - even though I was using the same system for the host and for the guest.

To cut a long story short, I tried a lot of things - Pygrub, virt-manager - and none of them would work. They all failed in some incomprehensible way, and Googling just produced confusing, conflicting advice, which typically started with "rebuild the kernel..." or "install this completely different toolset".

Yet Xen is in widespread use by cloud hosting companies - for example, by Amazon. I can only suppose that if you have the resources to experiment with different distros, kernel builds, toolsets and configurations, you can eventually find a combination which works. But it certainly isn't for the casual user like me, who just wants to get something working in a day or so. It may be that Ubuntu, or its latest version, is part of the problem. The Ubuntu web pages seem to say it should all work, but there are also a lot of references to things that don't quite work the way they're supposed to under Ubuntu.

I suppose I should have known, really. I worked for a while with a company which had a virtualized software product. They'd started with VMware as the base, and then customer pressure had forced them to port to Xen - just before I joined and found myself responsible for it. It was a nightmare - nothing worked, Citrix (the owners of Xen) were incapable of providing support, and the Xen open source community just laughed when we asked them for advice. "Oh, you're using the xyz toolset - nobody uses that any more, everyone is using pqr. The latest release is pretty good, it mostly works and there are quite a few patches for the stuff that isn't really there yet." The project over-ran by months and only "worked" thanks to numerous hacks and workarounds.

So, here I am reinstalling the machine. I'm annoyed about the time I've spent trying to understand Xen, that I'll now have to spend getting to grips with all the utilities for VMware. But it surely can't be worse... can it?

Tales from the cup shelf #1: Prague

For years now, whenever we travel (often), we try to buy a souvenir mug. Looking at the cup shelf the other day (well, there are three of them actually - I did say we travel often), I realised that collectively they amount to quite a story.

The first one comes from Prague, our first trip there together in 1993. The Communist era was barely over yet the city was already transformed, cleaned up and full of life. Music was everywhere. Walking around, we were constantly being given fliers for evening chamber concerts, while street performers gave excellent impromptu recitals on nearly every corner.

When I was a teenager - no mugs from then I'm afraid - I happened to buy an album of harpsichord recitals by an artist called Zuzana Ruzickova (give or take a few accents). I was enthralled by it, it's by far the best harpsichord album I've ever found. So you can imagine how I felt when one of the fliers was for a performance by none other than Ms Ruzickova herself. And it was extraordinary. Unlike a piano, a harpsichord can only produce notes of one intensity. No matter how gently or hard you hit the key, the sound is always the same. So the only way to make it louder is to play more notes. That's why harpsichord music always has passages with an incredible number of short notes - to make it louder. Just as I was wondering how she could possibly move her fingers that quickly, in a passage composed entirely of demi-semi-quavers, she doubled up again, to hemi-demi-semi quavers. The effect was electrifying.

All of these concerts - we went to others too - were in delightful buildings, older than the music itself, with mysterious inner courtyards and staircases. These led to small rooms - whence chamber music - where you could literally reach out and touch the musicians (not that we did, though I did introduce myself to Ms Ruzickova after the concert, to thank her for such a wonderful introduction to what is often quite frankly a pretty boring instrument).

Of course we visited Wenceslas Square, site of Jan Palach's 1969 self immolation. And the beautiful Vysehrad Park on top of the hill overlooking the Danube, and Prazsky Hrad, the castle high up on the banks. Everyone does. But my memories are of other places, of the quiet cloistered side street leading to the unforgettable Blue Duck restaurant, where we spent a long and largely afternoon-destroying lunch. Or of the tram ride from our hotel, which was in the suburbs, rattling through bustling residential neighbourhoods to the city centre. And then there was the Tatra museum - a whole collection of those odd Czech luxury cars with their rear-mounted, air-cooled V8 engines.

The mugs, the perfect size for after-dinner coffee, were once six, but now only two remain. They've done better than the crystal glasses, though - of the twelve we bought, only one is left, and even that is chipped. (Quite how we brought all this booty back with us on the plane, I do not recollect.)

My first Rest API

For several months I've been working on an application that I always knew would sooner or later need a Rest API added to it for configuration and monitoring. These days, if you don't have a Rest API, you're nobody - even Cisco has added one to IOS. This would be the first one I've created myself.

My goal was to make this as simple and as little work as possible. My only previous experience of Rest APIs was the exact opposite. The designers had gone out of their way to make the Rest API as complex and burdensome to implement as possible, spending whole days in minute reviews of parameter names and metadata like XSD files. They had also built an exceedingly complex implementation - a perfect illustration of the French expression une usine à gaz. The bits that needed to have high performance were in PHP, while the low performance string handling was done in C. Honest. Python was in the mix somewhere too, and you could probably find Cobol and APL if you looked hard. It was a big mess, and the exact opposite of what I wanted to achieve.

My main goal was to do as little work as possible, both in building the initial infrastructure and even more so when making additions later. I'm a one-man band on this project, 15,000 lines of code in the last six months, and the less work it takes to do a relatively peripheral thing like this, the better. That said, I'm always willing to spend a little longer getting the infrastructure right if it makes less work later on, and especially if it makes for less error-prone repitition.

I wanted to make it as "truly Restful" as possible, respecting the Rest orthodoxy. Though this turns out to be harder than you'd think. Most writings on the subject are distinctly obscurantist, leaving me with the same feeling as when I try to understand writings on philosophy or (worst of all) sociology - I know what all the words mean and I can pretty much figure out the sentences, but I have no clue what they are actually trying to say. A principle which is held in especially high esteem is called HATEOAS. People write pages and pages about how bad it is not to follow this principle, but nowhere have I found an example which illustrates what it actually means. I think it means that you should have lots of hyperlinks to associated resources, which is a good idea in human-oriented web stuff too. So that's what I did, although honestly it might mean something completely different.

My underlying code is all written in C++. The configuration interface is exposed through a singleton policy_manager class, which has functions like add_interface or show_acl that access the underlying C++ objects. I'm a big fan of using Python to do anything which isn't performance critical, so the first thing to do was expose this as a set of Python functions. Boost::python to the rescue for this part - it took only an hour or so to expose these functions in Python.

One thing I really did not want to do, was to have to repeat each function declaration over and over for different parts of the interface. I already had the usual .h/.cpp files for the C++ code. It was a 15 minute task with emacs to transform the function declarations in the .h file into some macros that captured the essentials of the functions, i.e. the names and details of the parameters, in a way that could expanded differently as required in the different places.

The C pre-processor (CPP) is very limiting in what it can do. To me the nec plus ultra of integrated macro facilities has always been the DEC assemblers. They didn't do much more than CPP, but the little there was made a huge difference to the power and flexibility. Of course there's always M4 - I used it for a big project once in the past and it is amazingly powerful, but I'd prefer to do without the extra build complication, not to mention remembering how to use it.

I came up with what amounts to an Application Specific Language to describe my functions. It's not especially pretty but it gets the job done. Oh, for a "shift" function (like the shell) to deal elegantly with variadic parameters.

All that done (it actually didn't take very long), I wanted an RPC so that my Rest server didn't have to be in the same process as the operational stuff. Pyro was the answer to that. It's a truly amazing little package, that lets you export a Python object with close to zero effort. My RPC server is about a dozen lines of code. It creates the Python object corresponding to my singleton policy_manager, and exports it. Problem solved, with amazingly little work.

The next step was to select a framework for the Rest server itself. I looked at Django, but it's really designed to do a lot more than just serving web pages, and correspondingly rich and complex. I settled on Flask, which has a very intuitive and simple way of relating URLs to the code that serves them.

To minimize the amount of work needed, I made every class work the same way. For the POST method, there is an add_... function, for GET a show_... function and so on. So the heart of the server is a table which maps Rest prefixes to the family name of the function. As a trivial example, the prefix 'interfaces/' maps to "interface", so a GET to 'interfaces/' results in a call to show_interface. There's some generic code to do things like extracting names from URLs (e.g. 'interfaces/eth0/hosts/1.2.3.4') then depending on the verb to construct the corresponding function call.

One tricky point was the construction of links in the Rest output. The underlying C++ code knows nothing about Rest, and certainly not about the specific URL structure in use, and I want to keep it that way - so it can't generate explicit links. The solution is to pass a class name as well as the leaf instance name - e.g. {'class':'interface', 'name':'eth0'} - as the value for a link to another object. In the C++ code these attributes are pointers, so the generic attribute output code understands that pointers should be passed in this form. Then the Rest server code has a post-scanner that takes the output from the show_... functions, and using its class-to-URL mapping table, turns these into full-formed URLs.

One nice thing is that the Python code in the Rest server has no idea what is and isn't available for each class. If a class doesn't support POST, for example, then the server will generate a call to add_<class> anyway. Pyro will eventually determine that there is no such function, returning a corresponding error, which the Rest server turns into a 405 (method not permitted) error.

I made a decision early on to support only Json. This seems to be pretty normal now. XML generates a lot more work around metadata, for no added functional value. It wouldn't be hard to also generate XML, but it's hard to see why it would be useful.

In the end it took about three days of work, spread out over the Christmas and New Year break, to get a fully functioning Rest interface. There has been quite a bit more work since in the backend, as I've realised how information should be presented. The Rest server is about 250 lines of Python. Each new class adds exactly one more line, to the mapping table. And there's no PHP!

Bye bye helicopter

Well, the day has come when I have to face up to not flying the helicopter any more. It's been obvious for a while, really, or at least it should have been. I've been trying to fly at least once a month, but not always succeeding. And slowly, my skill level and comfort level have been dropping off.

Flying a helicopter is not like riding a bicycle (in a number of respects, actually). It's a very, very delicate thing, and it takes constant practice to be comfortable with it. Take off and landing are much more delicate than in a plane, you have to touch down with absolutely no lateral motion or rotation, and if there's the slightest wind this is a lot harder than it looks.

The odd thing about flying helicopters is that it can only be done if you are completely relaxed. And it just happens to be about the most unrelaxing activity you can ever undertake. The slightest false move will, at best, wreck half-a-million or so dollars' worth of machinery, and at worst kill you or somebody else. So if you're not completely on top of things, you tense up and then it's just about impossible to do anything smoothly or properly.

The very worst example of all this is slope landings. Imagine you have to land to pick up an injured person, but there is nowhere absolutely level. You line up parallel to the slope, and very gently touch the uphill skid on the ground. There must be absolutely no motion of any kind relative to the ground, even if you're doing this in a gusty wind. Now you slooowly lower the other skid to the ground, by lowering the collective, meanwhile moving the cyclic to the uphill side so the rotor shaft remains vertical relative to gravity, not the ground. Takeoff is the opposite. The slightest horizontal motion can tip the heli onto its side, with expensive and possibly fatal results. If this sounds hard - well, you have no idea until you've tried it. And you have to be able to do this with absolute confidence and smoothness to claim to be a helicopter pilot.

There's a famous pianist (I can't find who) who said, "If I don't practice for one day, I can hear it. If I don't practice for two days, the orchestra can hear it. And if I don't practice for a week, the audience can hear it." There are some things where, no matter how much practice you've had and how much natural talent, if you don't practice, practice, practice, you will lose your touch.

Flying the helicopter is one of the most enjoyable things I've ever done - especially when I was really on top of it. You have a sense of controllability that absolutely nothing else gives you. You can hover at any altitude, go backwards and sideways, do pedal turns, with absolute precision. It's really an amazing feeling. But for actual practical use... there's really not that much you can do with a helicopter, that you can't do with an airplane. Of course that's not true if your business is long-line work (using the aircraft as a flying crane), or rescuing injured people from road accidents, or a bunch of other extremely useful things. But those are professions, not hobbies.

In theory a helicopter can land on any more-or-less flat and level piece of land with room for the rotor blades, like your backyard, or a restaurant parking lot. You can land anywhere, go anywhere. Well... except you need permission from the landowner, which is unlikely to be forthcoming because of liability issues. In California you can't land on a public road. If you go out into the wilderness somewhere, nobody is likely to see you - but you could be in big trouble if they do. And speaking of wilderness, landing in a Designated Wilderness - which means most land in National Parks and a lot else too - is flat-out illegal and will get your helicopter confiscated. I called the Bureau of Land Management (BLM) once to ask about landing on BLM land - which covers an awful lot of middle-of-nowhere, desert and so on. Several people were very helpful but in the end nobody knew the answer. And forget keeping your heli in the backyard, regular use of an off-airport location requires all kinds of permission which you'll almost certainly never get.

And that's if you have your very own helicopter. If you're renting - and there are very few places that will let you fly without an instructor, other than for specific training purposes - then you're subject to the owner's limitations. They have to worry about liability too, plus they don't want you wrecking their aircraft. Most likely, this means no off-airport landings, and no overnight stays.

Owning a heli makes a wonderful dream, but unless you were an angel investor in Google then you almost certainly can't afford it. It's not just the capital cost, which is comparable to an airplane, but also the running cost. Just insurance is well over $10,000 per year, and maintenance is much more than for fixed-wing - which given the number of flight-critical rapidly-moving parts is not a surprise.

And there is one really bad thing about helicopter flying - airports. Given you can't really land anywhere else, this is kind of important. The problem is, airports are designed for and mostly used by airplanes. And airplanes fly very differently from helis. For one thing, even the humblest trainer is a lot faster, especially on final approach. So flying anywhere near an airport is little short of terrifying, as you feverishly scan the skies for other aircraft, whilst trying to work out where you are going to land and how to get there while keeping well out of everyone's way.

In an airplane, there's only one thing to do. There's one runway in use (except at the largest airports), and that's where you're going to land. You fly a conventional and well-defined traffic pattern, as does everyone else. You need to keep your eyes wide open but you won't find anyone suddenly flying at right angles to you - except those pesky helicopters of course. But the helicopter can land anywhere, and certainly doesn't need a runway. A few airports have designated helipads, and even well-defined routes to get to and from them - but most don't. So you study the airport diagram in the A/FD and try to figure out where you can stay away from the traffic pattern, but it's often not obvious.

Towered airports are mostly simpler, since the tower will generally take care of separation. But not always - Concord tower once very nearly killed me, giving me an unconventional departure that took me straight through the path of an airplane that was also talking to them. We missed by about 100 feet. Another problem is that you need to be very clear with tower what you're planning. A busy airline airport once gave me a takeoff clearance without being precise about the route. The controller was pretty grumpy anyway, and I didn't go through exactly what I was planning with him - which was a big mistake. After departure I got the dreaded "please write a number down to call on landing". Nothing came of it, in fact they were quite nice about it, but they made it very clear that my departure route had been a surprise to them.

It has given me some unforgettable flying moments, like my very own self-fly helicopter tour of Kauai. And flying (perfectly legally) under the Golden Gate bridge. I don't for a moment regret the time and money that I spent learning to fly the heli. But staying current enough to fly safely and with confidence, for no obvious practical reason, is another question.

All this came to a head when my instructor bluntly told me that I was no longer proficient enough to solo in his helicopter. It didn't come as a big surprise, I'd been feeling less confident myself too. But it did mean I had to make a big decision. Do I fly more often, at least a couple of times a month, and more for a while, to bring my skills back to the right level and keep them there? But that would take a lot of time, which I really don't have, and even more money - renting a Robinson 44 is $500-600 per hour. It's just not realistic, for me anyway.

So that's it, bye bye helicopter. Maybe I'll fly an occasional hour of dual, just so I don't forget completely. I fear that I'll find it more frustrating than anything else, feeling my own lack of skill and confidence and knowing that I can't go for an occasional solo flight. On the bright side, my bank account has been rejoicing ever since that depressing meeting with my instructor.