Raw Sockets, Raw Brain

I've been working for a while on a networking app that involves moving packets at great speed from one interface to another, while doing a little processing on them. Key to the performance we need is to avoid copying data or making kernel calls for each packet. We already did some testing a while back with Netmap, which provides direct access from user-mode code to hardware buffers and buffer rings - perfect for our application. Intel now has something similar called DPDK, but we chose Netmap because it was there when we needed it and also because, being open source, it avoids having to do a secret handshake with Intel.

However Netmap has some limitations so we have to modify it before we can fully integrate with it. In the meantime I wanted to do some functional testing, where ultimate performance isn't essential. So it seemed obvious to code a variation which uses sockets, accepting the performance penalty for now.

Little did I know. Getting raw sockets to work, and then getting my code to work correctly with them, was really a major pain. Since I did eventually get it to work, I hope these notes may help anyone else who runs into the same requirement.

What I wanted was a direct interface to the ethernet device driver for a specific interface. Thus I'd get and send raw ethernet packets directly to or from that specific interface. But sockets are really designed to sit on top of the internal IP network stack in the Linux kernel (or Unix generally) - for which they work very nicely. It's pretty much an unnatural act to get them to deal only with one specific interface. It's also, I guess, a pretty uncommon use case. The documentation is scrappy and incomplete, and the common open source universal solution of "just Google it" didn't come up with much either.

Just getting hold of raw packets is quite well documented. You create a socket using:

    my_socket = socket(AF_PACKET, SOCK_RAW, htons(ETH_P_ALL))

and you will get, and be able to send, raw packets, with everything from the ethernet header upwards created by you. (You'll need to run this as root to get it to work).

Since my application runs as "bump in the wire", picking up all traffic regardless of the MAC address, I also needed to listen promiscuously. This also is reasonably well documented - it's done by reading the interface flags, setting the requisite bit, and setting them again:

    ifreq ifr;
    memset(&ifr, 0, sizeof(ifr));
    ioctl(my_socket, SIOCGIFFLAGS, &ifr);
    ifr.ifr_flags |= IFF_PROMISC;
    ioctl(my_socket, SIOCSIFFLAGS, &ifr);

Production code should check for errors, of course.

I got my code running like this, but it was still getting packets destined for all interfaces. This is where it got tricky and had me tearing my hair out. Some documentation suggests using bind for this, while others suggest using an IOCTL whose name I forget. It's all very unclear. Eventually, bind did the trick. First you have to translate the interface name to an interface id, then use that for the call to bind:

    sockaddr_ll sa;
    ifr.ifr_addr.sa_family = AF_INET;
    strncpy(ifr.ifr_name, sys_name.c_str(), sys_name.size()+1);
    ioctl(my_socket, SIOCGIFINDEX, &ifr);
    sa.sll_ifindex = ifr.ifr_ifindex;
    sa.sll_family = AF_PACKET;
    sa.sll_halen = ETH_ALEN;
    my_mac_address.to_msg(sa.sll_addr);
    bind(my_socket, (sockaddr*)&sa, sizeof(sockaddr_ll));

A couple of wrinkles here. It may seem intuitively obvious that sockaddr_ll is in effect a subclass of sockaddr, but it isn't documented that way anywhere that I could find. And finding the header files that defines these things, and then the header files they depend upon, and so (almost) ad infinitum, is a nightmare. In the end the best solution I could come up with was to run just the preprocessor on my source files, and look at the resulting C code. And note the ugly cast in the call to bind, because in the world of C there is no such thing as inheritance - the common superclass is actually a macro, and as far as the compiler is concerned sockaddr and sockaddr_ll are completely unrelated.

Another wrinkle is the bind function itself. I use boost::bind all the time, far too much to want to type or read the full qualified name, so my common header file contains "using boost::bind". That absolutely wipes out any attempt to use the socket function of the same name. The only way round it is to define a trivial function  called socket_bind (or whatever you prefer), whose definition in its .cpp file studiously avoids using the common header file. It's only a nuisance, but it did take a little thought to come up with a reasonable workaround when I first ran into the problem.

So, with all this done, I was receiving raw ethernet frames, doing my thing with them, and sending them on through the paired egress interface. Wonderful.

Except actually, not. The frames I was receiving were way longer than ethernet frames. Since I'm using jumbo-frame sized buffers (9000 bytes), I'd receive them OK but not be able to send them. But sometimes, they were even too large for that, and I wouldn't receive anything at all. And this was where things got really frustrating.

The first move, of course, was to check the MTUs (maximum permitted frame size under TCP and IP) on all the relevant interfaces. They were fine. Then I found a suggestion that TCP will use the MTU of the loopback interface, relying on the driver to straighten things out. So I set that down to 1400 too. It still made no difference.

At that point, my code didn't send ICMP messages for too-large packets, which a router or host is supposed to do. I spent a whole Saturday in a distress-coding binge writing my ICMP implementation, and changing my super-slick multi-threaded lock-free infrastructure to accommodate it. It did make a very small difference. Instead of just blasting away with giant frames, it would send each packet initially as a giant frame, then retransmit it later in smaller, frame-sized chunks. The data did get through, but at a pititful rate with all those retransmissions and timeouts.

Finally, after much Googling, I discovered the "tcp segmentation offload" parameter. That made no difference. With more Googling, I also discovered "generic segmentation offload". That made things better, though still far from good. I had Wireshark running on all four interfaces - the two test systems running iPerf, and both interfaces on my system in the middle. (All this is running as VMs under VMware, by the way - see earlier rant reasoned discourse about the problems I had trying to get Xen to work). Wireshark clearly showed packets leaving the first system as correctly sized ethernet frames, yet when they showed up at the second system they'd magically coalesced into jumbo frames.

After much cursing I found the third thing I had to turn off, "generic receive offload". The design assumption here is that practically all network traffic is TCP - which after all is true. So the hardware (emulated in my case) combines smaller TCP packets into huge ones, to reduce the amount of work done in the network stack. It's an excellent idea, since much of the overhead of network processing is in handling each packet rather than the actual data bytes. But of course it completely broke my application.

This is not one of the better documented bits of Linux. There is - of course - a utility to manage all this stuff, but it's so obscure that it's not part of the standard Ubuntu distribution. You have to explicitly install it. So a summary of what is required to solve this problem is:

    sudo -s

    apt-get install ethtool

    ethtool -K ethn tso off

    ethtool -K ethn gso off

    ethtool -K ethn gro off

All of this requires root privileges. Whoever wrote ethtool had a sense of humor - the '-K' option sets parameters, the '-k' option (lower case) shows them. It would have been too hard, I suppose, to think of a different letter for such fundamentally different operations.

With that done, my code sees the packets at their normal (no greater than MTU) size. Finally, I could get on with debugging my own code.

Wakkanai - Tales from the Frozen North

A few years ago we visited Hokkaido, taking the train across the island to Abashiri and then driving our rented car all around the southern coast. We spent an especially memorable night in the wind capital of Japan, Erimo Misaki. When I wrote about that trip, I ended with "we're looking forward to returning in winter when everything is covered in snow". And so when Isabelle had a meeting planned in Sapporo in December, we just had to make an adventure out of it.

Since we'd already seen a lot of the southern half of Hokkaido, we decided to go north. The truth is that there isn't much to visit in northern Hokkaido - it's pretty, but extremely empty. We aimed for the northernmost town in Japan, Wakkanai, which is just a few miles from the northernmost accessible place in the whole country, Soya Misaki. It's also the basis of a pun in Japanese, since it sounds very similar to "I don't know" - "Where is this train going?", "Wakkanai". "But you're the driver, surely you must know". Actually, that's probably about the most interesting thing about Wakkanai.

First, though, we had to get there. We flew with JAL from San Francisco to Haneda. This is just so much better than arriving in Narita. First, it's a short subway or monorail ride from the city, or an almost-affordable taxi ride - compared to a long and infrequent train ride or the interminable limo bus (forget a taxi, which will cost at least $250). Second, it just works amazingly well. There are never any lines - for immigration, for check-in, for baggage. If US airports worked even a tenth as well it would be a huge improvement.

The next day we had free time in Tokyo. We had lunch at our favourite kaiten Sushi restaurant, then went for a walk in the neighbourhood of our hotel in Ebisu,  and discovered Shirokanedai Park. This is a nature wilderness smack in the middle of Tokyo - we could hear the Yamanote line trains as we wandered around the gorgeous autumn foliage. Cats and dogs aren't allowed, but nobody had told the ginger cat I caught on camera racing through the undergrowth. Later we had a very pleasant evening with old friends from the days when I used to visit Japan several times a year.

Next morning, return to Haneda. It's the fifth-busiest airport in the world - in the US, only Chicago and Atlanta handle more passengers. Yet it always seems deserted, and once again there were no lines, at all. On takeoff there were fantastic views of Fuji - it was a very clear winter morning. Sapporo had a generous cover of snow, with a lot more falling during the afternoon. The view from our twelfth-floor hotel room of the constantly changing snowfall was magnificent. I ventured outside only to buy the Japan Railways timetable - as on every trip - which turned out later to be a very wise purchase.

Next day Isabelle was busy. I decided to take the train somewhere just for the pleasure of it, and ended up in Noboribetsu on the coast. The name means "special climb", except it doesn't because Kanji are just used for their phonetic value in Hokkaido place names. "Betsu" is actually the Ainu word for river, and appears everywhere - always written as 別 meaning "special".

Noboribetsu is a typical small rural town, which is to say there's absolutely nothing going on there and it has a faintly derelict, mouldy feeling to it. I walked down to the harbour, which was of no interest at all, then found the only place that was open for lunch. It was no surprise to see practically everyone from the restaurant waiting on the platform for the return train to Sapporo.

The next day our train, the Super Soya, departed at 7.48 precisely - Japanese trains always run to time. Well, almost always, as we found later. The train chugs along for five hours, mostly spent on a single track winding through snow-covered fields in narrow valleys as it heads generally northwards towards Wakkanai. The views were magnificent. Finally, after a mountain pass and tiny villages, we saw the sea, and soon after we arrived at Wakkanai station.

Here we had to hurry. Not having a car, the only way to Soya Misaki was by bus. There's exactly one in the afternoon, and it leaves 33 minutes after the train arrives. In that time, we had to get our bags to the hotel, and get back to the station. We made it, with a few minutes to spare - just as well, because the only alternative was a taxi at absolutely astronomical cost (around $150 for the round trip).

The bus trundled along through Wakkanai, all grey and cold and snowy. At the new shopping mall - which seems to be the happening place in town - a group of old ladies, their weekly (monthly?) trip to the big city over, boarded the bus to return to their tiny fishing villages.

Finally we arrived at Soya Misaki, several other people getting off with us. This is the northernmost point in Japan that you can visit - there's a tiny uninhabited island that's further north but you'd need your own boat to get there. The southern tip of Sakhalin, in Russia, is about 40km away. On a clear day you really can see it, or so they say. This day was cold - below freezing, and with a strong breeze. There's very little to see, just a couple of monuments. We looked for the monument to the victims of KAL007 - the plane that in 1983 inadvertently overflew the USSR and got shot down. But we couldn't find it. Only later did we discover that there are several more monuments on top of the hill, a short walk from the shore but not possible in the 25 minutes before the return bus. And you really don't want to miss that, because the next one is another 3 hours.

Once you've scanned the horizon for Sakhalin, and taken each other's pictures in front of the various monuments, the only thing left is to visit the northernmost tacky tourist junk shop in Japan. Every attraction in Japan has these (though there were none in Noboribetsu), selling much the same variety of made-in-China local handicrafts. This one was special, though, as definitively the furthest north in Japan. We bought some postcards and some dried squid, of which more later.

The only thing left now was to wait for the bus. There's a shelter, but it's set back from the road. And we really didn't want to miss it. It was very cold and windy, and the idea of spending another three hours in the gift shop was enough to make me happy to brave the cold. The sight of the bus, when it arrived (precisely punctual of course), was truly the best part of the excursion.

We trundled back to Wakkanai, sharing our dried squid with a couple of Taiwanese girls who were on the same pilgirmage as us. It tasted funny. Later we discovered that this wasn't the classic Japanese snack, which is enjoyable in a chewy and getting-stuck-between-your-teeth kind of way, but an "improved" version made from New Zealand squid soaked in corn syrup. Yuck.

Once back in town, there really wasn't much to do. From our room we had a view over the generously-sized but completely empty harbour. In summer, Wakkanai is the departure point for the much-visited islands of Rebun and Rishiri, but in winter they're covered in snow and nobody goes there. There are also - maybe - boats that visit from Russia.

For dinner, we chose the most famous restaurant in town. Oddly, it's called kuruma-ya, written 車屋, which means "car dealer". We were almost the only people there, but we had an excellent meal including gigantic Hokkaido crab legs and much other seafood.

Our flight next day was at 1.30pm, so to occupy the morning we took a taxi out to Cape Noshabu, the northernmost point in the town itself. There's not much there, so after the obligatory snowy picture, we continued along the coast road. It is very bleak. There are a few houses in the snow, facing out over the cold, grey ocean, surrounded by rickety wooden structures for drying seaweed (kombu). You can see the two islands, grey mountains looming out of the mist and chill.

We still had time when we returned. We discovered the Russian shopping street. Even though the hotel clerk said there are no more tourist boats from Russia, there is this whole street of tourist-type stores, with everything labelled in Russian. Compared to Khabarovsk in Sakhalin, Wakkanai is the sunny south. But the street was almost deserted and looked very sad and run-down. Maybe they run in summer.

Finally it was time to go to the airport. It was sunny when we left, but when we arrived just 20 minutes later there was a blizzard and you could barely see across the runway. We waited, and waited - the display said something about "investigating the weather conditions" which wasn't very encouraging. Finally Isabelle had exhausted the possibilities of the tiny gift shop and we started to go through the security. Just then there was an announcement, and the guard held up his hands in a big "X". What had we done wrong?

It took only a moment to realise that the flight had finally been cancelled. It was very lucky that we hadn't gone airside, because it meant we were in the first few people in the queue at the desk. But what on earth to do now? There's exactly one flight a day to Tokyo, and even if we got on the next day's flight it could just as well be cancelled too. That would be a big problem, since our flight back home was that night. There are a couple of flights to Sapporo, but they're on little commuter turboprops - there were more than enough people waiting to fill the next two flights.

The alternative was the train, again. Luckily Isabelle had retrieved our bags, and with them the JR timetable (moral: never let it out of your hands). I realised we might just make the midday train, which left for Sapporo in half an hour. But how to get back to the town? The taxi rank, outside in the blizzard, was deserted.

Isabelle rushed outside and talked to a guy who had exactly the same reflex as us. Somehow, despite not having a word in common, she communicated with him and agreed to share a taxi, should one arrive. Which it did, in the nick of time. Constantly encouraged by our new friend, the driver went as quickly as he could on the snow and ice covered roads. We arrived at the station seven minutes before the departure time, enough to buy our tickets and see the blessed train arrive.

We had no idea what we'd do once the train took us back to Sapporo, but it was much better to be stuck in Sapporo than Wakkanai! The Tokyo-Sapporo air route is the busiest in the world, so getting back in time for our flight home would not be a problem. JR timetable to the rescue again - it showed that we should easily be in time for the last couple of flights.

This train journey was painful, unlike the previous day. The train was much older, and stunk of diesel fumes. There was no food service, so we survived on a few nuts that we had with us. It was overheated and stuffy, and for much of the trip it was dark outside. But still, we weren't stranded in Wakkanai! And for part of the way, we sat at the front of the train getting a driver's-eye view which is much better than looking sideways. Tiny country stations flashed by, no more than a short platform and a lamppost - yet surely a lifeline to some remote community. The track was completely buried in the snow, only the shiny tops of the rails showing. Visibility was terrible, no more than a couple of hundred yards in the swirling blizzard, and you can certainly appreciate the driver's task, having to slow down for invisible bends, known only through his minute knowledge of every twist and turn in the mountainous track.

One thing we noticed is how polite Japanese trains are. American trains have incredibly loud horns that you can hear from miles away. Japanese trains just have a little high-pitched whistle that goes "pheeeep" - you can barely hear it from inside the train. It's as though the train is saying "shitsurei itashimasu" (the super-polite version of "excuse me") in a squeaky voice just like the shop ladies.

Eventually we arrived in Sapporo. And we were late! This is unknown in Japan, a full quarter hour behind schedule, most likely due to the terrible conditions in the mountains.

Finally we got to the ANA desk at the airport - no queue of course. The lady was very helpful, took all our details, told us there was room on the last flight - and then showed us the price. It was over $1000! I'd been kind of expecting that - I'd managed to call ANA from the train and they weren't very helpful, with our "Explore Japan" tickets which are much cheaper than the published fare. Without much hope, I explained again that we hadn't chosen to travel this way, that the flight had been cancelled and all the rest. There was much "sooouu desu nee" and typing and phone calls. And then - mirabile dictu - she handed us two boarding passes. With my few words of Japanese I'd just saved over $1000!

The plane and then the subway delivered us to Shinagawa station at exactly midnight. We couldn't believe the crowds. Trains departing outbound were so full that people were left standing on the platform. It needed the white-gloved crowd-pushers from the Yamanote line. It was almost impossible to get through all the people to the taxi rank outside. It was the second-last Friday before the Christmas holiday, and evidently Tokyo's salary-men (and women) were partying to the full.

We'd barely eaten all day, so dinner was a priority. When we arrived from the US we'd discovered an Italian wine bar on the wrong side of the tracks at Ebisu station, practically in the catacombs down at street level. We rushed back there. Every few minutes one of us would say, with a sense of wonder, "We're not in Wakkanai!". It was such an incredible relief to be back in Tokyo.

The next day, Saturday, was our final day, but the JAL flight from Haneda doesn't leave until midnight. We went shopping in Ginza, which I haven't done for a very long time. It's an amazing place, in such a Japanese way. The quality of everything on sale is just astounding, whether it's food, household stuff, porcelain... anything. I could have spent the whole day just revelling in the place, admiring things that normally I'd never even pay attention to. I found a little etched brass model of the Toyota FJ, which I just couldn't resist (although it is still sitting wrapped, accusingly, on my desk, waiting to be built). It comes from a lovely series of architectural models, allowing you to build things like a typical Tokyo street scene. And of course I just had to visit Tenshodo, one of the world's greatest model train shops, three tiny floors crammed full of every make in the world (even Hornby from Britain, I wonder who on earth buys that in Japan), and all within a few paces of the Ginza Crossing.

And so, finally, to Haneda, and our flight home. Wakkanakatta.

Ubuntu + Xen = Disaster!

I'm writing this as I reinstall Ubuntu 12.10 on my new test machine. My idea - to test some networking software I'm working on - was to buy a new high-power desktop machine (done), install Ubuntu on it (done), then run Xen to create a simulated network.

Xen is really the open source software from hell. Nothing works the way it's apparently supposed to, and actually in the end nothing works at all. You search around on Google looking for people who've had the same problem, you try the fixes they suggest. Sometimes they work, taking you a few milliseconds further towards failure, other times they have no effect. It's just incredibly frustrating.

So now I'm reinstalling the system and I'll run VMware instead. It's a shame, I liked the idea of Xen, being free and open source and all that. But unless you just have way too much time on your hands, it's hopeless. For me, this whole business of building systems is a distraction from the important matter at hand, which is writing code for my own software. The less time I spend on it, the better.

The first problem was the book I bought. I was on a business trip for a week, so I thought the ten hours or so on the plane, and lonely nights in a hotel, would be the perfect occasion to get myself up to speed. I bought "Running Xen" by Matthews et al. The "et al" turns out to be extremely important. The book is terribly written, by Matthews' entire class of students, none of whom can write very well, and for sure can't write consistently. It never really quite tells you how to do anything, constantly distracting itself with dire warnings about all the bad things that can go wrong. Actually, given my experience with Xen, maybe that's not completely inappropriate.

I bought another book, "The Book of Xen". This is at least well written, and it probably isn't the authors' fault that none of the recipes they give for how to do things actually turn out to work.

Anyway, let's go back to when I didn't know how bad this all was. I installed Xen, made the necessary configuration changes, and rebooted. Woohoo! There I was running a hypervisor. I could type "xm list" and see it, complete with my one and only virtual machine (Dom0 in Xen-speak). So now, all I had to do was follow the recipe from the book, and I'd have my software - which was already running on the bare machine - running on a VM.

Everybody tells you that if you want your VM to use a file as its virtual disk, you absolutely shouldn't use the "loopback driver", you should use something called blktap. So naturally that's what I tried to do. I copied magic incantations into various configuration files, created my virtual disk, and tried to mount it.

Nothing worked. I tried all sorts of combinations of things, and of course I Googled all the error messages and the like. I found lots of suggestions, but nothing that actually helped. That was when the curtain started to open on the fire and brimstone of open source hell. There were loads of responses along the lines of "ah, but you need to install xxx then edit /etc/yyy". But xxx doesn't work on Ubuntu. Google. "xxx doesn't work on Ubuntu, but you can do xxxzzz instead". Well yes, but "xxxzzz" doesn't actually do what "xxx" does. And so on. One article even says "the great thing about Xen is that there are so many different ways to do the same thing." Maybe they had their tongue in their cheek. In any case they were wrong, because the truth is that there are so many ways to fail to do the same thing.

I was about to give up when I suddenly thought to try the dreaded loopback driver, which is supposed not to have good performance. (I've heard that Butler Lampson, one of the great minds in computer science, once said "Performance is a characteristic of a correctly functioning system" - in other words, it doesn't matter what the performance is, if it doesn't actually work. When I had the opportunity to ask him he denied having said it, but it's true anyway).

And... it worked! Replacing "tap:aio:" by "file:" suddenly had everything working correctly. I could mount my virtual device, treat it just like a disk, copy my host system onto it. Now I was ready for the next step, boot the VM with an exact copy of the Ubuntu 12.10 system that was running as the host.

I even tried going back to blktap for the guest system, figuring that maybe there was some kind of conflict with running it in the host. Well, that of course got nowhere. But I edited the config file and... it got another five whole lines further in the console log before failing in a different way.

I Googled the error messages, as you do with open source (open source would just never work at all without Google). The problem was that the root file system wasn't getting mounted. This may (or may not) have something to do with the way Xen deals with booting guests. Instead of doing what the hardware does, and executing files from the guest's disk image, it boots using the host kernel, then somehow flips over to the guest in mid-boot. Why it does this I don't know, but it creates whole chapters in the textbooks explaining how to deal with kernel incompatibilities and the like. And evidently I'd just been bitten by one of these - even though I was using the same system for the host and for the guest.

To cut a long story short, I tried a lot of things - Pygrub, virt-manager - and none of them would work. They all failed in some incomprehensible way, and Googling just produced confusing, conflicting advice, which typically started with "rebuild the kernel..." or "install this completely different toolset".

Yet Xen is in widespread use by cloud hosting companies - for example, by Amazon. I can only suppose that if you have the resources to experiment with different distros, kernel builds, toolsets and configurations, you can eventually find a combination which works. But it certainly isn't for the casual user like me, who just wants to get something working in a day or so. It may be that Ubuntu, or its latest version, is part of the problem. The Ubuntu web pages seem to say it should all work, but there are also a lot of references to things that don't quite work the way they're supposed to under Ubuntu.

I suppose I should have known, really. I worked for a while with a company which had a virtualized software product. They'd started with VMware as the base, and then customer pressure had forced them to port to Xen - just before I joined and found myself responsible for it. It was a nightmare - nothing worked, Citrix (the owners of Xen) were incapable of providing support, and the Xen open source community just laughed when we asked them for advice. "Oh, you're using the xyz toolset - nobody uses that any more, everyone is using pqr. The latest release is pretty good, it mostly works and there are quite a few patches for the stuff that isn't really there yet." The project over-ran by months and only "worked" thanks to numerous hacks and workarounds.

So, here I am reinstalling the machine. I'm annoyed about the time I've spent trying to understand Xen, that I'll now have to spend getting to grips with all the utilities for VMware. But it surely can't be worse... can it?

Tales from the cup shelf #1: Prague

For years now, whenever we travel (often), we try to buy a souvenir mug. Looking at the cup shelf the other day (well, there are three of them actually - I did say we travel often), I realised that collectively they amount to quite a story.

The first one comes from Prague, our first trip there together in 1993. The Communist era was barely over yet the city was already transformed, cleaned up and full of life. Music was everywhere. Walking around, we were constantly being given fliers for evening chamber concerts, while street performers gave excellent impromptu recitals on nearly every corner.

When I was a teenager - no mugs from then I'm afraid - I happened to buy an album of harpsichord recitals by an artist called Zuzana Ruzickova (give or take a few accents). I was enthralled by it, it's by far the best harpsichord album I've ever found. So you can imagine how I felt when one of the fliers was for a performance by none other than Ms Ruzickova herself. And it was extraordinary. Unlike a piano, a harpsichord can only produce notes of one intensity. No matter how gently or hard you hit the key, the sound is always the same. So the only way to make it louder is to play more notes. That's why harpsichord music always has passages with an incredible number of short notes - to make it louder. Just as I was wondering how she could possibly move her fingers that quickly, in a passage composed entirely of demi-semi-quavers, she doubled up again, to hemi-demi-semi quavers. The effect was electrifying.

All of these concerts - we went to others too - were in delightful buildings, older than the music itself, with mysterious inner courtyards and staircases. These led to small rooms - whence chamber music - where you could literally reach out and touch the musicians (not that we did, though I did introduce myself to Ms Ruzickova after the concert, to thank her for such a wonderful introduction to what is often quite frankly a pretty boring instrument).

Of course we visited Wenceslas Square, site of Jan Palach's 1969 self immolation. And the beautiful Vysehrad Park on top of the hill overlooking the Danube, and Prazsky Hrad, the castle high up on the banks. Everyone does. But my memories are of other places, of the quiet cloistered side street leading to the unforgettable Blue Duck restaurant, where we spent a long and largely afternoon-destroying lunch. Or of the tram ride from our hotel, which was in the suburbs, rattling through bustling residential neighbourhoods to the city centre. And then there was the Tatra museum - a whole collection of those odd Czech luxury cars with their rear-mounted, air-cooled V8 engines.

The mugs, the perfect size for after-dinner coffee, were once six, but now only two remain. They've done better than the crystal glasses, though - of the twelve we bought, only one is left, and even that is chipped. (Quite how we brought all this booty back with us on the plane, I do not recollect.)