In 1890, the US Government passed the Sherman Silver Purchase Act. This extraordinarily misguided piece of economics required the government to purchase all available silver at a fixed price, convertible (indirectly) to gold. The immediate and predictable consequence was a boom in silver mining throughout the West, especially in Colorado. The next - equally predictable - consequence was massive over-production of silver and - thanks to the convertibility to gold - depletion of federal gold reserves. The act was repealed in 1893, instantly followed by the collapse of the price of silver and the abandonment of all those mines.
By then, though, construction of railroads to serve all these mines was well under way. In the far southwest corner of Colorado, the Rio Grande Southern connected Durango, via a great deal of beautiful but virtually uninhabited countryside, to Ridgeway. After its brief heyday the line never turned a profit, yet somehow it struggled on despite the total absence of the traffic it was built to serve. The distances were huge and roads non-existent, so the small amount of farming and what little mining was left just about paid for the few trains that ran.
By the 1920s, road competition was starting to hurt. The line superintendent had the brilliant idea of taking some second hand Buick and Pierce Arrow automobiles, and converting them to railcars. They were big enough to carry the tiny passenger load and, more important, handle the mail contract and the parcel traffic in a box-car tacked on to the back of the car chassis. Thus was born the Galloping Goose, and thereafter steam was used only for special occasions, when the traffic justified longer trains.
Seven Geese were built altogether, though only five carried passengers and normal freight. They rattled their noisy and smelly way through the deserted Colorado back country for two decades, saving the line from oblivion in the Great Depression of the 1930s. Finally in 1949 the railway lost the mail contract. In desperation, the freight car portion was opened up and converted to panoramic passenger accommodation. But it was no good, and the line finally closed completely in 1952.
By then, though, the Rio Grande Southern and its big brother, the Denver and Rio Grande Western were known to every railroad enthusiast in the country. All of the Geese were saved, spending decades as static ornaments in the little towns they served until, in the last ten years or so, they have been restored to running order. So now you can travel on each and every one of these unique (and smelly and noisy) vehicles, surely the only class of locomotive for which this is true.
When I built my garden railway, I always wanted to have one of these. Accucraft, a boutique builder of large-scale rolling stock, made several different Geese for a while about ten years ago. They are unobtainable now, but I kept an eye on eBay and after a few weeks a model of No 7, the final Goose, showed up at home. She's in beautiful condition, hardly ever operated I'd say, and as a bonus, had been expertly fitted with a Soundtraxx sound card, compete with authentic Goose sound effects. She still had all the documentation and instructions from the conversion. I tried her out with simple DC input, and she ran perfectly.
Unfortunately, though, I had to put a stop to that. My layout uses DCC, which is to say computer-based train control. Every locomotive requires a decoder, a tiny miracle of modern electronics that turns 18V AC on the track, with a superimposed control protocol, into DC to operate the motors and all the accessories. For modern locos, it's just a question of plugging in a decoder, and everything just works. Or so they say - I've never converted one of these, so I wouldn't know.
For older locos, it's somewhere between tricky and a nightmare. The biggest problem is that the rails are, obviously, connected directly to the motor. You have to figure out a way to break this connection - 100% reliably. Older LGB locos are particularly difficult, since the connection is direct between the pickup system and the motor. There is no wire that can conveniently be cut. The decoders I mostly use (NCE D408SR) are fairly robust - except that even a fleeting contact between any of the motor or accessory outputs, and the track voltage, will toast them in a heartbeat. My worst experience was with my LGB track cleaner, where a failed piece of insulation toasted the decoder, all the other electronics in the loco, and in the ensuing meltdown took out some of the control circuits for the layout.
In this case, though, the motor and pickup wiring was easy to separate. Then there is the big decision: to do a "full" DCC installation, with everything like lights and sound running of separate decoder accessory outputs; or a "lightweight" installation, just connecting everything to the motor output. The latter is easier but means that lights and the rest can only work when the loco is moving, which I find a lot less satisfactory. In this case it was fairly easy to rewire it for a full installation.
A sound card makes it much harder. The Phoenix uses the track supply both for its own power, and as a control signal to tell it what noise to make - not only the speed of the train, but other Goose sounds like changing gear. It has a rechargeable battery, to supply power when it isn't moving, but lead-acid batteries don't like being left uncharged and I think mine has long since gone to the great battery charger in the sky. There is a separate charger circuit, though, and the obvious thing to do is connect this to the track pickup, using the motor output only for control. So that is what I did.
Big mistake. When I connected everything up, the sound card operated even when it shouldn't. By the time I finally realised that I'd just arranged a sneak feed between the track and the motor output, it was too late. Toasted decoder. Removing the connections to the sound card and installing a brand new decoder confirmed this. Thank goodness for NCE's "no questions asked" $25 replacement warranty.
A sound card has inputs that make it whistle, toot, clang bells and so on by command. As delivered, my Goose had reed switches underneath that could be operated by fixed magnets in the track. This isn't very practical but with DCC it's obvious to run these from the decoder. Obvious, but not easy. There is no common neutral between the sound card and the decoder, so the operation has to be electrically isolated. This means using either opto-couplers or relays. As long as space is available, I prefer relays - they're not polarity sensitive, and generally more robust.
I ended up building an auxiliary card on a small piece of stripboard with three relays: one each for the horn and bell, and a two-pole relay to isolate the sound card completely. Sound can be very atmospheric, but there are times when you'd just like the trains to look good, chugging along in silence. It also has the 6 volt regulators that run the lights - the decoder produces 18V, which is way too much.
There remained the problem of providing independent power to the sound card. $10 on eBay or so will buy an isolated DC-DC converter, taking the decoder voltage as input and generating 12 totally isolated volts as output. End of sneak path.
Finally, after several weeks of sitting on my bench, Goose #7 is ready for service, with controllable lights and thoroughly Goose-like sound effects. The picture below shows everything that hides inside the freight car. Its a good job there's plenty of room!
Odd thoughts about flying, aerobatics, software engineering and other things that cross my mind.
Monday, 3 November 2014
Monday, 20 October 2014
Cisco - their future?
(I wrote this over three years ago, when Cisco first started to show serious signs of losing the plot. They seemed to make a miraculous recovery, but now things look grimmer than ever, and my conclusion remains as valid now as it was then).
----------------------------
Cisco's recent woes will be no surprise to those who know the company from the inside. The only surprise is how long it has taken. I was surprised, though, by the naivety of some of the analysts' remarks, and in particular the apparently universal view that all Cisco needs to do is get rid of a few dubious acquisitions like Flip, for everything to be restored to its rosy past. One I read said, "The fall in Cisco's gross margin from 67% to 50% is due to the lower margin on consumer products. Disposing of these is a necessary step to restoring margins." Or something like that anyway.
The simplest of arithmetic shows that this cannot be true. Linksys is by far the largest part of the consumer portfolio, but it is still well under 5% of revenue. The others are just noise. Even if all these business ran at zero margin, the total effect on overall margin would be a couple of percentage points. So it can't be that.
Cisco has been trapped in an awkward spot for a long time, and the consumer business is just one aspect of a well-motivated attempt to get out of it. When you have a near-100% market share, there is just nowhere else to go. Well over half of large enterprises in the US still furnish their entire network infrastructure more-or-less unquestioningly from Cisco. Practically all of Cisco's actual profit comes from the switches and routers that are to be found in nearly every wiring closet and data center in the western world. Everything else is funded out of this apparently inexhaustible gravy train. Growing this to a $40B business has been an amazing achievement, but it's not easy to find another one like it, and this is no longer a rapidly growing sector as it was a decade ago. Hence the moves to the consumer, the data center and others.
But nothing lasts for ever. The core functions of enterprise networks, exemplified more than anything by the Catalyst 6500 family, are already a commodity. For well under $500, Broadcom or Marvell will sell you a chip that does everything you actually need from the Cisco product. They'll even give you complete designs, ready to be sent to a board shop. The days are long gone when Cisco's added value lay in the extraordinary breadth of IOS's support for obscure proprietary protocols. Networks now are all-IP - as indeed John Chambers has been saying. It's only inertia, combined with quite a lot of Fear, Uncertainty and Doubt, that makes people stick with Cisco's high-margin products. And just to twist the knife, the maturity of WiFi means that the enterprise is - finally - moving away from the all-wired model that puts multiple ethernet jacks in every office and cube. That alone has the potential to demolish Cisco's profitability. Worse, competition from HP, Huawei, and white-box vendors - none of whom have Cisco's cost structure or grossly inflated overheads - is driving down the margin.
Let me expand a little on the "grossly inflated overheads". Cisco's development organization functions like 18th century Europe or India - lots of principalities with headstrong leadership, who (mostly) do not compete with each other for territory, but have no intention of sharing anything either. That's why every Cisco product family has its own system architecture, its own ASIC development, and - increasingly - its own software. (Time was when everything ran IOS, which was reasonably common across all platforms, but now IOS has no future relevance for most of them. Only the name, IOS-something-or-other, remains common to what are in fact totally different software systems). The good thing about this strategy is that it avoids the "one egg, one basket" problem that for example brought down DEC in the 90s. The bad thing is that it is hugely expensive, and tends to get worse and worse - it's rare that divergence returns naturally to convergent paths.
This has been a deliberate strategy on Chambers' part. The last time Cisco had any real cohesive technical strategy was under Ed Kozel, in the late 90s. Since then the CTO has been paid to not interfere in the operation of the individual principalities. Charlie Giancarlo seemed to be looking for some kind of unification during his brief reign, and for his pains - ahem - decided to seek new opportunities. Since then there hasn't even been the pretence of any overall leadership of product development, just a "council" whose role is to rubber-stamp whatever each group was doing anyway.
So what can Cisco do? The core business is moving to a commodity, and shrinking anyway. Numerous attempts to move out of this area have met, at best, moderate success (by Cisco standards that is - even the failures make more revenue than many mid-sized companies, but they aren't profitable). What does Cisco really have? The answer, of course, is a vast and mostly loyal customer base. Cisco as a vendor and integrator still has huge strengths - in particular, its (entirely justified) reputation for digging customers out of the dirt no matter how they got there. It's still true, in the enterprise at least - as it once was for IBM - that "nobody gets fired for buying from Cisco".
So Cisco needs to make the same transformation that IBM did 20 years ago under Lou Gerstner, from being primarily a product company to being primarily a service company. And, in the process, to realise that practically all of its product development is context, not core. That will be very tough to accept - it means questioning everything, even the core switch business, not just closing down Flip. It will take a while to happen, and the risk in getting from here to there is huge.
----------------------------
Cisco's recent woes will be no surprise to those who know the company from the inside. The only surprise is how long it has taken. I was surprised, though, by the naivety of some of the analysts' remarks, and in particular the apparently universal view that all Cisco needs to do is get rid of a few dubious acquisitions like Flip, for everything to be restored to its rosy past. One I read said, "The fall in Cisco's gross margin from 67% to 50% is due to the lower margin on consumer products. Disposing of these is a necessary step to restoring margins." Or something like that anyway.
The simplest of arithmetic shows that this cannot be true. Linksys is by far the largest part of the consumer portfolio, but it is still well under 5% of revenue. The others are just noise. Even if all these business ran at zero margin, the total effect on overall margin would be a couple of percentage points. So it can't be that.
Cisco has been trapped in an awkward spot for a long time, and the consumer business is just one aspect of a well-motivated attempt to get out of it. When you have a near-100% market share, there is just nowhere else to go. Well over half of large enterprises in the US still furnish their entire network infrastructure more-or-less unquestioningly from Cisco. Practically all of Cisco's actual profit comes from the switches and routers that are to be found in nearly every wiring closet and data center in the western world. Everything else is funded out of this apparently inexhaustible gravy train. Growing this to a $40B business has been an amazing achievement, but it's not easy to find another one like it, and this is no longer a rapidly growing sector as it was a decade ago. Hence the moves to the consumer, the data center and others.
But nothing lasts for ever. The core functions of enterprise networks, exemplified more than anything by the Catalyst 6500 family, are already a commodity. For well under $500, Broadcom or Marvell will sell you a chip that does everything you actually need from the Cisco product. They'll even give you complete designs, ready to be sent to a board shop. The days are long gone when Cisco's added value lay in the extraordinary breadth of IOS's support for obscure proprietary protocols. Networks now are all-IP - as indeed John Chambers has been saying. It's only inertia, combined with quite a lot of Fear, Uncertainty and Doubt, that makes people stick with Cisco's high-margin products. And just to twist the knife, the maturity of WiFi means that the enterprise is - finally - moving away from the all-wired model that puts multiple ethernet jacks in every office and cube. That alone has the potential to demolish Cisco's profitability. Worse, competition from HP, Huawei, and white-box vendors - none of whom have Cisco's cost structure or grossly inflated overheads - is driving down the margin.
Let me expand a little on the "grossly inflated overheads". Cisco's development organization functions like 18th century Europe or India - lots of principalities with headstrong leadership, who (mostly) do not compete with each other for territory, but have no intention of sharing anything either. That's why every Cisco product family has its own system architecture, its own ASIC development, and - increasingly - its own software. (Time was when everything ran IOS, which was reasonably common across all platforms, but now IOS has no future relevance for most of them. Only the name, IOS-something-or-other, remains common to what are in fact totally different software systems). The good thing about this strategy is that it avoids the "one egg, one basket" problem that for example brought down DEC in the 90s. The bad thing is that it is hugely expensive, and tends to get worse and worse - it's rare that divergence returns naturally to convergent paths.
This has been a deliberate strategy on Chambers' part. The last time Cisco had any real cohesive technical strategy was under Ed Kozel, in the late 90s. Since then the CTO has been paid to not interfere in the operation of the individual principalities. Charlie Giancarlo seemed to be looking for some kind of unification during his brief reign, and for his pains - ahem - decided to seek new opportunities. Since then there hasn't even been the pretence of any overall leadership of product development, just a "council" whose role is to rubber-stamp whatever each group was doing anyway.
So what can Cisco do? The core business is moving to a commodity, and shrinking anyway. Numerous attempts to move out of this area have met, at best, moderate success (by Cisco standards that is - even the failures make more revenue than many mid-sized companies, but they aren't profitable). What does Cisco really have? The answer, of course, is a vast and mostly loyal customer base. Cisco as a vendor and integrator still has huge strengths - in particular, its (entirely justified) reputation for digging customers out of the dirt no matter how they got there. It's still true, in the enterprise at least - as it once was for IBM - that "nobody gets fired for buying from Cisco".
So Cisco needs to make the same transformation that IBM did 20 years ago under Lou Gerstner, from being primarily a product company to being primarily a service company. And, in the process, to realise that practically all of its product development is context, not core. That will be very tough to accept - it means questioning everything, even the core switch business, not just closing down Flip. It will take a while to happen, and the risk in getting from here to there is huge.
Friday, 17 October 2014
Cisco - and the end of an era
This week has seen the close of the final chapter of my involvement with Cisco. That may sound strange, considering that I left the company eight years ago. But on Monday, they suddenly closed down the team in the UK that I built 15 years ago. They were probably the best software team in the whole company - and if that sounds like an overblown claim, I'll justify it later. So it's ironic that this happens as Cisco says they need to invest massively in software and hire thousands of people. But these things happen in large companies.
I joined Cisco in 1999, to run their almost-new UK development team. It was about 25 people then, and within a year we'd increased it to nearly 100. Hiring in Silicon Valley was crazy and the company was desperate to get engineers on board. An anomaly in the way the European and US HR systems worked together (or didn't) meant I hired all those people with (as I recall) just three requisitions.
My own recruitment process had been strange. The guy who started the UK team had decided to retire, and was looking for a replacement. He got my name from a couple of former colleagues. The discussion went something like, "I need someone to take over the team, and X and Y said you're the right person for the job. When can you start?" It wasn't quite that simple, though, because a bunch of people at HQ in San Jose had to approve me as well. So I flew over and met them, and they all said "[UK manager] says we should hire you, what will it take to get you on board?" Well, one person out of the half a dozen did actually ask me a few interview-style questions. And I met the VP of the group - my boss's boss - but he'd been fired by the time I started, a few weeks later. As I discovered later, that was an indication of the way things work at Cisco.
I knew about half of my new team already. The network software development community in the UK is small, and several of us worked together at DEC's network team, that had dissipated a few years earlier. Hiring consisted of sitting down with the team every couple of weeks and running through the list of people from our past lives who we should call. Given the prestige associated with Cisco back then, most of the ones we wanted ended up joining us. Actually I think they all did.
Half the team was just outside London, about a mile north of Heathrow. From my office window I could see, and even hear, Concorde taking off every morning at 11. The other half was in Edinburgh, where they had an office as unlike the boring Cisco norm as could be - a converted whisky warehouse down by the former docks in Leith. Directly opposite was the imposing modern montrosity of the Scottish Office. Around the corner was the Malmaison Hotel, where I stayed every time I visited. The converted seaman's hostel housed not only a very classy and comfortable hotel (just writing about it brings to mind its comfortingly luxurious rooms with their huge, soft white-sheeted beds), but also an excellent restaurant.
Leith wasn't always a classy place - 50 years ago you definitely would not go there at night. Even now it hovers between being an 'in' place and hideous 1960s tower blocks. But if you need to forget, the Malt Whisky Society is a short walk from the office - and we had a corporate membership. You could sip cask-strength (50%) spirit at very reasonable prices. I do mean sip, at this strength it's almost undrinkable, and the next day's hangover can be a problem too.
For the first couple of years, Cisco was rich. Who could forget the weekend in Zermatt, staying at a Leading Hotel of the World, spouses invited too, the agenda carefully planned so that most of the day could be spent on the slopes. (And during the rare moments of work, my introduction to MPLS Multicast, a topic of immensely arcane complexity). For two years we held our office Christmas party at Gleneagles, a spectacular 1930s Scottish hotel famed for its golf course and its gothic opulence.
I said earlier that this was the best software team in Cisco. We'd carefully selected its members from the finest of all the people we'd worked with before. I was the 29th person there, and within a year we'd grown it to 80 people. Not everyone in a team that size can truly be an A+ player, whatever Google would have you believe. But we had way more than our fair share. In San Jose, Cisco was suffering terrible attrition. The week I joined, my boss was too busy to speak to me because two leading lights - famous names in the industry - had quit on the same day, to go to classic startups and make their fortune. But in the UK, Cisco was the best show in town. During the 15 years of the group's life, not a single one of the A+ players left - for the simple reason that there was nowhere else for them to go.
Starting a remote group is hard. Its founder, my predecessor, had scraped together whatever unloved, unwanted project remnants he could get. Yet within a couple of years we had flagship projects, like Cisco's IPv6 implementation and the high-speed packet forwarding path, CEF, simply because all the people in San Jose who knew these topics had left for greener pastures.
It wasn't all roses, though. Cisco IOS (never iOS!) was the heart of all its most successful and profitable products, and the maintenance burden was massive. At one point we calculated that we were maintaining about 750 distinct versions of the code. At the same time there were two huge (and as it turned out misguided and ultimately unsuccessful) infrastructure projects going on, which each required major disruptive change throughout the entire code base. At my very last all-hands meeting before I left Cisco, one developer summarised it neatly in a so-called question: "We spend nearly all our time fixing bugs and committing them into dozens of subtly different branches, and the rare actual development we do is forced upon us by projects that are nothing to do with us. We never get to do anything for the benefit of the stuff we actually own. Isn't something seriously wrong?" Well, I was their VP, I could hardly agree with them. Not openly anyway, though it was exactly what I thought too.
But I'm getting ahead of myself. About 18 months after I joined, there was a huge shakeup in senior management in San Jose (not an uncommon event). I was asked to move to the US and take on leadership of the whole of IOS (well, the most interesting parts anyway). It was the job of my dreams. It would have been impossible to refuse, even though it meant giving up our much-loved house in France, a house we'd never expected to move away from. (During all the time I was working for Cisco in the UK I was commuting every Monday and Friday from the south of France, staying in my flat in London during the week. Sometimes tough, but not without its advantages).
Our traumatic move from Valbonne to California took place in June 2001. For four years I did what I'd been told was wanted, injecting leadership into the group, encouraging good ideas, defending my team (nearly 500 people) from the bureaucratic madness of a giant company, trying to juggle the conflicting demands of a dozen hardware fiefdoms. We made some pretty cool stuff happen. Then it slowly dawned on me that actually this was the last thing anyone wanted me to do. What they really wanted the IOS team to do was fix bugs and keep its mouth shut. There were some serious egos in Cisco (still are, though mostly not the same ones) and they were the ones in charge. It didn't help that we'd had our own management turmoil. The protective, supportive and incredibly smart manager who'd brought me to the US had been replaced by a complete idiot whose preoccupation (and sole talent) was brown-nosing all he could, and trying to keep his job. The only good thing he ever did for Cisco - but it was really good - was to take a senior position at one of their major competitors, where he single-handedly demolished their entire core software base.
It was very far from an enjoyable time. My last year at Cisco was the most miserable of my professional life. Finally I took a job at a startup, and started to enjoy going to work again. At Cisco, I felt that the 20% of my job that was enjoyable made up for the 80% that wasn't, an endless succession of political meetings where nothing was ever decided. In the startup world, I enjoy my job 99% of the time.
When I moved to the US, the UK team remained part of my bigger group. I visited them three or four times a year. I also started a development team in Tokyo, but that's definitely another story.
It's now been longer since I left Cisco, than the time I spent working there. The bitterness I felt when I left has long faded. At DEC I had the good fortune to work, briefly, for one of the giants of the computer industry, Alan Kotok. He explained very well: "Everything that's not at corporate HQ is attached by a giant rubber band, held in place by a fragile little pin. At the slightest disturbance the pin will shake loose and the band will twang back to HQ." And that, sadly, is what happened to my UK team that had been so carefully built and nurtured. It was a huge shock to learn that they were being jettisoned, even though they could have been of such huge value to the company as it seeks its troubled and uncertain pathway in the future of networking.
I joined Cisco in 1999, to run their almost-new UK development team. It was about 25 people then, and within a year we'd increased it to nearly 100. Hiring in Silicon Valley was crazy and the company was desperate to get engineers on board. An anomaly in the way the European and US HR systems worked together (or didn't) meant I hired all those people with (as I recall) just three requisitions.
My own recruitment process had been strange. The guy who started the UK team had decided to retire, and was looking for a replacement. He got my name from a couple of former colleagues. The discussion went something like, "I need someone to take over the team, and X and Y said you're the right person for the job. When can you start?" It wasn't quite that simple, though, because a bunch of people at HQ in San Jose had to approve me as well. So I flew over and met them, and they all said "[UK manager] says we should hire you, what will it take to get you on board?" Well, one person out of the half a dozen did actually ask me a few interview-style questions. And I met the VP of the group - my boss's boss - but he'd been fired by the time I started, a few weeks later. As I discovered later, that was an indication of the way things work at Cisco.
I knew about half of my new team already. The network software development community in the UK is small, and several of us worked together at DEC's network team, that had dissipated a few years earlier. Hiring consisted of sitting down with the team every couple of weeks and running through the list of people from our past lives who we should call. Given the prestige associated with Cisco back then, most of the ones we wanted ended up joining us. Actually I think they all did.
Half the team was just outside London, about a mile north of Heathrow. From my office window I could see, and even hear, Concorde taking off every morning at 11. The other half was in Edinburgh, where they had an office as unlike the boring Cisco norm as could be - a converted whisky warehouse down by the former docks in Leith. Directly opposite was the imposing modern montrosity of the Scottish Office. Around the corner was the Malmaison Hotel, where I stayed every time I visited. The converted seaman's hostel housed not only a very classy and comfortable hotel (just writing about it brings to mind its comfortingly luxurious rooms with their huge, soft white-sheeted beds), but also an excellent restaurant.
Leith wasn't always a classy place - 50 years ago you definitely would not go there at night. Even now it hovers between being an 'in' place and hideous 1960s tower blocks. But if you need to forget, the Malt Whisky Society is a short walk from the office - and we had a corporate membership. You could sip cask-strength (50%) spirit at very reasonable prices. I do mean sip, at this strength it's almost undrinkable, and the next day's hangover can be a problem too.
For the first couple of years, Cisco was rich. Who could forget the weekend in Zermatt, staying at a Leading Hotel of the World, spouses invited too, the agenda carefully planned so that most of the day could be spent on the slopes. (And during the rare moments of work, my introduction to MPLS Multicast, a topic of immensely arcane complexity). For two years we held our office Christmas party at Gleneagles, a spectacular 1930s Scottish hotel famed for its golf course and its gothic opulence.
I said earlier that this was the best software team in Cisco. We'd carefully selected its members from the finest of all the people we'd worked with before. I was the 29th person there, and within a year we'd grown it to 80 people. Not everyone in a team that size can truly be an A+ player, whatever Google would have you believe. But we had way more than our fair share. In San Jose, Cisco was suffering terrible attrition. The week I joined, my boss was too busy to speak to me because two leading lights - famous names in the industry - had quit on the same day, to go to classic startups and make their fortune. But in the UK, Cisco was the best show in town. During the 15 years of the group's life, not a single one of the A+ players left - for the simple reason that there was nowhere else for them to go.
Starting a remote group is hard. Its founder, my predecessor, had scraped together whatever unloved, unwanted project remnants he could get. Yet within a couple of years we had flagship projects, like Cisco's IPv6 implementation and the high-speed packet forwarding path, CEF, simply because all the people in San Jose who knew these topics had left for greener pastures.
It wasn't all roses, though. Cisco IOS (never iOS!) was the heart of all its most successful and profitable products, and the maintenance burden was massive. At one point we calculated that we were maintaining about 750 distinct versions of the code. At the same time there were two huge (and as it turned out misguided and ultimately unsuccessful) infrastructure projects going on, which each required major disruptive change throughout the entire code base. At my very last all-hands meeting before I left Cisco, one developer summarised it neatly in a so-called question: "We spend nearly all our time fixing bugs and committing them into dozens of subtly different branches, and the rare actual development we do is forced upon us by projects that are nothing to do with us. We never get to do anything for the benefit of the stuff we actually own. Isn't something seriously wrong?" Well, I was their VP, I could hardly agree with them. Not openly anyway, though it was exactly what I thought too.
But I'm getting ahead of myself. About 18 months after I joined, there was a huge shakeup in senior management in San Jose (not an uncommon event). I was asked to move to the US and take on leadership of the whole of IOS (well, the most interesting parts anyway). It was the job of my dreams. It would have been impossible to refuse, even though it meant giving up our much-loved house in France, a house we'd never expected to move away from. (During all the time I was working for Cisco in the UK I was commuting every Monday and Friday from the south of France, staying in my flat in London during the week. Sometimes tough, but not without its advantages).
Our traumatic move from Valbonne to California took place in June 2001. For four years I did what I'd been told was wanted, injecting leadership into the group, encouraging good ideas, defending my team (nearly 500 people) from the bureaucratic madness of a giant company, trying to juggle the conflicting demands of a dozen hardware fiefdoms. We made some pretty cool stuff happen. Then it slowly dawned on me that actually this was the last thing anyone wanted me to do. What they really wanted the IOS team to do was fix bugs and keep its mouth shut. There were some serious egos in Cisco (still are, though mostly not the same ones) and they were the ones in charge. It didn't help that we'd had our own management turmoil. The protective, supportive and incredibly smart manager who'd brought me to the US had been replaced by a complete idiot whose preoccupation (and sole talent) was brown-nosing all he could, and trying to keep his job. The only good thing he ever did for Cisco - but it was really good - was to take a senior position at one of their major competitors, where he single-handedly demolished their entire core software base.
It was very far from an enjoyable time. My last year at Cisco was the most miserable of my professional life. Finally I took a job at a startup, and started to enjoy going to work again. At Cisco, I felt that the 20% of my job that was enjoyable made up for the 80% that wasn't, an endless succession of political meetings where nothing was ever decided. In the startup world, I enjoy my job 99% of the time.
When I moved to the US, the UK team remained part of my bigger group. I visited them three or four times a year. I also started a development team in Tokyo, but that's definitely another story.
It's now been longer since I left Cisco, than the time I spent working there. The bitterness I felt when I left has long faded. At DEC I had the good fortune to work, briefly, for one of the giants of the computer industry, Alan Kotok. He explained very well: "Everything that's not at corporate HQ is attached by a giant rubber band, held in place by a fragile little pin. At the slightest disturbance the pin will shake loose and the band will twang back to HQ." And that, sadly, is what happened to my UK team that had been so carefully built and nurtured. It was a huge shock to learn that they were being jettisoned, even though they could have been of such huge value to the company as it seeks its troubled and uncertain pathway in the future of networking.
Monday, 23 June 2014
The Garden Railway
One summer when I lived in France, I started building a simple garden railway. It was just a large oval of LGB track with some loop sidings, and a couple of trains. With my son, we automated it using the LGB train detectors and some creative wiring. It was fun to build, and watch the trains going round while drinking the first Pernod on the terrace. But soon after that it got packed up with everything else when we moved to California.
Soon after we moved to our present house, I noticed that one of our neighbours had some LGB track in their garden. Talking with him inspired me to build a very small layout in the enclosed atrium of our Eichler, which is about 15 feet square. This time I automated it using a computer, with the JMRI control system. I had three trains that took turns to go round, but in the tiny space (for LGB) it wasn't all that interesting, and it was in the way too. It didn't last long.
Our garden is big enough for a decent sized railway, but I could never figure out how to make it compatible with having an actual garden. Then my daughter decided to visit, with her son. At three years old, he's completely obsessed with Thomas the Tank Engine and anything and everything to do with trains - for a while his favourite video was a documentary about the construction of the French TGV network, though he surely understood none of it.
He won't actually be here until nearly a year from now, but it was a good excuse to get started. We suddenly realised that if we used the concrete pool surround, I could create a permanent layout. As a bonus, it is relatively flat which would make it much easier to keep the trains on the rails.
I took stock of the LGB track sections which had been gathering dust in the garage for over a decade. I had enough to build along two sides of the pool - a total of about 60 feet - with a reversing loop at each end, making a kind of dogbone suitable for continuous running. On one side of the pool there's enough space to put several parallel sidings, so different trains can be run at the same time.
I also bought a Bachmann Thomas set, with Thomas and his two coaches, Annie and Clarabel, while on eBay I found the coal truck labelled "S. C. Ruffey" (geddit? - he was quite a card, that Reverend Awdry).
The first attempt had one of the loops on a corner of the lawn. I quickly realised this was a bad idea. The LGB trains could just about cope, but Thomas his coaches certainly couldn't. A quick extension along the third side of the pool gave access to a big concrete area, solving that problem.
When I was a lad and used to drool over Hornby-Dublo layouts in the Meccano Magazine, they always had reverse loops in them. There was a reason for that: because they could. On a three-rail system, like Hornby-Dublo or Marklin, there is no electrical problem. But in a two-rail system, like LGB, the two rails are the positive and negative sides of the power supply to the trains. A quick sketch will show that a reverse loop inevitably connects them together, creating a short circuit. Many and varied are the ingenious solutions to this problem. After several false starts, I eventually built a little control box for each end of the layout, using the LGB train detectors to operate some relays that switch things around as required.
In the process, I discovered some limitations of the LGB system. The trains are superb, and the track is indestructible. But the control system for points has some odd idiosyncrasies. The detectors can in theory operate the points directly, but they contain wimpy little magnetic reed switches, rated at just half an amp. It doesn't take much to toast them. They get red hot, which fuses the contacts together - guaranteeing instant destruction. Luckily it isn't hard to replace the failed switches (just as well, because like all LGB parts, they're eye-wateringly expensive).
The real answer is to use them to drive only very light loads, like relays or computer inputs. Another problem is the electrical switches that you can arrange to be changed as the points move. I was relying on these to change the polarity of my reverse loops - until I found that often one would switch and not the other, creating a short circuit across the power supply. More relays were the answer, and the LGB switches are back in the parts box.
The electrical control of the trains themselves uses the DCC system. In the old days, the rails carried the actual electric voltage for the motor in the locomotive. This was very restrictive, and made it very complex to be able to run more than one train at a time, or even have more than one train on the track. DCC avoids this by providing a permanent voltage, overlaid with a control signal that tells each train individually what to do. As a bonus, it's easy to interface to a computer.
The ultimate goal of all this is to make it run completely automatically. In parallel with building the railway itself, I've been experimenting with using an Arduino to run the control system. It's coming along nicely, more on that soon I hope.
Today was the first day that I've run the full layout, with several trains. I have a Bachmann Shay which I added sound to and converted to DCC back in France - very painfully. It pulls a train of eight or so log and other freight wagons. There are a couple of LGB European locomotives, and of course Thomas with Annie, Clarabel and Mr Scruffey. Control for the moment is still manual, using a virtual control panel built with JMRI. There are still hours of endless fun to be had - and quite a few minutes of frustration, no doubt.
Tuesday, 6 May 2014
My Dad, Trains and his Signal
I should have realized how fascinated my Dad was with trains at an early age, but somehow I didn't.
When I was tiny - just four years old - he would ask kindly engine drivers at Liverpool Street station to show me round the cabs of their giant steam engines - Britannia and others - that pulled the express trains that would roar through our local station. The vast expanse - as it seemed to a small boy - and the roaring heat of the firebox are impressions I've never forgotten.
Or he'd take me to a nearby bridge to watch the trains go by. He bought me my first trainset when I was five. It was clockwork, with an oval of O-gauge track and two tinplate carriages. It just went round and round but I loved it. I moved up to an electric train - Triang OO gauge - on my seventh birthday. It was a success, because I have loved everything to do with trains ever since.
But I never realized how deep his own interest was until much later. After he retired my Dad would often call phone-in radio programmes, and it was to one of those that he told his own childhood railway story. Money was scarce back then, and toy trains were expensive even for the people who had money. He would have loved one but it was unimaginable. But one Christmas he got as a present a toy railway signal. Just the signal, nothing else - no track, no trains, just the signal.
Signals then were more than just lights. They had a long red arm which moved up and down, its position telling the train driver whether he could proceed or not. Down and horizontal, it meant stop. Up at an angle, meant go. At night, red and green lights shone through colored glass. It was operated by a signalman, in a signal box possibly a long way off. It was his sheer muscle power, transmitted via up to a mile of thick steel cable, that made it move.
For my Dad, the rest of the railway lay just in his imagination. A train was coming, the signal must be pulled to show clear. The train rushes by in a cloud of steam and smoke, the noise deafening, loaded goods wagons banging and bouncing over the joints in the track. Then it has gone leaving only a cloud of dust. The signal must be set to danger behind the train. But wait, there's another one coming, a passenger train, the express. The gleaming engine rushes past, passenger's faces peer out of the varnished carriages. Again the signal must be pulled off, the signalman putting all his weight behind the lever as the heavy arm rises.
But all this was only in his imagination. In reality there was just a small boy and a toy signal, as he moved the arm up and down. After he had told his story on the radio, other people called in to say he'd reduced them to tears.
When I was tiny - just four years old - he would ask kindly engine drivers at Liverpool Street station to show me round the cabs of their giant steam engines - Britannia and others - that pulled the express trains that would roar through our local station. The vast expanse - as it seemed to a small boy - and the roaring heat of the firebox are impressions I've never forgotten.
Or he'd take me to a nearby bridge to watch the trains go by. He bought me my first trainset when I was five. It was clockwork, with an oval of O-gauge track and two tinplate carriages. It just went round and round but I loved it. I moved up to an electric train - Triang OO gauge - on my seventh birthday. It was a success, because I have loved everything to do with trains ever since.
But I never realized how deep his own interest was until much later. After he retired my Dad would often call phone-in radio programmes, and it was to one of those that he told his own childhood railway story. Money was scarce back then, and toy trains were expensive even for the people who had money. He would have loved one but it was unimaginable. But one Christmas he got as a present a toy railway signal. Just the signal, nothing else - no track, no trains, just the signal.
Signals then were more than just lights. They had a long red arm which moved up and down, its position telling the train driver whether he could proceed or not. Down and horizontal, it meant stop. Up at an angle, meant go. At night, red and green lights shone through colored glass. It was operated by a signalman, in a signal box possibly a long way off. It was his sheer muscle power, transmitted via up to a mile of thick steel cable, that made it move.
For my Dad, the rest of the railway lay just in his imagination. A train was coming, the signal must be pulled to show clear. The train rushes by in a cloud of steam and smoke, the noise deafening, loaded goods wagons banging and bouncing over the joints in the track. Then it has gone leaving only a cloud of dust. The signal must be set to danger behind the train. But wait, there's another one coming, a passenger train, the express. The gleaming engine rushes past, passenger's faces peer out of the varnished carriages. Again the signal must be pulled off, the signalman putting all his weight behind the lever as the heavy arm rises.
But all this was only in his imagination. In reality there was just a small boy and a toy signal, as he moved the arm up and down. After he had told his story on the radio, other people called in to say he'd reduced them to tears.
Tuesday, 22 April 2014
Retrotech: Nixie Tube Clock
It all started when my friend George gave me a tee-shirt for my birthday, with the internal circuit of a Dekatron counter tube on it. The Dekatron was a 1950s piece of electronic wizardry, capable of counting to 10 with almost no external circuitry, as opposed to the numerous tubes that would be required otherwise. This reawakened my interest in retro-electronics, never far from the surface. A few years back I built several tube (valve) amplifiers, which I still use for daily listening, and I still have a well-equipped workshop for this kind of stuff.
I'll gloss over the Dekatron, since hopefully I'll have something to write about that quite soon. But then one thing led to another and I started looking at things I could do with Nixie tubes. These are about the coolest thing in retro-electronics. They look like normal tubes but inside they have ten electrodes and can display any digit from 0 to 9 in an orange neon glow. When they first appeared, in the late 1950s, they were way too expensive for consumer equipment. They were used in laboratory equipment and the like. My school, in around 1967, had a Sumlock Anita calculator, the first electronic calculator, which used ten Nixie tubes to display its result (and some very clever tube-era electronics to do the calculations).
Now of course LCD displays are cheaper, smaller, and generally more convenient. There's no commercial interest in Nixie tubes. But they are really cool, and there are still lots of them about. The Russian electronics industry carried on making them for a lot longer, maybe even still is. They're available for $5-10 each for the common types. As a bonus, if you have friends who collect stamps, you get all sorts of interesting stamps with them from places like Moldavia and Ukraine.
Retro-electronics is all very well, but to do anything with these tubes you have to give them something useful to display, like the time. And that's where a partnership with modern electronics comes in handy. For one thing they run on around 150 volts. With 1960s technology, that would mean a big, heavy transformer and associated parts. Today, it can be generated with a few tiny components that fit into a square inch or so on a corner of a circuit board using a boost-mode switched power supply circuit.
Powerful microprocessors are now available for a dollar or so, so it's easy to generate something interesting to display. There are several kits on the market for digital clocks using Nixie tubes. I chose one from PV Engineering in England - it had the best looks in my opinion, and it didn't hurt that it is quite a bit cheaper than any others. It took a few days to arrive, and when I opened the package there were six never-used Nixie tubes, an empty circuit board, and a bag of components.
In anticipation of its arrival, I'd already cleared off my workbench - a project that had been waiting for at least a year. So the only thing left was to launch into its construction. I started on Sunday morning, and by an admittedly very late dinner time it was working and housed in the very nice clear plastic box designed for it. I suppose the total actual assembly time was about 4-5 hours. By far the most painful part was wiring the tubes themselves. Unlike most vacuum tubes they have wire ends which have to be individually soldered into place, and there are eleven of them (necessarily) on each. Actually there are 13, because two aren't used and have to be cut off. This is a bit nerve-wracking the first time, it would be so easy to cut off the wrong wires and be left with useless tubes. The total number of solder joints to be made is probably about 500, of which roughly half are for the tubes themselves and their associated wiring. But the instructions are excellent and make it easy to do everything step by step and even test things along the way.
A Nixie-tube clock all by itself would be neat, but since it's controlled by a computer (one of the omnipresent PIC range - there are probably a couple of dozen in every car) it can do a lot more. At 50 seconds past every minute the digits all flash and spin, and it shows the date for a few seconds. There's an alarm clock. Underneath each tube is a three-color LED, with a nifty light tube made of shrink-wrap that shines up into the glass base of the tube. The color can not only be set, it can be set differently for each hour of the day and can be made to change slowly or quickly through the whole spectrum. It comes with a tiny GPS receiver which can be stuck onto a window and which ensures that it always shows the right time and date, even after a lengthy power outage. I wish all the other myriad household devices that include superfluous clocks were half as smart.
For now it's still sitting on my bench while I decide where I want it. It indeed looks very cool.
Update, April 2025
More than ten years later, my Nixie tube clock is still very much in use. Or rather, is again in use.
After about four years it stopped working, just showing random values in the display. A quick mail to the supplier got me instructions for a complete reset, which did the job. But the GPS receiver had stopped working too, so every time the power was interrupted it had to be manually reset. This is quite complicated and the result was that I just stopped using it. I put it away somewhere and that was the end of it. Then we moved to France, and with the chaos of unpacking into a new home I couldn't even say where it was.
A few months back I was looking for something else, and came across it. I plugged it in, and it worked - though still without GPS time. But I discovered that if I hung the GPS receiver out of the window of my office, after a few minutes it would synchronize. We wanted to put it in the living room though, in a place where there is no access to a window.
I'd been looking at the supplier's website, and I saw that he now has an NTP receiver. NTP is the universal internet time service, used by all computers to get the correct time. This tiny module connects to the home WiFi network and listens to NTP.
I also bought a 220 volt power supply. With the two of those installed, my Nixie clock now has pride of place in the living room.
Sunday, 8 September 2013
Choosing a Linux Distro for an Embedded System
Many years ago, when my old employer Digital Equipment Corporation was trying to stave off the advance of Unix, it came up with the slogan "Unix ain't Unix". In other words, there were a lot of systems at the time (around 1988) that all called themselves Unix, but which in fact were all different - in management, in commands, and in the APIs they supported.
Fast forward to 2013, and Linux is really the only Unix that matters any more. (Yes, I know about BSD - more later on that - and I stand by my statement). But the statement "Linux ain't Linux" applies with just as much truth.
I've been using Linux as my main development environment for a couple of years now. I started with Ubuntu 11.04, for no better reason than I happened to have a DVD of it. It was a pretty decent system - it stayed up for weeks at a time and had a usable Windows-ish GUI. If only that were still true. Recent releases of Ubuntu rarely stay up for more than a day or two at a time, typically before the window manager dies leaving the rest of the system ticking away but completely unusable. And of course they made the incomprehensible decision to replace Gnome, which is dull but functional, with Spirit. (There's an old - c. 1780 - and rather delicious quote that "Englishwomen's shoes seem to have been made by someone who has heard shoes described but never actually seen any". Ditto with Spirit and the Mac). The first thing I do whenever I bring up a new Ubuntu system is to replace it with "Gnome Classic". (The latest version of Gnome in turn seems to have been developed by someone who has heard Spirit described but never actually seen it).
For the last year I've been developing an embedded system for Internet traffic management and monitoring. From the beginning we've taken for granted that it would run on Linux. The question is, which Linux distro should we use? There are numerous choices: Ubuntu, Fedora, Red Hat, Centos, Arch, Gentoo - and those are just the well-known ones.
For sure Ubuntu is a poor choice. It's desperately trying to be a replacement for Microsoft Windows, and has way too much clutter and extra stuff for an embedded system. We're trying to keep the footprint small, both memory and virtual disk, and we really don't need to have three different GUIs, LibreOffice, three different database systems... you get the picture. So Ubuntu was out from the beginning.
I worked with a company that had selected Gentoo. The advantage of Gentoo is that you get to choose absolutely everything about the system, down to the tiniest details like which implementation of cron you use. The disadvantage of Gentoo is that you have to... do all that. It's true that it will give you an absolutely minimal system, tailored exactly as you need it, but it's a lot of effort, not to mention the learning curve. It might make sense when we're bigger. but right now we need everyone focussed on stuff that will really differentiate us.
Somewhere along the line we looked at BSD - we were using something at the time whose support was much better there than on Linux. What a nightmare! Everything has to be built from source - they have a repository system but it is 'temporarily out of service'. It's truly a system for hobbyists, like Xen.
I looked at Centos, and got as far as installing it on a system. Then I realised that it is rooted so far in the past that I'd almost have to dig out my stock of IBM punch-cards. In particular, it supports a truly ancient version of GCC (4.4 I think). We make extensive use of features from C++11, which means we need at least 4.7. There are ways to have a development environment which is distinct from the system's own build environment, but they look pretty terrifying and weren't something I wanted to try and get my head around - for the same reason I didn't want to become a Gentoo expert.
That left Arch. I'd heard good things about it, and it also tries to be minimalist, so it seemed to be the way to go. I installed an Arch system without too much trouble, and got our system up an running on it. The only problem was log4cxx, which isn't available as a package and which wouldn't build from source either. Like much Linux software out there, it has a bunch of outdated assumptions about implict include files which don't work with recent versions of gcc. But the changes were simple and we quickly had a version that would build.
Networking in Arch is very quirky. It starts with ethernet devices, which instead of being called eth0 and so on, have names which reflect the PCI heirarchy like 'ep5d3'. It's a nuisance but not a major problem. But then it turns out they selected a completely different way to manage networking than other Linuxes. User administration is completely different, too. I'm sure the answer would be "but you can always build whatever you want and do it your own way." True, but not especially helpful.
Anyway, we persevered with Arch, and got our systems running. It took the passage of time to realise that Arch is constantly changing - as in, every day. An Arch system installed and configured today won't be the same as one installed tomorrow. Anything and everything can change - the kernel, the utilities, the drivers. When Boost 1.53 came out, Arch had it a few days later. Switching Boost versions is not something to be undertaken lightly, and indeed our system wouldn't build - some incompatible change involving locales, themselves a completely incomprehensible feature of Linux.
Our biggest problem came from trying to integrate the Intel DPDK package for high-performance user-space networking. Now, DPDK is essential to what we're doing. But it is hardly a model of stability either, with a new version coming out practically every week. The combination of this with the ever-changing sands of Arch, especially kernel changes, just made it impossible to keep up. If we got things working on Monday, they'd be broken again on Tuesday.
We looked into somehow selecting our own stable intercept of Arch. In a VM environment, it's easy enough to build a master VM and just use that. But our system also has to run on bare metal, which is not so easy. There is, supposedly, a way to take a snapshot and make a private repository. But once again, the investment in time is just not something a tiny group like ours can afford to make if we are to ship a product in a reasonable time.
And so, with great reluctance, I made the decision last week that we will ship our product on Ubuntu. I know that it is really not the right choice for an embedded system. But it works, and it doesn't change on a daily basis. We're used to its quirks, like yet another gratuitously incompatible set of network configuration tools. Hopefully we'll have the luxury of re-examining this later on when we have more people and more time to look at it.
Fast forward to 2013, and Linux is really the only Unix that matters any more. (Yes, I know about BSD - more later on that - and I stand by my statement). But the statement "Linux ain't Linux" applies with just as much truth.
I've been using Linux as my main development environment for a couple of years now. I started with Ubuntu 11.04, for no better reason than I happened to have a DVD of it. It was a pretty decent system - it stayed up for weeks at a time and had a usable Windows-ish GUI. If only that were still true. Recent releases of Ubuntu rarely stay up for more than a day or two at a time, typically before the window manager dies leaving the rest of the system ticking away but completely unusable. And of course they made the incomprehensible decision to replace Gnome, which is dull but functional, with Spirit. (There's an old - c. 1780 - and rather delicious quote that "Englishwomen's shoes seem to have been made by someone who has heard shoes described but never actually seen any". Ditto with Spirit and the Mac). The first thing I do whenever I bring up a new Ubuntu system is to replace it with "Gnome Classic". (The latest version of Gnome in turn seems to have been developed by someone who has heard Spirit described but never actually seen it).
For the last year I've been developing an embedded system for Internet traffic management and monitoring. From the beginning we've taken for granted that it would run on Linux. The question is, which Linux distro should we use? There are numerous choices: Ubuntu, Fedora, Red Hat, Centos, Arch, Gentoo - and those are just the well-known ones.
For sure Ubuntu is a poor choice. It's desperately trying to be a replacement for Microsoft Windows, and has way too much clutter and extra stuff for an embedded system. We're trying to keep the footprint small, both memory and virtual disk, and we really don't need to have three different GUIs, LibreOffice, three different database systems... you get the picture. So Ubuntu was out from the beginning.
I worked with a company that had selected Gentoo. The advantage of Gentoo is that you get to choose absolutely everything about the system, down to the tiniest details like which implementation of cron you use. The disadvantage of Gentoo is that you have to... do all that. It's true that it will give you an absolutely minimal system, tailored exactly as you need it, but it's a lot of effort, not to mention the learning curve. It might make sense when we're bigger. but right now we need everyone focussed on stuff that will really differentiate us.
Somewhere along the line we looked at BSD - we were using something at the time whose support was much better there than on Linux. What a nightmare! Everything has to be built from source - they have a repository system but it is 'temporarily out of service'. It's truly a system for hobbyists, like Xen.
I looked at Centos, and got as far as installing it on a system. Then I realised that it is rooted so far in the past that I'd almost have to dig out my stock of IBM punch-cards. In particular, it supports a truly ancient version of GCC (4.4 I think). We make extensive use of features from C++11, which means we need at least 4.7. There are ways to have a development environment which is distinct from the system's own build environment, but they look pretty terrifying and weren't something I wanted to try and get my head around - for the same reason I didn't want to become a Gentoo expert.
That left Arch. I'd heard good things about it, and it also tries to be minimalist, so it seemed to be the way to go. I installed an Arch system without too much trouble, and got our system up an running on it. The only problem was log4cxx, which isn't available as a package and which wouldn't build from source either. Like much Linux software out there, it has a bunch of outdated assumptions about implict include files which don't work with recent versions of gcc. But the changes were simple and we quickly had a version that would build.
Networking in Arch is very quirky. It starts with ethernet devices, which instead of being called eth0 and so on, have names which reflect the PCI heirarchy like 'ep5d3'. It's a nuisance but not a major problem. But then it turns out they selected a completely different way to manage networking than other Linuxes. User administration is completely different, too. I'm sure the answer would be "but you can always build whatever you want and do it your own way." True, but not especially helpful.
Anyway, we persevered with Arch, and got our systems running. It took the passage of time to realise that Arch is constantly changing - as in, every day. An Arch system installed and configured today won't be the same as one installed tomorrow. Anything and everything can change - the kernel, the utilities, the drivers. When Boost 1.53 came out, Arch had it a few days later. Switching Boost versions is not something to be undertaken lightly, and indeed our system wouldn't build - some incompatible change involving locales, themselves a completely incomprehensible feature of Linux.
Our biggest problem came from trying to integrate the Intel DPDK package for high-performance user-space networking. Now, DPDK is essential to what we're doing. But it is hardly a model of stability either, with a new version coming out practically every week. The combination of this with the ever-changing sands of Arch, especially kernel changes, just made it impossible to keep up. If we got things working on Monday, they'd be broken again on Tuesday.
We looked into somehow selecting our own stable intercept of Arch. In a VM environment, it's easy enough to build a master VM and just use that. But our system also has to run on bare metal, which is not so easy. There is, supposedly, a way to take a snapshot and make a private repository. But once again, the investment in time is just not something a tiny group like ours can afford to make if we are to ship a product in a reasonable time.
And so, with great reluctance, I made the decision last week that we will ship our product on Ubuntu. I know that it is really not the right choice for an embedded system. But it works, and it doesn't change on a daily basis. We're used to its quirks, like yet another gratuitously incompatible set of network configuration tools. Hopefully we'll have the luxury of re-examining this later on when we have more people and more time to look at it.
Subscribe to:
Posts (Atom)





