The Doing Nothing Contract, or How Not to Run Large Projects

 Soon after I left DEC, in 1995, I got involved in what would have been the biggest Systems Integration (SI) project they had ever done. This is the story of the project.

I joined DEC when I left university, 20 years earlier. It was a fantastic place for an engineer to work, and I enjoyed nearly every day I worked there. But in 1995 it was obviously going downhill - it was acquired first by Compaq in 1999, and then by HP - and I found a way to make a decent exit. While I was looking for another job, I started a consultancy business which turned out to keep me busy for the next four years.

About a year later I ran into a former colleague on a plane. He told me about a project that they were working on for a major European telco. It was going to be huge. Did I know anyone who might be able to help? Well, yes, there was me. I think he knew that and was just being polite. I did point out that my daily rate was over double what DEC would normally pay. This wasn't a problem, he said, because they wanted to assemble an elite team of top-level architects to get the overall design right. Within a week I had a purchase order for three months of my time, 40 hours per week, at my usual high daily rate.

The following Monday I showed up at the DEC office in Reading, England. There were two other people in the "elite" team. One was Dave, who I knew quite well - like me, he had already spent 20 years as an employee.

The project was very interesting, on an oft-repeated theme. Telephone networks have always been built using proprietary systems built by specialized suppliers like (then) GEC and Alcatel, costing at least ten times more than normal contemporary computers. The client had figured out that this was just a big distributed computing application, and wanted to run their national telephone and data network using off the shelf computer hardware and, as far as possible, software.

It seems like a wonderful idea, but it has been tried several times and so far has never really worked (which I guess is a spoiler for this article). The problem is that, even now and certainly 25 years ago, these telcos were used to being almost their suppliers' only customer. They could make incoherent or outrageous demands, confident that their suppliers would have to follow. The price tag reflected this, but the people making the demands - the engineers - weren't the people paying the bills, so those dots never got joined up inside these huge bureaucracies. A typical interaction would go:

Supplier:   the project will use database X and transaction software Y
Telco:       that's no good, we need features P, Q and R that X and Y don't have
Supplier:  we could add those, but it's bespoke engineering and will add (lots) to the cost
Telco:       no, we want off-the-shelf software, we don't want to pay for custom development
Supplier:  X and Y is what's on the shelf, you want something else, you have to pay for it
Telco:       (utter incomprehension)

In its latest iteration, this has led to the ETSI NFV (Network Function Virtualization) project, which in its 8 years of existence, so far, has yet to deliver an actual functioning network.

Anyway... our mission was to use off the shelf DEC computers and software to build the switching control system at the heart of the network. It isn't really a hard problem. The basis of it is extremely simple: take a number that someone has dialled (this was a while ago) and translate it into a series of simple instructions to the physical switches, like "connect channel 92 of trunk 147 to channel 128 of trunk 256".

The only thing that makes it hard is the scale - this has to work for millions of users and concurrent calls. But even then, none of these actions have to be closely synchronised. It isn't like, say, Facebook. where something you upload needs to become instantly visible to a billion users around the world.

Within a week, Dave and I had figured out how to put the available software components together and have a working, scalable prototype within a couple of months. Turning that into a production system would be a much bigger job, needing integration to the telco's dozens of management systems, but that was all low risk, routine stuff. We started writing code.

What we had completely failed to take into account, working in our cosy little office, was the DEC bureaucracy. In a much larger open-plan office nearby was an already-large team of project managers, program managers, project documentation specialists, and for all we knew telephone sanitizers as well. They operated in blissful ignorance of any actual technical details, as they came up with the cost estimates that would be at the heart of the formal bid for the project.

DEC has always been thought of as a computer manufacturer, but they had a large and thriving SI business as well. Over the years they had built up a series of procedures and processes for managing these projects. They were mostly pretty small - integrate a driver for a new piece of hardware into an operating system, or build a user interface around a database application. But some were big - tens of person-years - and a few were really big, like our telco project.

So part of the process was to know when the project was too big for the current level of project management. When that happened, the project would get escalated to the next tier of project management. They would bring in their own team to look at the design, the business aspects, the risk, and everything else.

The first thing the new team would do is to multiply all the existing work estimates by two or more, just as a matter of principle because that is nearly always right. (A very successful SI company CEO who I knew years ago always multiplied all engineering estimates by pi. He claimed it worked every time). Then they would add a whole new layer of program managers, project documentation specialists, telephone sanitizers and all the rest. Then they would start looking at the details, invariably resulting in another factor of two or so.

Our project had already been through two such escalations. Realistically this was probably a 50 person-year project, but the estimates were already in the hundreds, maybe 20 times the original figure. As a result it triggered yet another escalation, to the ultimate level, the corporate Large Projects Office (LPO) in Geneva. 

The LPO was to SI what J K Rowling's Dementors were to Hogwarts. Their job was to suck all joy, and possibility of success, out of a project. I have no idea whether they actually delivered any Large Projects, but I doubt it. Within a week they had doubled all the existing estimates and added yet another layer of program management and the rest. The project had now reached a size - approaching 1000 person-years, 10 times any realistic estimate - that just flat-out terrified the country management. A project on this scale, if it went wrong - which was just about guaranteed - could take the whole company down, and certainly result in some very senior people needing to seek new career opportunities.

The whole team was called together in a large meeting room. DEC had decided to no-bid the project. Permanent employees would be reassigned as soon as possible, while all contractors were terminated immediately.

This is where things got surreal for Dave and myself. We went to some project management type and pointed out that there was no provision for termination in the purchase orders we had received. DEC had bought 90 days of my time, and 180 days of Dave's, just as if they had bought a thousand cases of beer.

"That," said the project management type, "is covered by our standard terms and conditions. It is implicit in the purchase order."

"Maybe," we replied, "but what isn't in the contract, isn't in the contract. The only contract we have is the PO. And the PO has no mention of any standard terms and conditions."

They quickly accepted that we had a point. "OK, but in that case you will have to accept to work on any other project for the duration of the contract."

That was fine by us. A week or so later, such a project cropped up. We started reading documents and figuring out what was needed. But a day later we were called into our original project manager's office.

"The new project is refusing to pay your rates. They are much higher than the normal DEC contract rate, and they won't pay. They say we have to pay because we agreed to the abnormal rate. But we're not willing to subsidize other projects. So you must stop work on the project immediately. And you are not to work on anything else either. You are forbidden to work on any project except the one you were originally hired for." And that had been cancelled, there was absolutely nothing to be done for it. So we were forbidden to do any work at all.

It was Dave who came up with the name "The Doing Nothing Contract". I was commuting from Nice at the time, flying to England on Monday and back on Friday. DEC insisted on me being at the office to do nothing - I was not permitted to do nothing from home. That was the beginning of the weirdest few weeks of my professional life. I'd found a hotel just outside town, an idiosyncratic converted farmhouse with low rates and enormous rooms furnished entirely from estate sales of dubious quality.

Dave (who did live fairly locally) and I would roll up to the office around 10, read the news and chat for a while, then around 11.30 go off to a pub for lunch. We'd be back by 2.30, spend another hour or two nattering (no web to surf back then), and go home. I still had plenty of friends from when I lived in Reading, so I never spent an evening on my own. I put on about five pounds during the brief period this lasted.

After a couple of weeks, I got another call from the project manager.

"Look, this is silly." I agreed. "How about if we pay off half the remaining contract, and call it quits?"

That was fine by me, I already had other work lined up and this was just free money. I left that afternoon and didn't return. Dave made the same suggestion, but his contract had a lot longer to run, so they said no.

We stayed in touch. A few days later, they called him in and told him they would simply terminate the contract "in breach", which is to say they would just stop paying him. Dave had been at DEC for years and knew exactly how things worked on the inside.

"But if you do that, I'll sue."

"Sure, yes, off the record, that's what we'd advise."

"And DEC never contests things like that, so you'll just settle, and end up paying the full amount, plus costs."

"That's true. But that will come out of a different budget, not ours."

Even they could see the silliness of all this, though. Soon afterwards they settled with him on the same basis as myself. He walked away with tens of thousands in unexpected cash, and went back to his day job doing IT projects for insurance companies. And that was the end of the Doing Nothing Contract.

A First Visit to the United States, 1976

In 1975 while working at Digital Equipment Corporation (DEC) I wrote a new terminal driver for the IAS timesharing system for the PDP-11. The 11 had an amazing variety of hardware interfaces for communicating with terminals (and for everything else too). In those days, hardware really was hardware, and device drivers messed with registers containing actual bits stored in actual transistors. Now of course, device drivers are just sending neatly packaged messages to other computers.

But enough of that. In our lab in England, we had only a few of the multifarious bits of hardware that I needed to test against - there was the DC11, the DH11, the DJ11, the DL11 (which came in at least five different varieties) and a brand new one, the DZ11. They were all completely different, having practically nothing in common. So, once I'd got all the basic functionality working, I had to go somewhere that had all of these. That meant only one place - Maynard, Massachusets, home to "The Mill", and the company's headquarters. And so, in early June 1976, I set off on my first ever business trip, my first journey outside Europe, and my first ride on a jet plane.

Every journey starts with a single step. My first step was the bus, which conveniently stopped outside the house where I lived, to Reading station, where I could catch another bus to the airport. I had a huge suitcase, containing several copies of the vital tapes with my code on - no Internet in those days, nor even Decnet, so if anything happened to the tapes, my trip would be pointless. As I boarded the bus, the driver said, "Looks like you're off on a long journey then." He seemed quite taken aback when I said, "Yes, I'm going to America".

The flight was on a TWA Boeing 707, three seats either side of a central aisle and a total of about 140 seats. First class, for the privileged few (not me) was like US domestic first class now - wider seats with a bit more recline, about the same as today's Premium Economy, though with better food. There were just two flights available per day, my TWA flight and British Airways on a VC10. A quick look at Expedia now shows seven wide-body flights per day (if I've correctly navigated the confusing mess of code-shares), which means over 20 times the capacity.

But before that, I had to navigate Heathrow Terminal 3. (It's a slightly embarrassing sign of age that I knew Heathrow before Terminal 4, which didn't open until 1986 and now seems positively ancient). It was my first time at Heathrow, not counting when I went there once or twice in the early 60s with my cousins just to look at the planes. It was very confusing. Now, after over 30 years of more or less continuous travel, it's easy to forget just how daunting it all seems the first time.

The plane was quite empty - I had a window seat, with someone in the aisle seat, but an empty seat in the middle. Back then empty seats were the norm. Even ten years later it was quite common to find a whole row of five seats on the unlamented L-1011 where you could stretch out and sleep through the flight. After all I'd heard about airline food, I was amazed to find that the meal was quite edible.

Once I'd retrieved my suitcase, I had to get out of Boston to Maynard, which lies about 20 miles to the west of the city. This could have been hard, but for one of the amazing and wonderful things about working for DEC back then - the private helicopter fleet. The company owned about a dozen Bell Jetrangers, which operated shuttles between all the New England sites, and to Boston airport. And this wasn't just an executive perk - any employee, including a lowly recent graduate like me, could use it, just by calling a number and making a reservation.

I'd never even seen a helicopter up close before, much less travelled on one. It picked us up in remote corner of the airport, then had to taxi to a place where it was allowed to take off. I was amazed as we hover-taxied along the marked vehicle routes, with airport trucks in front of us and behind, for all the world like a wheeled vehicle. Then we flew to Maynard. Passengers weren't given headsets, and the noise level inside was unbelievable.

I still needed a rental car. A colleague took me to a nearby Avis depot, and in no time I was the proud driver of a brand-new Plymouth Duster. American cars were still vast compared to European ones, and while classed as a "compact" it was twice as big as my car at home - despite having only two doors and vestigial rear seats. It fully lived up to the quality reputation that American cars had at the time. The following day I noticed that the water temperature gauge was pegged against the stop. I took it back to the same gas station and said, "It seems to be running hot". The mechanic (unwisely) pressed his hand to the engine, swore loudly, and gave me another car. There was no water in the engine - and probably never had been.

The new car was an AMC Pacer. This has to be one of the most bizarre mass-production cars ever built. It was like a greenhouse on wheels, absolutely perfect for a record-breaking summer where temperatures exceeded 100F for days on end.

But I'm getting ahead of myself. My first stop after arriving was the Mill, a huge collection of 19th century buildings which had once been a giant woollen mill. DEC had started here in a very small way in 1957, and by the time of my trip two decades later had filled the whole complex and overflowed into numerous other sites across Massachusetts and New Hampshire.

In an age of sterile campuses composed of indistinguishable low-rise buildings (like the Cisco campus in San Jose), it's hard to image anything as weird as the Mill. Software engineering was in building 5, at the very back of the site. To get to Main Street required traversing the whole complex, via aerial walkways connecting the buildings. This might have been straightforward except that the site wasn't level, and the buildings all had different ceiling heights. The entrance from the huge parking lot to building 5 was via a short walkway straight to level 5, referred to as 5-5, which happened to be where my temporary office was located. The next step on the journey was to descend two floors to 5-3, then take a walkway across to the sixth floor of building 3 (3-6). Walking the length of 3-6 allowed a peek into a lab containing a simple aluminium prototype front panel with two rows of lights on it, like the PDP-11 - except that there were 32 lights in each row. In 1976, this was the very first VAX, and a very closely guarded secret.

From the far end of 3-6, another walkway went to building 4 then down one floor and back across  another walkway to 3-4 (i.e. two floors down). The path continued like this until eventually you emerged into the main lobby, in building 12, and into the open air. Leaving through the parking lot, you passed by Ken Olsen's personal non-reserved parking space. It was a matter of corporate pride that nobody had reserved parking, not even the CEO - but woe betide anyone else who tried to park in that space.

Building 5 had been a warehouse for unprocessed wool at some point in its history. In consequence, the wooden floors were completely saturated with lanolin. If you made the mistake of walking round without shoes, your socks quickly turned into a soggy, sticky mess, beyond recovery by any washing machine. The fire risk was truly unbelievable - if a fire had started the whole building would have been an inferno within minutes. The Mill had its own fire department, bigger than the town of Maynard's, but it's hard to see what they could have done to save this particular building.

There were very few hotels in the area in those days. There was a distinctly low-end motel in Maynard. In Concord, two towns away, was the historic Colonial Inn, but that was full. I was booked into a Holiday Inn in Marlboro, about 15 miles away. On my map of Massachusetts this didn't look far, and there were very few roads marked. While Massachusetts is one of the smaller states of the US, it's still about a quarter the size of England - so my map wasn't very detailed. I set off trying to follow it, but soon became hopelessly lost. Eventually I realised I was passing in front of the same military base for a second time. I stopped and very cautiously approached the guard post, my hands in the air. I'd read all about how trigger happy Americans were, especially the military, and I was taking no chances. The guard was probably surprised, but very courteously and helpfully gave me the directions I needed.

The hotel was typical for its time, squashed into the space between two highways, with heating and cooling supplied by a huge roaring monster piece of machinery under the window. I quickly discovered that I didn't like hotel breakfasts - and still don't. I'd noticed on the way to work an establishment called Dunkin' Donuts, and decided to try that instead. For "the folks back home", I took a picture before going in there. You can see my Pacer, far right, and a good selection of contemporary American automobiles, particularly the excellent red specimen in the foreground, which seems to have a flat tyre. As I walked in the waitress said to me, in an astonished tone, "You from outta state?". I was probably the first person ever to take a picture. The breakfast was excellent, as long as you like massively sweetened fried dough covered in fake chocolate.

Chastened by my navigational experiences, I tried to buy a more detailed map. In England, then and now, it's easy to find high-quality large scale (50,000:1) maps by the Ordnance Survey. So I was surprised to discover nothing similar in the US. (They do in fact exist, at 25,000:1, but they are extremely hard to find). After some searching I found a large street atlas of the area, which I still have somewhere. I carried it with me on every single trip for several years, until eventually I'd memorised what I needed. It was organised around towns, each one having its own page, with no overlap to the adjacent towns. And naturally, each town had its own street names. There were very few roads that carried a single name across town boundaries. So you'd be driving along, say, Elm Street in Hudson, and you'd cross the town line, turn to the appropriate page - and then have to solve the topographical puzzle of finding that you were now on Oak Street in Marlboro. To make it harder, each town had an arbitrary orientation, aimed primarily at making the best use of the space on the page.

I was lucky to have good friends in the US already. My former boss had moved out there a year earlier with his family, and I'd met his friend Lois when she travelled to England. And a colleague - Roger - was travelling with me. That meant that I was well looked after - I rarely had to spend an evening by myself. Lois at that time was renting a house in Marlboro, close to the hotel, where one evening I met the famous Dave Cutler who was then writing the the VMS operating system, more or less single-handed. Shortly afterwards she moved to a delightful, and very old, house in Harvard, where I ate many excellent dinners on later trips.

At the weekend it was time to do some exploring. I'd heard about Framingham Mall as a place to go, so I took my new book of maps and aimed for Framingham. I still had a lot to learn - the Mall was nowhere near the centre of the town, which on a Saturday morning was like a ghost town. This was really the nadir of American town centres, before their gradual rebirth during the 80s and 90s. I found nothing of interest except a rather scary guy asking for money for some religious charity. I gave him $5 for my own safety, and quickly left. About the only activity left in the town was a busy railroad freight yard, which was the beginning of my interest in New England's very complicated railway system.

With Roger I visited Boston, a must-do considering it's the most historic major city in the country. We went to the newly-opened Faneuil Hall Marketplace, formerly the city's main produce market, reopened as a tourist attraction earlier that year, and still in 2012 very much in the same role. We ate at Durgin Park, a restaurant there which is famous - and  much-visited - for the exceptional rudeness of its service. We were not disappointed. We walked the Freedom Trail, and all the other things you're supposed to do in Boston.

One thing that struck me, and which has changed a lot, was the size of the cars. In 1976, only a few eccentrics drove non-American cars - VW Beetles and early, crude Toyotas and Datsuns. And the American cars were huge. I still remember seeing a Jaguar Mk 10, one of the biggest cars on the road in England, which was completely dwarfed by the cars around it. Now cars have shrunk, the most recent American cars are often derivatives of European ones, and those vast boat-like 1970s Chryslers dwarf everything around them - the few that have not rusted away by now.

Mention of Boston wouldn't be complete without its near-twin, Cambridge, home to the two great universities of Harvard and MIT, and especially to Harvard Square on the periphery of the eponymous seat of learning. Superficially it's an area that has changed little in four decades, but on closer inspection much of its charm has gone now. On this first trip I was introduced to Bailey's, a very traditional ice-cream parlour with delightful art deco furniture, that served the most amazing hot fudge sundaes, drooling with an extravagant excess of rich, dark sauce. It closed some time in the 80s, as did later another institution, the Harvard Co-op. This was a department store but best of all it had an enormous book department, completely oriented to the needs of the universities and crammed with textbooks on every academic subject imaginable. Before the Internet it was the perfect place to research obscure topics, better than a university library. I still have many shelves filled with books purchased there, including my extremely obscure favourite, the Dictionary of Synonyms in Selected Indo-European Languages.

Lunchtime drinking was still commonplace in Maynard in 1976. The drinking establishment on Main Street, imaginitively called The Maynard Pub, did a brisk trade in pitchers of beer at lunchtime, and again after work. I've even been told that it played an important role in the original design of Decnet. In the morning the architecture team would meet to design network protocols. Then after a well-lubricated lunch, they'd reconvene. If they could still understand what they'd done in the morning, it was simple and robust enough to be adopted.

Close to Maynard is the town of Concord, famous for its Old North Bridge where, in 1776, the revolutionary American forces defeated the British army and triggered American independence. This was exactly 200 years later, the year of the Bicentennial. We went to see the famous bridge, an unremarkable wooden structure which in fact has been rebuilt several times over two centuries, set in a very pleasant park. It makes a perfect early-morning walk if you're staying in Concord and wake up early, jetlagged.

There's a visitor centre where we went along to an illustrated talk about the events of 1776. The presenter didn't know how to react to the British sense of humour when, after she asked for questions, Roger piped up with, "Can we have our country back please?".

The hotel in the area was The Colonial Inn, a brisk walk away from the North Bridge in the centre of Concord. It has been there for a couple of centuries, though most of the rooms are in a rather charmless modern annexe dating only from 1949 - and not updated since. It makes up for that with boundless character and a very special New England kind of charm. Towards the end of the trip we were able to get a couple of rooms there, and moved from the Holiday Inn. It has been my base in the area ever since, despite the temptations of the numerous soulless modern hotels that have since opened. They serve a hearty breakfast of pancakes with maple syrup, which I'd never heard of before. In the evening they have a selection of traditional New England dishes in a delightful period dining room, which makes up for the sometimes haphazard service. (Staying there a few years later, I ordered a bottle of decent wine - the waiter's comment, as he pulled the cork, was "Jesus Christ, what a long cork!").

The difference in architecture between Britain and New England was very striking. To someone used to brick buildings, it was astonishing to see that practically all construction was of very tidily painted white wooden boards. Whole towns are made up of these pristine white houses grouped around a verdant square, the wooden church in pride of place.

During the day I was working, slaving over a hot computer in a windowless lab that took up much of one floor of the Mill. It was filled with PDP-11s of every kind, equipped with all of the vast panoply of peripherals available. One was called The Ark, because allegedly it had two of everything. All development work was done in the lab - nobody had terminals in their office. PCs and laptops were nearly a decade from being invented. Terminals were mostly the DEC replacement for the venerable ASR33 teletype, the LA30, or else early DEC video terminals - the VT05 and VT52. On this trip I saw a prototype VT100 - which went on to become the de facto video terminal and is still with us in the form of the control sequences used for terminal-like communication.

If someone was working in the lab, the only way to contact them was via a single shared telephone in the middle. When it rang someone would answer it (if you were lucky), then connect to a loudspeaker and announce who the call was for. If you were luckier still the person would come along and answer. But often, they weren't there and the call would just hang until someone came by and hung up the phone, possibly hours later. I lost our daily shared "Maynard call" from England several times this way.

1976 was the year of Bicentennial - 200 years of US independence. And nowhere was this taken more seriously than New England. I was lucky enough to be there on July 4th when everyone celebrated. I saw my first American parade complete with firetrucks, popcorn and families everywhere - and of course barbecues. It was impossible not to be drawn into the general festivity.

One of the pieces of hardware I needed, the DZ11, was so new that nobody had any. I had to drive to the plant where they were made, which was in a country town in Western Massachusetts. It was my first glimpse of rural America, of the distances and the isolation - though nothing compared to the West. I carefully carried it back, got it installed, and tested my software on it. I'd done everything I needed to, and it was time to go home.

The helicopter took us from Maynard to its corner of Logan airport, and the TWA 707 brought me home to England. It was the first trip of many, a journey that became tediously routine at times. But those first impressions of a foreign country remain forever.