Dr Larry Roberts, RIP - a personal retrospective

I learned yesterday that Dr Larry Roberts passed away on December 26th, at the relatively young age,  these days, of 81. I had the good fortune to work closely with Larry at his company Anagran, and since then also.

To me he was always just Larry, not Dr Roberts or Dr Larry. We were colleagues (though he was my boss) and we worked closely together. It was a privilege to know him this well. He was quite a humble man close up, though this wasn't at all the common perception of him. It fell to me to take his very visionary technical concepts, and turn them into something an engineer could go off and build. Let's just say there was sometimes quite a gulf between the two.

Larry created Anagran to pursue his conviction that flow-based routing was a superior technology, that could change the way the whole Internet worked. That was fitting, because if any one person could be said to have created the Internet, it was Larry. Other people make the claim, on the basis of having invented some core technology. But Larry was the person who convinced the US Government, which is to say the defense research agency (DARPA), to fund it. That made it accessible to every university and many private companies, long before most people had even heard of it - my then-employer was connected to the Arpanet, as it was called then, in the early 1980s.

A brief explanation of flow routing is as follows. Conventional (packet based) routers look at every data packet, at the addresses and other information in it, to decide how to treat it and where to send. It takes incredible ingenuity to do this fast enough to keep up with high-speed data transmission. Larry's idea was to do this only once for each network connection (e.g. each web page), thereby amortizing the work over, these days, hundreds of packets. It's not simple though, because for each packet you have to find the connection - the "flow" of flow routing - that it relates to. This too requires considerable ingenuity. By 2000 or so, the engineering tradeoffs were such that flow routing was demonstrably cheaper to build. However the established vendors, especially Cisco, had invested huge amounts and large teams in the technology of packet routers, and weren't about to make the switch.

In 1998 Larry created Caspian Networks to pursue this idea, attracting huge amounts of funding - over $400M during the life of the company. They did build a product, but the technology was barely ready for it. The result was large and expensive, and sold only to a handful of customers.

Larry realised this was the wrong approach. In 2004 he created Anagran, to apply the flow routing concept to a much smaller device. Thanks to a brilliant CTO from his past who he enticed to join him again, this was achieved. The Anagran FR-1000 was about a quarter the size, power consumption and cost of its traditional equivalent, the Cisco 7600. Technically, it was a huge success.

I left Cisco to join Anagran as the head of engineering in 2006. It took us another year to get to a shippable product, and then we learned the sad truth. Networks were so critical to companies' business that they weren't about to take the risk of switching to an unknown vendor just to save a relatively tiny amount in their overall IT budget.

Larry was not just a visionary. Part of his concept for Anagran was a truly innovative way to manage traffic, based on completely new mathematics and algorithms. He had implemented this himself, as a proof of concept, in what must surely be the biggest spreadsheet ever created. It used every single one of the 32768 rows supported by Excel. If a single cell was changed, it took about 10 minutes to recalculate the sheet. The concept, once you understood it, was simple enough, but turning it into something that would deal with a real world traffic mix and that could be implemented in our hardware, was a big job. It occupied most of my time for over a year, and even today we are constantly improving it. The result is described in US Patent 8509074. It was through working on this together that we really got to know each other.

This turned out to be key to the survival of Anagran. We repurposed the hardware we had built, to use this algorithm to control users' traffic in a service provider network, and successfully sold it as such. The company's eventual demise was a result of being a hardware company: hardware has to be "refreshed", which is to say reinvented from scratch, every few years. And our revenue was not enough to sustain that. Software, on the other hand, just carries on, constantly under change but never needing to be started over. Cisco IOS probably still has lines in it from when it was first coded in 1983.

Larry was a genius and a visionary, but nobody can be everything at once. Some people found him overwhelming, and he could be brutally abrupt with people who didn't know what they were talking about. He was also a huge optimist when it came to Anagran's business prospects, which led to strained relationships with the investors.

Anagran finally closed down in 2011. I'm very pleased to say that Larry's brilliant flow management invention survives, since the company's assets - especially the patents - were purchased by the company I founded in 2013, Saisei Networks, and his work is very much still in use. We continued to work with Larry in his post-Anagran ventures and I saw him often.

We'll miss you, Larry, even - maybe especially - the times when your ability to see ahead of everyone else, and incomprehension that they couldn't see it too, made life challenging.

Rest in Peace.

Return to Anza Borrego

An Englishman can't really complain about the Bay Area weather, but it does get chilly and miserable around the end of the year. So we made a last minute decision to escape to the desert for a few days. It was our fifth trip to Anza Borrego, and our second with our Toyota FJ - you can read about the first, exactly three years ago, here. Since then we made one quick trip to see the wildflowers a couple of years ago. We rented a gigantic GMC Yukon XL, the only 4WD that Enterprise in San Diego could find for us. We nicknamed him Obelix, after the super-strong character in Asterix.

It's a long drive and it was dinner time when we arrived at our rented condo. The condo (really a house, joined by one wall to its neighbor) was very pleasant, thanks to vrbo.com - nice furniture, fantastic view, very comfortable enormous bed, to which we retired early. Just as well because we were awoken at 7am by the Grumpy Old Man next door, complaining that we were blocking his garage. We weren't, and as far as I could tell he didn't go out all day anyway, but Grumpy Gotta Grump. It was the perfect opportunity to make an early start, out at sunrise. But we went back to bed anyway, and it was after 11 by the time we started.

Day 1: Badlands, Truckhaven Trail

We wanted to revisit the badlands. It's an extraordinary place, visible from above at Font's Point. Driving through them is a completely different experience, only possible with a serious 4WD vehicle.

One frustration with the park is that there is no perfect map. The best paper map shows most trails, but not all of them - it would be too cluttered. The USGS 25000:1 topo maps are amazingly detailed, showing trails and how they relate to other features. What's more, they're free to download to an excellent iPad app, which gives you GPS location and many other features. The only problem is that they are updated very infrequently for rural areas - maybe every 50 years or less. They show "Jeep Trails" which have long since been banished to wilderness areas or just disappeared, and they don't show trails which have been created lately - as in, within the living memory of most of the population. There are several important trails in Anza Borrego that come into this category.

In this case we chose a trail which does appear on the topo map, though not on the paper map. It starts at the end of the dead-end road headed due north from where Yaqui Pass Road meets Borrego Springs Road and turns left. At first it seems like the driveway for a few houses and lots, but then it sets out confusingly eastbound, with several unmarked side trails. Eventually it joins Rainbow Wash, where you can turn left to the bottom of Font's Point, or right as we did. You need to turn left at the Cut Across Trail, which means keeping your eyes open because this is another that is not on the topo map. Most of it is just a sandy trail crossing several washes. At the end it enters the badlands, winding in and out of the landscape of low hills made of something between dried mud and sandstone. It had rained just before our arrival and there were green shoots everywhere - except in the badlands, which are truly lunar with not a plant in sight. The soil must be very alkaline.

Badlands in the setting sun

Winding through the badlands brings you eventually to Una Palma. At least it used to be - now the trunk of the palm lies on the ground. Five Palms, further along, has only four. We didn't count at 17 Palms.

The trail exits via Arroyo Salado onto the main road (S-22). There's a more interesting route, though, along the old Truckhaven Trail, which climbs out of the arroyo to the north-east. This road was built in the 1920s, the first road access to Borrego Springs. "Doc" Beaty led the effort by local ranchers, using mule-drawn scrapers to his own design.

I drove it on my last trip and found it mostly easy, climbing from one arroyo to another. There is one difficult stretch, bulldozed up the side of an arroyo to bypass a landslide further down. Even that, though steep and rocky, was easy enough if taken slowly and carefully. What a difference this time! The steep climb is very eroded and rocky. It requires very great care, constantly steering around and over big rocks. There is a second climb, part of the original 1920s road, which was just a steep dirt road before. Now it too is deeply rutted and full of big rocks. By chance I found the dashcam video of my 2015 trip, which shows the difference very clearly. Still, FJ made both climbs without a care in the world, using low gear but with no need for lockers.

Dinner on our second night was at Carlee's, the best bar in town (maybe the only one too), steak and ribs accompanied by margaritas and beer, followed by a few games of pool.

Day 2: Canyon Sin Nombre, Diablo Dropoff, Fish Creek

Today's goal was to drive through Canyon Sin Nombre (that's its name, No Name Canyon) then across to the Diablo Dropoff, a very steep one-way trail into Fish Creek Wash. We did this back in 2015, one of the classic Anza Borrego journeys. Sin Nombre is like a large-scale version of the badlands, with tall canyon walls made of similar crumbly almost-sandstone. There are lots of side canyons that you can hike into and explore.

The link to the dropoff is Arroyo Seco del Diablo, another long, twisty and spectacular canyon, and an easy drive. At least, it was last time. About half way through we came upon a stopped truck, whose crew of two were puzzling over how to traverse a large and very recent rockfall. There was no way either of our vehicles could climb over it. There was a possible bypass, which involved climbing onto and over a pile of soft sand about six feet tall. There were no tire tracks either over the rockfall, or over the sand pile, meaning we were the first people to try it.

We spent some time discussing possible tracks. Between us we were fully equipped, with shovels, jacks, traction boards and a winch. Still neither of us wanted to get stuck, and above all neither of us wanted to roll off the side of the sand pile.

Our new companion, Ryan, went first but didn't get far. He hadn't engaged lockers, and the wheels just spun in the deep sand as he tried to climb it. Worse, he slid alarmingly sideways. He backed down again, and we discussed some more, using the time to shovel the worst of the soft sand out of the way.

While he aired down to try again, I made my attempt. I'd already aired down to my usual 25 psi - not real airing down, like to 18 psi, but enough to make life easier for the tires over sharp rocks and such.  I engaged low gear, turned on all the locking, made a running start at the hill... and bingo, there I was on top. I paused briefly but the car was at an awkward angle, way short of its rollover angle but still very uncomfortable. There was another tippy moment dropping off the hill and then... I was through!

Ryan followed shortly, after locking everything he could. Then we were off to the Diablo Dropoff. This is a pretty steep angle in a shallow canyon, in itself not too serious. But the trail has been very badly damaged by people trying to go up, their wheels spinning and making deep holes in the sandy surface. The challenge is to negotiate these without losing lateral control, which is to say sliding sideways to a bad conclusion. From within the vehicle it's not too bad, though the occasional slight sideslip as a wheel goes into a hole certainly gets your attention. It looks a lot worse from outside.

There's a second drop further down, a bit easier in my opinion, and a bit of moderate rock crawling at the bottom. And then you're in Fish Creek, which is an easy sandy wash. We drove upstream as far as Sandstone Canyon, which is like a smaller version of Titus Canyon in Death Valley, winding through the narrow gap between high sandstone walls. We got about half way in before encountering the rockfall which has blocked it for years. There are tire tracks over the rocks and deeper into the canyon, but neither we nor our new companions were ready to try that.

We'd been so absorbed by all these events that we hadn't eaten lunch, and now it was 4pm and the sun was setting fast. We found a place in the main canyon where we could catch the very last of the sun while we feasted on cheese and crackers. From there it's a long drive out to the hard road, taking nearly an hour, with continuous magnificent scenery.

Our final stop was the Iron Door, the dive bar in Ocotillo Wells which is a great place for a post-trail beer. And nothing else. The very first time we went there, my partner asked for tea. "We got beer" was the response. "OK, I'll have a beer" - a wise reaction.

Day 3

Our main goal for today was a repeat run up Rockhouse Road. We did this during our wildflower visit, with Obelix who for his size did a surprisingly good job on the narrow twisty upper part of the trail.

Inspiration Point and the Dump Trail

But first, I wanted to visit Inspiration Point. This is another viewpoint over the badlands, a little north of Font's Point, with its own trail from the main road. The paper map shows the trail continuing westwards towards the main road again, though none of this is depicted on the topo map. And indeed there's a short but steep dropoff which goes straight into a very narrow, twisty track between the low hills of the western badlands. There were plenty of tire tracks, which is always encouraging, especially when you don't have a good map to help you at ambiguous junctions, of which there were plenty.

Just after one of them, we came to an unpassable rock fall in the bottom of the narrow canyon. Even if we could have climbed over or round it, the trail disappeared on the other side, replaced by a deep sand drift. We backed up to the last junction, and spotted some tracks that climbed out of the shallow wash. We followed these as they twisted around, the original canyon always in sight to the left, sometimes very close, sometimes further off. The other tracks gradually faded away until finally we were following the traces of just one vehicle, which had probably passed in the last 24 hours. We hoped he knew what he was doing.

Eventually his tracks did a long, shallow S-turn down into the floor of the canyon. From there it was a straightforward drive along what at this point has the picturesque name of the Dump Trail. The reason eventually becomes clear, at a crossroads on the corner of the county dump. The paper map shows the trail simply ending there, which seems improbable - and very annoying if true. By now there were lots of tracks again, so there must be some way out.

Eventually, after a few exploratory wanderings, we followed the dump's fence south and then west, ending up on its access road. From there it was a short drive to the main road.

Rockhouse Road

Rockhouse Road provides access to the eastern end of the cutely named Alcoholic Pass, leading over the ridge from Coyote Valley. We'd thought about hiking up it - we did it once in the opposite direction, on our very first visit, stopping at the ridge. But there was a strong, cold wind. We went further up the trail than we did with Obelix, onto the narrow part that eventually leads to Hidden Spring. It was very rocky and in poor condition, so we decided to stop and have our lunch. It was so windy that we ate inside FJ, something we normally never do. While we were eating we were passed by two FJs racing along the trail. I guess they made it to the end - we saw them again later on the main road.

Looking down from Rockhouse Road

The view from our lunch spot was spectacular, from several hundred feet above the valley floor and Clark Dry Lake. This time there was no carpet of wild flowers, but the ocotillos were just starting to bloom, with their bright red flowers contrasting with their deep green leaf-covered stems.

Font's Point and Vista del Malpais

There were still a couple of hours before sunset when we reached the main road. We've always visited Font's Point, the classic overview of the badlands, so that's where we went. It's an easy drive up a very wide sandy wash - I've done it a couple of times in 2WD rental cars. You just have to be careful to stay in the tire tracks and avoid any deep sand - though I understand rental cars routinely get stuck. Once we saw one that had barely made it off the highway before burying itself up to the hubs in sand.

Badlands, from Vista del Malpais

As we were coming back down the wash, I noticed a Jeep zip off into a side turning, Short Wash. I've seen it on the map but never before managed to figure out where it was - the topo map doesn't show it. It's always interesting to drive a new trail, but this one had something else: a side trail to a place called Vista del Malpais (Badlands View). That seemed interesting, so we turned right. None of this is shown on the topo map, so finding the side trails was a challenge. We found the turnoff using clues from the bends shown on the paper map. A narrow, twisty trail led through the badlands, ending before a final short hike to the ridge. The view was breathtaking, much closer than at Font's Point. We soaked up the view, then turned back onto Short Wash.

We were a little surprised, maybe a quarter mile later, to see a sign for Vista del Malpais up another side track. We followed it, along a bigger trail that ended in a small parking lot on the ridge. The real Vista del Malpais was very impressive too, but we were happy to have found our very own one.

After that it was back to the house. Dinner that night was at La Casa de Zorro, Borrego Springs' only "fancy" restaurant, conveniently only a mile from our house. We've eaten there before and it was decent, but this time we were not so impressed. In future we'll probably stick to Carlee's and the other every-day places in the town. And then, next morning up early for the long drive up I-5 back home.