Mixed Blessings Of Greenfield Software Development

The biggest software project I have ever worked on, and hopefully ever will work on, was Gecko. I was not one of its original architects, so my work on Gecko was initially very tightly constrained by design decisions made by others. Over the years some of those decisions were rescinded, and some of the new decisions were mine, but I always felt frustrated at being locked into designs that I didn't choose and (with the benefit of hindsight, at least) would not have chosen. "Wouldn't it be great", I often thought, "to build something entirely new from scratch, hopefully getting things right, or at least having no-one other than myself to blame for my mistakes!" I guess a lot of programmers feel this, and that's why we see more project duplication than the world really needs.

I was lucky enough at Mozilla to work on a project that gave me a bit of an outlet for this — rr. I didn't actually do much coding in rr at the beginning — Albert Noll, Nimrod Partush and Chris Jones got it underway — but I did participate in the design decisions.

Over the last two years Kyle Huey and I have been working on a new project which is broader in scope and more complicated than rr. The two of us have designed everything (with some feedback from some wonderful people, who know who they are) and implemented everything. We've been able to make our design choices quite freely — still constrained by external realities (e.g. the vagaries of the x86 instruction set), but not by legacy code. It has been exhilarating.

However, with freedom comes responsibility. My decision-making has constantly been haunted by the fear of being "That Person" — the one whom, years from now, developers will curse as they work around and within That Person's mistakes. I've also come to realize that being unencumbered by legacy design decisions can make design more difficult because the space is so unconstrained. This is exacerbated by the kind of project we're undertaking: it's a first-of-a-kind, so there aren't patterns to follow, and our system has novel, powerful and flexible technical capabilities, so we can dream up all sorts of features and implement them in different ways. It's scary. We have to accept that mistakes will be made and just hope that none of them are crippling. After all, if developers are cursing our design decisions years from now, that means we succeeded!

One of our biggest challenges is constantly trying to strike the right balance between forward planning and expediency. It's always tempting to respond to uncertainty by planning for all possibilities, by building abstractions that make it easier to change decisions later. Another temptation is to obsessively study issues to reduce the uncertainty. All that represents delay for our minimum viable product, and therefore must be minimized. My general approach here is to try to think about the future, but not code for it; to tell stories about how we would handle possible future scenarios, but not invest in them yet.

I think we're doing OK. Being nearly two years in is long enough to have regrets, and we don't have many. It will be very interesting, and humbling, to review our scorecard over the longer term.

Marriage On Credit

A few weeks ago I went to a wedding. It was a good one. I realized that weddings, or at least Christian weddings, are like buying something important on credit. The couple get a fine new relationship — unlike anything they've had before, if they've stuck to Christian ideals — but they pay for it with extravagant vows. Those vows cost nothing at the time, but the couple commit themselves to pay, potentially, a very steep price indeed. At the altar, do people take seriously the possibility that they're committing to a lifetime of caring for an invalid? A lifetime of putting up with frustrating habits? Infertility? Unfulfilled sexual desire? Poverty? Bonding with a dysfunctional family? Diverging interests? I want to stand up in weddings and shout "you have no idea what you're doing!" I should never be a marriage celebrant :-).

Sometimes people think marriage is conventional and boring, but taking the long view, marriage is for crazy risk-takers.

It's unfortunate that we celebrate the making of vows more than the keeping of them — paying off the credit: the slow grind of adjusting one's expectations, of putting up with the less than ideal, of carefully engineering the relationship to strengthen it over time. Wedding anniversaries deserve bigger parties than weddings.

I've got a big anniversary coming up next year so I should take my own advice :-).

Routeburn-Caples

Yesterday I got back from another excellent tramp on the Routeburn Track. This time was a bit different from last time mainly because we had a party of ten people — family, friends, and family of friends, with a wide variety of ages and experience levels.

The highlight for me was the second day, Routeburn Falls Hut over the Harris Saddle to Lake McKenzie Hut. Last time it was wet with low visibility but this time the weather was absolutely perfect, a cool but clear and sunny day. Most of us did the side trip to climb up Conical Hill (1500m) for truly outstanding views east to the Routeburn Valley, west across the Hollyford Valley to the Darran Mountains, north to Lake Mckerrow, Martins Bay and the Tasman Sea, and south up the Hollyford River towards Lake Gunn. Later in the tramp we met a couple who've done the Routeburn every year for several years but never had good enough weather to bother going up Conical Hill, so we were very blessed indeed.

Another highlight was a literal "carols (and other hymns) by candlelight" sing-along at Routeburn Falls. That was special!

Taking our big diverse group was a lot of fun; it's always a privilege to introduce people to the joys of tramping in New Zealand. Cooking bacon and eggs for ten for breakfast at the Falls Hut was epic. Everyone did well and I think everyone had a good time. If this trip wasn't enough to get people excited about tramping then I guess nothing will!

We said goodbye to our main group at Lake Howden Hut on the third day; they walked for another hour to exit at Hollyford Road, while my children and I walked south to Mckellar Hut. The next day we crossed the Mckellar Saddle to the Caples Valley, stayed the night at Mid-Caples Hut, and yesterday walked out at the Greenstone road end. Last time we did a similar extension through the Greenstone Valley, which ends at the same place. I think I slightly prefer the Caples route; the forest and river in the upper Caples are delightful and there's a good view from the top of the saddle. There's also a nice swimming hole near the Mid-Caples Hut. Having just me and the kids for a few days on a quiet track made a nice contrast from the responsibility of leading a big group!

rr 5.1.0 Released

This is a minor maintenance release. It fixes a few issues that showed up in Ubuntu 17.10 and Fedora 27, such as glibc switching to the openat system call which then had to be supported by rr's syscall buffering for good performance. info proc maps works in gdb during rr replay now. Of course various other bugs were fixed.

One interesting thing about this release is that it's the first to have guaranteed backward compatibility with an earlier release: traces made by rr 5.0.0 should be replayable in rr 5.1.0 (and any later release, I hope).

Maintaining An Independent Browser Is Incredibly Expensive

Mozilla's revenue increased to over $500M in 2016. Good for them! Some people have asked why all that money, and Mozilla's 1,200-ish staff, are needed just to build Firefox.

In practice building and maintaining a truly independent browser is incredibly expensive. (By truly independent, I mean a browser with its own implementation of HTML, CSS, JS, etc; this kind of independence is critically important to the open Web, and the world, and hence Mozilla's mission.)

Why is it so expensive? First you have to do the obvious stuff: implement the client browser engine and interface. Web standards are vast and complicated and ever-evolving, so this is a vast and complicated task. For this kind of security-critical, long-lived software in an intensely competitive mass market, you have do a very high-quality job, paying attention to all the various "ilities" — security, compatibility, performance, portability, accessibility, internationalization, maintainability, and so on. But also these days a large server-side investment is needed: downloads, updates, addons, crash collection, telemetry, push notifications, sync, and so on. Then you have to support all your developers with CI, massive test infrastructure, analysis and viewing of crash and telemetry data, etc. You also support your developers by writing custom tools and improving open-source tools; for example the rr project was born at Mozilla to improve life for Mozilla's C++ developers.

If you care about the open Web standards you're implementing, then your people need to spend time doing standards work. Since you have an independent engine you need to constantly evangelise Web developers and manage other external relationships.

If you want to stay competitive for the long term you need something like Mozilla Research, building stuff like Rust and Servo to explore technology that may give you a competitive edge in years to come.

Your product needs to be in the hands of users to be relevant, so you need marketing, distribution, events and other kinds of user outreach.

Then of course you have all the people needed to let the above people do their work effectively — HR, PR, lawyers, accountants, logistics, office managers, personnel managers, executives, and so on.

I worked at Mozilla for a long time. Over the last five years headcount was at about the quoted level, even during the FirefoxOS years. We were constantly butting up against headcount limits, having more work than we had people to do it. 1,200 people is barely enough. In the future it might not be enough.

Of course if you're willing to give up on engine independence you can save a lot of those expenses by adopting another vendor's engine, probably Chromium, and drafting off Google's investment. It's no surprise that for-profit companies would take that path, and that only Apple and Microsoft, companies with deep pockets and deep fear of Google, have declined to do so (though even Microsoft has caved on Android). Of course, using the same engine as Chrome limits your options for competing against Chrome, but it's tough to argue that those extra options are worth more than the cost. This is one reason among many why Mozilla is so important: it's Mozilla's mission that justifies the costs of maintaining an independent browser engine.