A Risk Of Transactional Politics For Christians

I've subscribed to Christianity Today for a long time. I was surprised by the editorial advocating Trump's removal from office, but it makes sense to me.

NZ politics is very different from American politics, for which I am profoundly grateful, but it's worth thinking about what to do in a situation where the candidate I strongly preferred on policy was thoroughly immoral. I think one of the biggest issues is the potential corrupting effect of supporting such a candidate, to my own soul and the souls of Christians around me. I think I could easily start out supporting the candidate for policy reasons while declaring their behavior detestable, but there is a great risk that over time that separation breaks down because tribalism, and a tendency to avoid cognitive dissonance, lead to defending or minimizing more and more hitherto unacceptable behavior (Isaiah 5:20). I think it's clear this happens to a lot of people. I believe that kind of personal corruption is an extremely great evil; it would be better for Christians to suffer in all kinds of ways than to fall prey to it (Matthew 16:24-26). This is all even before we consider the impact on our proclamation of the gospel message.

So a good test would be: can I resist the temptation to defend the flaws in this hypothetical flawed candidate? If not, then it is very dangerous to support them. Even if I have adequate mental discipline, there's also the question of whether they are likely to corrupt others in this way.

Nelson Lakes Tramping: Lessons And Observations

Overall, we were really lucky with the weather. We arrived just after a big storm, so there was a lot of water in the streams and significant track damage, but we had very little rain in all ten days. On the Travers-Sabine circuit we had excellent conditions the entire time. I'm so grateful for that.

Because of track damage we had to abandon our plans to visit Blue Lake and significantly alter our route and schedule accordingly. It was actually fun to make up plans on the fly; it felt empowering to know we had the supplies to be relatively self-sufficient for several days and could go wherever we wanted within that time. We appreciatead that it's a feature that the huts in the area (apart from Angelus Hut) don't require or allow advance booking.

Apart from the storm-induced damage, the tracks in the area were very well maintained and easier than I expected.

As always, one of the great things about hut tramping is meeting lots of people from all over the world. On the more demanding tracks, we tended to meet people who were in New Zealand for several months (whereas on the Great Walks you often meet people who are only here for a few weeks). I asked a few people how they first got the idea of visiting New Zealand; more than once I heard they simply looked at the map and saw that New Zealand was about as far away as they could go.

On the Travers-Sabine circuit we ate mostly store-bought dehydrated food for dinner, and I simply don't like any brands that we've tried. I need to figure out how to pack and cook for a week-long tramp food that actually tastes good.

As I noted in a previous post, the water quality in the park was excellent, perhaps aided by the earlier storm. We drank untreated water from streams and rivers everywhere with no ill effects. Even standing pools of water were crystal clear.

We play a lot of card games in huts when tramping. This trip we brought playing cards, Citadels, Bang! (with expansions), and San Juan (with expansion). They're good games but they still got a bit boring, so we need to branch out.

Perhaps the most valuable thing we learned on this trip was how our fitness developed. As we walked day after day we experienced competing effects: we got more tired, but we also got fitter. About half-way through the Travers-Sabine circuit was probably the low point; after that, fitness started to win out. As I noted in the previous post, on day five of the circuit we ascended Speargrass valley to Robert Ridge considerably faster than we had done the same track at the start of our trip. At the end of the circuit everyone in our group agreed that we were feeling great and would have been quite happy to continue tramping, time and supplies permitting. That's an encouraging lesson for all of us.

It was great that we got to attend the "Chapel By the Lake" in St Arnaud twice. It seems like a good congregation, and it's always fun and helpful to see how God works in other churches. Thanks!

Overall it was a great holiday and I thoroughly detoxed from computer overexposure. Now I need to get back to work mode!

Nelson Lakes Tramping: Travers-Sabine Circuit

Following our trip to Angelus and a rest day on Sunday, on Monday we embarked on six-day walk of the Travers-Sabine Circuit, "clockwise". Our plan was to take a water taxi across Lake Rotoiti, walk up the Travers Valley to stay at Upper Travers Hut, and cross the Travers Saddle to West Sabine Hut on the second day. Then we'd leave the circuit proper for two days, walking to Blue Lake Hut and back to West Sabine. Then we'd spend two days finishing the circuit, walking back to St Arnaud with an overnight stay at Sabine Hut/Lake Rotoroa. Things did not go according to plan.

We had heard the previous week's big storm had caused numerous track washouts and changed the course of the Travers River. Department of Conservation people told us to take the Coldwater Track up the west side of the Travers Valley, because the parallel Lakehead Track on the east side was more badly washed out. (They were certainly right about that; we later met trampers who had had a miserable time on the Lakehead Track, and one very tough Polish woman who'd had to turn back.) When our water taxi arrived at Coldwater Hut to drop us off at the start of the track, the water around the jetty was completely clogged with rafts of floating logs and other vegetation! Presumably they'd been washed down in the storm. The skipper, who must have been 18 or 19 but seemed very capable, eventually managed to get us to the jetty after we helped maneuver some of the logs out of the way. We ended up having a very pleasant walk up the valley with just one significant washout on the Coldwater side forcing us to bush-bash off-track for about twenty minutes. It took us about eight hours of walking to get all the way to Upper Travers Hut but we all felt good. Upper Travers is a lovely area at the edge of the treeline below the Travers Saddle. Here we met a number of people — Kiwis, Minnesotans, and people from various European countries — we'd see again at West Sabine Hut and some at Sabine Hut too.

The next day we crossed Travers Saddle in perfect conditions. There were great views of Upper Travers valley, the East and West Sabine valleys, Mount Travers next to us and other mountains nearby. Then there was a long steep descent to the East Sabine river, first over boulder fields and then a rooty, steppy track through the forest — hard on the knees! Spectacular country, but we were glad to get to West Sabine Hut in the early afternoon. West Sabine Hut is pretty cramped but we managed to get those Europeans interested in playing Bang!, so we squeezed around the table and played eight-player Bang! all afternoon and early evening.

Late in the day a couple of DoC volunteers arrived to check hut tickets and update us on track news. The track to Blue Lake was known to be difficult due to washouts — a couple had gone up that afternoon and turned back — but some Te Araroa through-hikers had gone up and not come back so it seemed to be passable. The DoC volunteers were planning to try to get through the next day, as were we and a couple of others at the hut.

So on Wednesday we headed along the track by the West Sabine river towards Blue Lake. Much of it was washed out — replaced by new fast-flowing streams in many places — but we steadily made our way through the bush. Things got tricky when we reached a field of avalanche debris running down into the river. Previously there had been a track etched out among the scree and rocks, but that was gone. Worse, there was a big boulder at the edge of the river with water flowing fast around it, and the hillside above the boulder was a constant steep slope of fine scree. Some people ahead of us had successfully picked their way over the scree, and a couple of people in our group did too, but when I tried to follow I lost my footing and slid down the slope. My son behind me managed to grab my pack and haul me back so we both slid down into a shallow pool by the river ... otherwise I would have gone off the boulder into the deep end of the river. I scraped my elbow pretty well but things could have been worse. After that escape, we decided to give up on Blue Lake and turned back. On the way back we met the DoC volunteers heading up and gave them a full report. (We later learned that they made it to Blue Lake in six and half hours (usually three and a half hours), and were recommending a very wide/high detour around the avalanche field.)

With Blue Lake abandoned, we had to decide what to do with the two days we'd allocated to walk there and back. We had no desire to finish the walk earlier. We settled on lengthening our exit by staying at Sabine Hut and Speargrass Hut, then walking out to the car park but instead of going back to town, going up Mt Robert to stay at Bushline Hut for the last night, which has great views of Lake Rotoiti, St Arnaud and the surrounding valleys and mountains.

The walk from West Sabine Hut to Sabine Hut at Lake Rotoroa was pretty noneventful, just a few short track washouts to navigate around, though in many places the track has been seriously undermined and will need to be rerouted. Sabine Hut is in a beautiful location right at the lakeside, with great views across the lake. Probably not coincidentally, it also had by far the most sandflies outside of any hut on this trip. I had a nice swim there to clean off some of my accumulated grime.

The next day's walk to Speargrass Hut was lovely, but also uneventful. There's a boardwalk section through a swamp, and near the ridge over Speargrass there's a lookout with great views west to mountains. This was our shortest day of walking at just five hours. Speargrass Hut was interesting; there was a party of six Australian women plus another tramper, so the hut was full before evening. Then a couple of deer hunters showed up, and then a couple of very late-arriving European visitors, so they had to sleep on the floor. Speargrass Hut isn't big, so this was the fullest hut I've ever been in so far, but it wasn't a problem.

On the way to Speargrass I had come up with a new plan: instead of walking to the car park along the Speargrass Track we'd already traversed twice in our Angelus tramp, and then up to Bushline Hut, we could go up Speargrass valley again to the ridge and walk along Robert Ridge to Bushline, which would be new track with much better views. At the ridge above Speargrass Hut I was able to get enough cellphone signal to get a weather forecast which suggested the weather next morning would be excellent, just getting a bit windier and starting to rain in the afternoon.

So on Friday we got up earlier and left the hut around 7am to make the most of the good weather. We went up Speargrass valley again — our third time on this section of track in 9 days, but this time with perfect visibility to the mountain ranges in the west. This time we made it to the ridge in just two and a half hours, a full hour faster than our first ascent eight days earlier. There was some wind at the top but the conditions were indeed excellent. We also went pretty fast along Robert Ridge, getting to Bushline Hut in just another three and a half hours including a lunch stop. The views up there in all directions were, as advertised, spectacular.

We arrived at Bushline Hut around 1pm and played card games all afternoon. It was a great place for an extended visit, even though it did get cloudy and rainy in the afternoon. It's on the excellent Mt Robert loop track day walk, so we had numerous groups of day walkers sheltering in the hut during showers, who were fun to talk to. Drifting clouds and rain meant the view was constantly changing: one minute we'd have impenetrable fog, next minute a view all the way to the town just under cloud cover, another minute dramatic views of showers sweeping down the valleys across the lake. Because it was Friday night, Bushline Hut got overfull with people driving to the Mt Robert car park after work and walking up to get a head start on a weekend tramp to Angelus Hut. Again, though the hut was crowded, this was fun and not a problem. (Bushline Hut tip: one of the best views down into Lake Rotoiti can only be had from the toilets.)

Saturday was pretty uneventful — just an easy walk down Mt Robert on the extremely well-formed day-walk-standard track, then a walk around the lake back to town, about three hours altogether. Everyone enjoyed hearty meals for the first time in nearly a week.

I have some general observations to make about our Nelson Lakes tramps, but this blog post is already too long so I'll put them in a new one.

Nelson Lakes Tramping: Angelus Basin

On Thursday December 5 I flew down to Nelson with five friends and family members to spend ten days in the Nelson Lakes National Park, split into two tramping trips: a three-day trip to Angelus Hut, followed by a rest day and then a six-day trip around the Travers-Sabine circuit. Nelson Lakes is a mountainous area with lots of tracks: around lakes, along rivers and streams in bush-clad valleys, and some above the treeline over saddles and along ridges. Just before we arrived, the area was hit by a powerful storm that caused flash-flooding in the rivers and streams throughout the park. We just missed the storm itself, but the impact of the flooding was evident everywhere, including a lot of track wash-outs that made travel interesting, as I will describe later!

Upon arriving at the airport we got a shuttle straight to our trailhead at Mt Robert car park (with a brief stop to drop off some gear at the motel). We had intended to spend Thursday night at Bushline Hut near the top of Mt Robert, then on Friday proceed along Robert Ridge to our main destination, Angelus Hut among the peaks of the Travers Range. However relatively high winds (at least 50km/h) were predicted for the ridge on Friday, which combined with foggy and possibly rainy conditions meant we weren't comfortable with that option. Instead we chose the more sheltered Speargrass valley route: from the car park along Speargrass Creek to Speargrass Hut, then on Friday up the Speargrass valley and over the ridge to Angelus.

The Speargrass route worked out well. We didn't have any significant rain, but the track was dripping wet and the rivers and streams were roaring with water. At one point on the way to Speargrass Hut we had to cross a tricky scree pile beside Speargrass creek; the track had been completely eliminated by scree being washed down. Nevertheless we made it to Speargrass Hut in about three hours of pretty easy walking. There wasn't much of a view due to low-lying cloud, but the weather was otherwise clear. We spent the rest of the afternoon lazing around the hut, except for one person who discovered they really enjoyed chopping wood. Six other people joined us for the night (they were doing a one-night trip to Speargrass), making the 12-bunk hut exactly full. It was a bit cold so we got a fire going and had a great evening. Because this was only a three-day trip we had decent food — sausages and bread for dinner.

The next day, about 9am, we headed up the valley route to Angelus. It was fairly easy going, though the track crosses the creek many times, and the creek was quite high so we had to get wet feet. Again there wasn't much of a view but the cloud and mist had their own dramatic effect. As expected we were sheltered from the wind until we reached the ridge at the top of the valley, where indeed it was quite uncomfortable, but from there it was only a half-hour walk to Angelus Hut — crossing a couple of patches of snow, which was fun.

Angelus Hut is in an amazing spot, between a couple of lakes at the bottom of Angelus Basin, ringed by a ridge that still has a lot of snowy patches this early in the summer. It was a bit too cold to enjoy the outside deck, and unfortunately it was also quite cold inside the hut because they had just about run out of firewood so there was no fire. Having arrived in time for lunch, around 1pm, we had lunch and made endless hot chocolates to keep warm. Visibility was generally poor due to drifting fog, but late in the afternoon the clouds lifted to give us a clear view of the surrounding mountains. We took a walk around the smaller lake to enjoy the view and the alpine environment — absolutely magical.

During the day quite a number of other trampers arrived, mostly arriving via the Robert Ridge which had turned to out to be okay on the day. For some reason a lot of them were Israeli; apparently word about Angelus Hut has spread in the Israeli travel community. There was also a volunteer hut warden present, who gave a good hut talk in the evening while I cooked pasta, salami and tomato sauce for dinner.

I was bemused by the "BOIL WATER BEFORE DRINKING" signs in Angelus Hut (and in many other huts on our trip). I have almost never treated hut water anywhere in New Zealand and Angelus seemed like an unlikely place for the hut water to require treatment. Most huts I've visited have less strident warnings, along the lines of "the water is untreated but generally safe to drink; you may wish to treat it before drinking". We ended up drinking hut water — and flowing stream and river water — untreated everywhere during our trip, to no ill effects. I hope this is not a new trend towards more risk-averse — and less informative — signage.

It was a beautiful moonlight night in the Angelus Basin, though not clear enough to see many stars unfortunately, and I got up and stood outside in the cold for a while to soak up the atmosphere and enjoy the majesty of God's creation.

We were hoping to take the Robert Ridge route to Mt Robert and then back to town on Saturday, but the warden got a weather forecast in the morning that made that sound inadvisable. So we returned via the same Speargrass route we came up, walking all the way back to the car park and then following the road back into town to check into our motel. It was a longish day but not too hard and we had an excellent dinner at a local restaurant. On Sunday one of our group return to Auckland while the rest of us rested and prepared for the next leg of our trip — which deserves its own blog post.

All in all, a little disappointing that we didn't get to walk the Robert Ridge, but a great tramp regardless.

Your Debugger Sucks

Author's note: Unfortunately, my tweets and blogs on old-hat themes like "C++ sucks, LOL" get lots of traffic, while my messages about Pernosco, which I think is much more interesting and important, have relatively little exposure. So, it's troll time.

TL;DR Debuggers suck, not using a debugger sucks, and you suck.

If you don't use an interactive debugger then you probably debug by adding logging code and rebuilding/rerunning the program. That gives you a view of what happens over time, but it's slow, can take many iterations, and you're limited to dumping some easily accessible state at certain program points. That sucks.

If you use a traditional interactive debugger, it sucks in different ways. You spend a lot of time trying to reproduce bugs locally so you can attach your debugger, even though in many cases those bugs have already been reproduced by other people or in CI test suites. You have to reproduce the problem many times as you iteratively narrow down the cause. Often the debugger interferes with the code under test so the problem doesn't show up, or not the way you expect. The debugger lets you inspect the current state of the program and stop at selected program points, but doesn't track data or control flow or remember much about what happened in the past. You're pretty much stuck debugging on your own; there's no real support for collaboration or recording what you've discovered.

If you use a cutting-edge record and replay debugger like rr, it sucks less. You only have to reproduce the bug once and the recording process is probably less invasive. You can reverse-execute for a much more natural debugging experience. However, it's still hard to collaborate, interactive response can be slow, and the feature set is mostly limited to the interface of a traditional debugger even though there's much more information available under the hood. Frankly, it still sucks.

Software developers and companies everywhere should be very sad about all this. If there's a better way to debug, then we're leaving lots of productivity — therefore money — on the table, not to mention making developers miserable, because (as I mentioned) debugging sucks.

If debugging is so important, why haven't people built better tools? I have a few theories, but I think the biggest reason is that developers suck. In particular, developer culture is that developers don't pay for tools, especially not debuggers. They have always been free, and therefore no-one wants to pay for them, even if they would credibly save far more money than they cost. I have lost count of the number of people who have told me "you'll never make money selling a debugger", and I'm not sure they're wrong. Therefore, no-one wants to invest in them, and indeed, historically, investment in debugging tools has been extremely low. As far as I know, the only way to fix this situation is by building tools so much better than the free tools that the absurdity of refusing to pay for them is overwhelming, and expectations shift.

Another important factor is that the stagnation of debugging technology has stunted the imagination of developers and tool builders. Most people have still never even heard of anything better than the traditional stop-and-inspect debugger, so of course they're not interested in new debugging technology when they expect it to be no better than that. Again, the only cure I see here is to push harder: promulgation of better tools can raise expectations.

That's a cathartic rant, but of course my ultimate point is that we are doing something positive! Pernosco tackles all those debugger pitfalls I mentioned; it is our attempt to build that absurdly better tool that changes culture and expectations. I want everyone to know about Pernosco, not just to attract the customers we need for sustainable debugging investment, but so that developers everywhere wake up to the awful state of debugging and rise up to demand an end to it.

Supercharging Gdb With Pernosco

At Pernosco we don't believe that gdb (or any other traditional debugger) has the ideal debugging interface, but users familiar with gdb can get started with Pernosco more easily if gdb is available. Also, gdb has many useful features and it will be a long time before Pernosco matches every single one of those features. Therefore we've integrated gdb into Pernosco. Moreover, leveraging Pernosco's infrastructure makes our gdb experience clearly the best gdb experience ever — in my opinion!

The best feature of Pernosco-gdb is speed. We built a gdbserver that extracts information from the Pernosco omniscient database, so forward or reverse execution in gdb is simply a matter of updating an internal timestamp and getting results for that time whenever gdb asks for information. Our users routinely skip forward or backward through minutes of application execution with a second or two delay in the UI. Achieving this is harder than you might think; gdb uses internal breakpoints to observe, e.g., loading and unloading of shared libraries, so we need tricks to avoid unnecessary internal stops.

I'm also proud of the synchronization we achieved between gdb and the rest of Pernosco. Our gdb sessions track changes to the current moment and stack frame made by other parts of Pernosco (including other gdb sessions!), and likewise gdb "execution" and changes to the current stack frame are reflected in the rest of Pernosco. (Pernosco stack walking is considerably more robust than gdb's, so synchronization can fail when gdb disagrees about what stack frames exist.) Starting a gdb session attaches it to the current process, so you can have multiple gdb sessions open, for the same process and/or different processes, all synchronized as much as possible.

Pernosco enables some nice improvements to watchpoints and breakpoints, as described here.

Deploying debugging as a cloud service enables some other improvements in the gdb experience. Gdb sessions run on our servers and users can run arbitrary code in them, so by necessity we put each session in a very tight sandbox. This means we don't have to worry about potentially malicious gdbinit scripts, DWARF debuginfo, etc; we can just configure gdb to trust everything.

Another benefit of the Pernosco model is that we take responsibility for configuring gdb optimally and keeping it up to date. For example we ensure .gdb_index files are always built.

We achieve all this with minimal changes to upstream gdb. Currently our only change is to disable gdb's JIT debugging support, because that creates a lot of usually-unnecessary internal breakpoint stops. We publish the sources for our build of gdb (though technically we don't have to, as gdb is not AGPL).

The Power Of Collaborative Debugging

An under-appreciated problem with existing debuggers is that they lack first-class support for collaboration. In large projects a debugging session will often cross module boundaries into code the developer doesn't understand, making collaboration extremely valuable, but developers can only collaborate using generic methods such as screen-sharing, physical co-location, copy-and-paste of terminal transcripts, etc. We recognized the importance of collaboration, and Pernosco's cloud architecture makes such features relatively easy to implement, so we built some in.

The most important feature is just that any user can start a debugging session for any recorded execution, given the correct URL (modulo authorization). We increase the power of URL sharing by encoding in the URL the current moment and stack frame, so you can copy your current URL, paste it into chat or email, and whoever clicks on it will jump directly to the moment you were looking at.

The Pernosco notebook takes collaboration to another level. Whenever you take a navigation action in the Pernosco UI, we tentatively record the destination moment in the notebook with a snippet describing how you got there, which you can persist just by clicking on. You can annotate these snippets with arbitrary text, and clicking on a snippet will return to that moment. Many developers already record their progress by taking notes during debugging sessions (I remember using Borland Sidekick for this when I was a kid!); the Pernosco notebook makes this much more convenient. Our users find that the notebook is great for mitigating the "help, I'm lost in a vast information space" problem that affects Pernosco users as well as users of traditional debuggers (sometimes more so in Pernosco, because it enables higher velocity through that space). Of course the notebook persists indefinitely and is shared between all users and sessions for the same recording, so you have a permanent record of what you discovered that your colleagues can also explore and add to.

Our users are discovering that these features unlock new workflows. A developer can explore a bug, recording what they've learned in the code they understand, then upon reaching unknown code forward the debugging session to a more knowledgeable developer for further investigation — or perhaps just to quickly confirm a hypothesis. We find that, perhaps unexpectedly, Pernosco can be most effective at saving the time of your most senior developers because it's so much easier to leverage the debugging work already done by other developers.

Collaboration via Pernosco is beneficial not just to developers but to anyone who can reproduce bugs and upload them to Pernosco. Our users are discovering that if you want a developer to look into a bug you care about, submitting it to Pernosco yourself and sending them a link makes it much more likely they will oblige — if it only takes a minute or two to start poking around, why not?

Extending this idea, Pernosco makes it convenient to separate reproducing a bug from debugging a bug. It's no problem to have QA staff reproduce bugs and submit them to Pernosco, then hand Pernosco URLs to developers for diagnosis. Developers can stop wasting their time trying to replicate the "steps to reproduce" (and often failing!) and staff can focus on what they're good at. I think this could be transformative for many organizations.

Omniscient Printf Debugging In Pernosco

Pernosco supports querying for the execution of specific functions and the execution of specific source lines. These resemble setting breakpoints on functions or source lines in a traditional debugger. Traditional debuggers usually let you filter breakpoints using condition expressions, and it's natural and useful to extend that to Pernosco's execution views, so we did. In traditional debuggers you can get the debugger to print the values of specified expressions when a breakpoint is hit, and that would also be useful in Pernosco, so we added that too.

These features strongly benefit from Pernosco's omniscient database, because we can evaluate expressions at different points in time — potentially in parallel — by consulting the database instead of having to reexecute the program.

These features are relatively new and we don't have much user experience with them yet, but I'm excited about them because while they're simple and easily understood, they open the door to "query-based debugging" strategies and endless possibilities for enhancing the debugger with richer query features.

Another reason I'm excited is that together they let you apply "printf-debugging" strategies in Pernosco: click on a source line, and add some print-expressions and optionally a condition-expression to the "line executions" view. I believe that in most cases where people are used to using printf-debugging, Pernosco enables much more direct approaches and ultimately people should let go of those old habits. However, in some situations some quick logging may still be the fastest way to figure out what's going on, and people will take time to learn new strategies, so Pernosco is great for printf-debugging: no rebuilding, and not even any reexecution, just instant(ish) results.

The BBC's "War Of The Worlds"

Very light spoilers ahead.

I had hopes for this show. I liked the book (OK, when I read it >30 years ago). I love sci-fi. I like historical fiction. I was hoping for Downton Abbey meets Independence Day. Unfortunately I think this show was, as we say in NZ, "a bit average".

I really liked the characters reacting semi-realistically to terror and horror. It always bothers me that in fiction normal people plunge into traumatic circumstances, scream a bit, then get over it in time for the next scene. This War Of the Worlds takes time to show characters freaking out, resting and consoling one another, but not quite getting it all back together. Overall I thought the acting was well done.

I think the pacing and editing were poor. Some parts were slow, but other parts (especially in the first half) lurch from scene to scene so quickly it feels like important scenes were cut. It was hard to work out was going on geographically.

Some aspects seemed pointlessly complicated or confusing, e.g. the spinning ball weapon.

Call me old-fashioned, but when a man abandons his wife I am not, by default, sympathetic to him, so I spent most of the show thinking our male protagonist is kind of a bad guy, when I'm clearly supposed to be siding with him against closed-minded society. I even felt a bit vindicated when towards the end his lover Amy wonders if they did the right thing. At least for a change the Christian-esque character was only a fool, not a psychopath, so thank God for small mercies.

I guess I'm still waiting for the perfect period War Of The Worlds adaptation.

Explaining Dataflow In Pernosco

Tracing dataflow backwards in time is an rr superpower. rr users find it incredibly useful to set hardware data watchpoints on memory locations of interest and reverse-continue to find where those values were changed. Pernosco takes this superpower up a level with its dataflow pane (see the demo there).

From the user's point of view, it's pretty simple: you click on a value and Pernosco shows you where it came from. However, there is more going on here than meets the eye. Often you find that the last modification to memory is not what you're looking for; that the value was computed somewhere and then copied, perhaps many times, until it reached the memory location you're inspecting. This is especially true in move-heavy Rust and C++ code. Pernosco detects copying through memory and registers and follows dataflow backwards through them, producing an explanation comprising multiple steps, any of which the user can inspect just by clicking on them. Thanks to omniscience, this is all very fast. (Jeff Muizelaar implemented something similar with scripting gdb and rr, which partially inspired us. Our infrastructure is a lot more powerful what he had to work with.)

Pernosco explanations terminate when you reach a point where a value was derived from something other than a CPU copy: e.g. an immediate value, I/O, or arithmetic. There's no particular reason why we need to stop there! For example, there is obviously scope to extend these explanations through arithmetic, to explore more general dataflow DAGs, though intelligible visualization would become more difficult.

Pernosco's dataflow explanations differ from what you get with gdb and rr in an interesting way: gdb deliberately ignores idempotent writes, i.e. writes to memory that do not actually change the value. We thought hard about this and decided that Pernosco should not ignore them. Consider a trivial example:

x = 0;
y = 0;
x = y;

If you set a watchpoint on x at the end and reverse-continue, gdb+rr will break on x = 0. We think this is generally not what you want, so a Pernosco explanation for x at the end will show x = y and y = 0. I don't know why gdb behaves this way, but I suspect it's because gdb watchpoints are sometimes implemented by evaluating the watched expression over time and noting when the value changes; since that can't detect idempotent writes, perhaps hardware watchpoints were made to ignore idempotent writes for consistency.

An interesting observation about our dataflow explanations is that although the semantics are actually quite subtle, even potentially confusing once you dig into them (there are additional subtleties I haven't gone into here!), users don't seem to complain about that. I'm optimistic that the abstraction we provide matches user intuitions closely enough that they skate over the complexity — which I think would be a pretty good result.

(One of my favourite moments with rr was when a Mozilla developer called a Firefox function during an rr replay and it printed what they expected it to print. They were about to move on, but then did a double-take, saying "Errrr ... what just happened?" Features that users take for granted but are actually mind-boggling are the best features.)

Improving Debugging Workflow With Pernosco

One of the key challenges for debuggers is that the traditional interactive debugging workflow — running your program interactively and starting it under the debugger or connecting to it once it's running, and pausing it to inspect its state — doesn't work well for a lot of people anymore. That workflow isn't convenient when the application normally doesn't run locally — e.g. because testing more often happens in CI, or on a phone, or the code you care about runs as part of a big distributed system. It also falls down when pausing the debuggee breaks the system. As software has increasingly moved to the cloud and mobile platforms, this has become a bigger deal and it's no wonder use of interactive debugging has waned. "Remote debugging" helps a bit, but it tends to be painful and although it can bridge gaps between machines, it doesn't bridge gaps in time.

We've published a couple of documents on how Pernosco tackles this, in particular how Pernosco integrates with CI and how Pernosco supports uploads from developers and QA (manual and automatic). A big part of the solution is just record-and-replay (with rr in our case). Being able to record execution on one machine, without stopping the application, and replay execution on another machine at another time, enables a lot of new workflows that mitigate the above problems. However Pernosco goes further in some important ways.

One issue is that just being able to replay execution isn't enough; we also want a good debugging experience during the replay. This means we need to capture compiled debuginfo, source code and other relevant information that aren't strictly necessary for the replay. In many cases that data isn't even available at the recording site, but it might be available somewhere (e.g. a symbol server or build artifact archive) for us to get later. So our debugging infrastructure has to support collecting information at the recording site, harvesting it from various sources later, and actually using it during the debugging session. This is not at all trivial, and Pernosco has a lot of code to handle this sort of thing, some of which needs to be customized for specific customers. For example, Pernosco identifies Firefox binaries built by Mozilla CI and knows how to locate the relevant symbols and sources from Mozilla's archives. For developer and QA-submitted recordings, Pernosco examines the trace to locate relevant debuginfo and source code and upload them. For source code hosted in well-known public repositories (e.g. mozilla-central or Github), we minimize overhead by uploading only local changes and having our debugger client fetch the public changes from the public repository at debugging time.

Note that rr on its own provides trace portability but debugging ported traces is tricky. With rr pack and rr record --disable-cpuid-features, it is generally possible to create rr recordings that can be replayed on other machines. However, when you replay with gdb, locating symbols and source files is problematic when the replay machine filesystem does not exactly match the recording machines. For example when gdb sees the shared-library loader load /home/roc/libfoo.so, that file might not be present at that location on the replay machine (or worse, it might be a different version) so gdb won't load the right symbols. You can try to work around this by populating a "sysroot" directory with the relevant files, copied and renamed from the trace, but figuring out which trace files need to go where is hard (because e.g. it depends on the symlinks present on the recording machine, which rr doesn't capture in the recording, and it's not even clear how you'd do that).

Another important feature for enabling new workflows is just having a cloud-based Web client. We want to minimize the barrier to getting into a debugging session, and it's hard to think of an easier way than publishing a link which the user clicks on to enter a specific debugging session — no installation, no configuration. Those links can be published wherever you already notify users about test failures.

One thing I'm really excited about is that Pernosco enables splitting failure reproduction from debugging. Traditionally, developers had to reproduce a bug locally when they wanted to use an interactive debugger to debug it. Pernosco lets you delegate the reproduction step to other people (or automation). For example, when QA staff find a bug, instead of writing down the steps to reproduce to send to a developer (and inevitably having a back-and-forth discussion about exactly what's required to reproduce the bug, etc), QA can upload a recording to Pernosco and pass the link to the developer. This saves time and money — especially when QA staff are cheaper and/or more scalable then your developer team.

Auckland Half Marathon 2019

I ran the Auckland Half Marathon last Sunday. My time was 1:46:51, a little slower than last year. I didn't quite have the mental endurance I guess. As always I hope I'll do better next year, though I am getting older...

As usual, I ran barefoot. This year I had climbing tape again which definitely makes it a bit easier on my feet at this speed.

People ask why I run barefoot. I didn't start running until I was about 35, and I started running on a beach where I'm usually barefoot. After I got used to that, it never felt right to run in shoes ... they're just weight on my feet. To be honest, I also like being a bit eccentric. I've never had any serious issues with injuries so at this point, I don't feel like changing anything. I did try wearing Vibram 5-Finger foot-gloves for my first half-marathon, but my feet got all sweaty and were still sore at the end so they don't seem to help me much. Sticking small patches of climbing tape to the hardest-wearing parts of my feet (the balls, and my second toes) is plenty of protection when running fast. When running slower (20K training runs around 2:10) I don't even need that.

Pernosco Demo Video

Over the last few years we have kept our work on the Pernosco debugger mostly under wraps, but finally it's time to show the world what we've been working on! So, without further ado, here's an introductory demo video showing Pernosco debugging a real-life bug:

This demo is based on a great gdb tutorial created by Brendan Gregg. If you read his blog post, you'll get more background and be able to compare Pernosco to the gdb experience.

Pernosco makes developers more productive by providing scalable omniscient debugging — accelerating existing debugging strategies and enabling entirely new strategies and features — and by integrating that experience into cloud-based workflows. The latter includes capturing test failures occurring in CI so developers can jump into a debugging session with one click on a URL, separating failure reproduction from debugging so QA staff can record test failures and send debugger URLs to developers, and letting developers collaborate on debugging sessions.

Over the next few weeks we plan to say a lot more about Pernosco and how it benefits software developers, including a detailed breakdown of its approach and features. To see those updates, follow @_pernosco_ or me on Twitter. We're opening up now because we feel ready to serve more customers and we're keen to talk to people who think they might benefit from Pernosco; if that's you, get in touch. (Full disclosure: Pernosco uses rr so for now we're limited x86-64 Linux, and statically compiled languages like C/C++/Rust.)

Food In Auckland 2019

Some places I like these days that are (mostly) pretty close to my house:

• Jade Town (Dominion Road): Uighur food. A bit Chinese, a bit Indian, but not much like either.
• Cypress (Dominion Road): New-ish yum cha (dim sum) place. Some interesting dishes I'd never seen before.
• Viet Kitchen (Dominion Road): More authentic, bit more expensive Vietnamese.
• Hot And Spicy Pot (Dominion Road, city): tasty pay-per-weight stir-fry.
• Barilla (Dominion Road): Decent cheap dumplings and other Chinese food.
• Tombo (Newmarket): Quality Korean-Japanese BBQ/hotpot buffet.
• Hansan (Newmarket, city): Still a favourite cheap Vietnamese-Chinese place.
• Master Dumpling (Newmarket): good dumplings, great sweet potato in melted sugar.
• Faro (Newmarket): Nice Korean lunch combo.
• Momotea (Newmarket): Taiwanese-style cafe with strong frozen drink selection.
• Selera (Newmarket): Malaysian cafe food.
• Sun World (Newmarket): Reliably good yum cha.
• Kimchi Project (city): Kimchi fusion-eseque place. Kimchi carbonara is great.
• BBQ Duck Cafe (city): Good value tasty Hong Kong-style cafe.
• Nol Bu Ne (city): Great value Korean food.
• Uncle Man's (city): Malaysian food — best roti around IMHO.
• Kiin (Mt Eden): Great cheap Thai food.
• Altar (Mt Eden): Good value sandwhich+fries lunch special.
• Corner Burger (Mt Eden): Great burger+fries+shake combo — the shakes are especially good.
• Gangnam Style (Takapuna): Korean BBQ buffet. The worst thing about this place is the name.
• Petra Schwarma (Kingsland): Really nice Jordanian food.
• Mama Rich (Greenlane): Good Malaysian cafe food, a bit cheaper than Selera.
• Chocolate Boutique (Parnell): Not exactly a restaurant but for dessert, a great option.

Pouakai Circuit

Last weekend I went with a couple of people down to Mt Taranaki to tramp the Pouakai Circuit, planning to do it leisurely over three days. We drove down on Saturday morning (about 5 hours from Auckland), had a fine lunch at the Windsor Cafe in Inglewood, and hiked from a carpark up to Holly Hut via the Kokowai Track. (Normally you'd take the Holly Track from the Egmont Visitor's Centre but that track is currently closed due to a slip, though it should be open soon.) The posted time was 4.5 hours but we did it in 3 hours; my companions are very fast, and I'm not too bad myself though the steep uphill with steps made me struggle a bit! The weather was great and we had some excellent views of Mt Taranaki along the way. It's a beautiful mountain with snow cover on a clear winter's day.

Holly Hut is a fine hut. Some generous donors installed LED lighting, which is great during the winter when the days are short. It was the first Saturday of the school holidays but there were just two other people there. Late Saturday afternoon we did the ~1 hour return side trip to Bell Falls, which were beautiful. The weather had gotten cloudy, drizzly and misty and the daylight was fading, but that added to the effect!

On Sunday morning we had a bit of a late start because we thought we'd spend Sunday night at Pouakai Hut, which is nominally 2.5 hours walk away. The track crosses Ahukawakawa Swamp, in a basin between the old Pouakai volcano and the slopes of Mt Taranaki — most picturesque, especially in drizzly foggy weather. We had been hoping to do the side track to the Pouakai summit on the way to the hut, but bursts of rain and the likelihood of seeing nothing but cloud dissuaded us. We got to Pouakai Hut in about 1.5 hours and had an early lunch, then had to think hard: with phone reception at the hut, we got a new weather forecast for Monday, which promised heavy rain, wind, lower temperatures and possible thunderstorms — even less promising than the rain currently beating against the hut. We decided to avoid that weather by finishing the circuit on Sunday and driving back to Auckland late. That plan worked out pretty well; it didn't rain much during the rest of Sunday and although our fast pace left us all feeling a bit tired when we got back to the carpark, we had fun and got back to Auckland before midnight. All in all, a short trip with lots of driving and lots of fast walking, but still a great chance to appreciate God's creation.

I'm looking forward to doing this trip again when the weather's better and the Holly Track is open! It's quite accessible from Auckland, and we could do it with less fit people if we take more time. We could also save some energy by starting at the Egmont Visitor's Centre at the top end of the road, then taking the Holly Track to Holly Hut, then returning via Pouakai Hut to the lower carpark and sending some fitter people up the road to fetch the car.

Is Richard Dawkins A Moral Realist?

An interview with Richard Dawkins in the New Scientist (21 September 2019) contains this exchange:

Graham Lawton: Another chapter in your book looks at progress in moral issues such as gender and racial equality, and you present a very upbeat picture. Do you worry that progress has gone into reverse?
Richard Dawkins: No. It's important to take the long view. I think there's absolutely no doubt that we're getting better as the centuries go by. The moral standards of a 21st century person are significantly different from those of a 20th century person.

Dawkins here seems to assume there are objective moral standards against which human moral opinions can be measured, i.e. moral realism. This surprised me because Dawkins is such a strong advocate for naturalism and it has always seemed obvious to me (including before I became a Christian) that naturalism is incompatible with moral realism — the famous is-ought gap. Sean Carroll, for example, has written about this much better than I could. In fact, Dawkins has apparently written (in River Out of Eden, quoted here):

The universe we observe has precisely the properties we should expect if there is at the bottom, no design, no purpose, no evil and no good.

so unless he's changed his mind about that, it seems Dawkins at least professes to be a moral anti-realist.

This confusion illustrates why I've always found moral anti-realism so deeply unsatisfactory in real life. One can argue that there are no objective moral facts, and that moral claims simply express opinions shaped by evolution and culture etc, and even try to believe those things — but the temptation to think, speak and act as moral realists seems practically irresistible ... so much so that even the most prominent moral anti-realists consistently yield to it, and hardly anyone even notices.

Addendum Arguably the New Scientist quote could be interpreted in other ways, e.g. that by "getting better" Dawkins meant more internally consistent, or more in accordance with his personal subjective moral opinions. However I think it's obvious most people would interpret it as "objectively morally better" and thus if Dawkins meant something else, he needs to work a lot harder at eliminating such misleading language.

Dissatisfied With Docker

I am not satisfied with Docker.

Untrusted users should be able to run their own container instances. Running a basic container instance means forking a process, putting it in the right kernel namespaces, and setting up mounts and virtual network interfaces, and those can all be done without privileges. Unfortunately, in Docker, access to the Docker daemon is tantamount to being root. Fixing this would improve both convenience and security.

In fact, a global system daemon should not be needed. Either users should be able to run their own daemons or container management should avoid having a daemon at all, by storing container state in a shared database.

Docker container builds are too slow. Installing a container image requires piping massive amounts of image data to the daemon over a socket, which is ridiculous. This could be avoided by passing a file descriptor to the container manager ... or you might even get rid of container image management and start containers by pointing directly to the files comprising the image.

Docker container instances start too slowly. It shouldn't take seconds to start or stop a simple small container. The system calls required to start a simple container run in milliseconds.

No doubt these observations are old news, so I assume there are better container implementations out there. Which one do I want?

Cape Brett 2019

It's the school holidays so I took one of my children and one of my friends (a young adult) for a tramping trip to Cape Brett Hut on Monday and Tuesday. It's nominally an eight-hour walk each way. The hut used to be a lighthouse-keeper's house and is in a spectacular setting right at the end of the Cape, on a grassy slope with the ocean on three sides. The walking track is through lovely bush with great views north to the Bay of Islands and south along the Tutukaka coast. It's an excellent trip, just a few hours drive from Auckland and because it's up north and coastal, it's good to do during the winter. The hut is almost never fully booked, perhaps because the walk is quite long and arduous (compared to other walks in the area).

I went there eight years ago with a group. That time we took a water taxi to the hut and a water taxi carried our packs out while we walked back along the track. This time I wanted to do it "properly": walking with our supplies both ways. The nature of the group was also quite different; eight years ago the group was much larger and with a greater variation of fitness, while this time there was just the three of us and we're all pretty fit (but I'm eight years older than last time!).

We stayed at Whangaruru Beachfront Camp on Sunday night so we could get to the trailhead early on Monday and start walking around 7:45am, not long after sunrise at 7:30am. The days are short and I wanted to reach the hut with plenty of daylight to enjoy the destination. We ended up being pretty fast and got to the hut in about six hours, just before 2pm! It's a tough track, with lots of steep uphills and downhills, and we all felt a fair bit of soreness in our legs. Nevertheless I was pretty pleased with our speed and my ability to keep up with the younger people. The weather was great both days — mild, mostly cloudy, and a light breeze in places — and the scenery was brilliant. We had time to rest in the sun and explore the end of the Cape before it got dark, then we cooked a tasty meal.

Around 8pm, when it had been fully dark for a while, a couple arrived at the hut. They told us they'd walked to the hut in just five hours, most of that in the dark. That deflated my pride a bit!

During the night the skies cleared and the moon set, giving an excellent view of the stars through the windows next to my bunk.

On Tuesday we again got up pretty early, around 7am. The sun hadn't risen but we'd already been in our bunks for ten hours. We got to see a lovely sunrise over the ocean. We left the hut at 8:20am and this time finished the walk out in just five hours and twenty minutes. Perhaps surprisingly, I felt a lot better on the second day than the first, and so did the others. Our packs were a little lighter, but I think the previous day's workout had made us all a bit fitter. I found it exhilarating grinding up steep hills without pausing and then stretching the legs for a fast walk along the flat or slightly downhill, and I also felt more agile on the steep downhill sections.

This was a great trip and I really feel thankful to God for the privilege of being able to it. I look forward to doing it again with other people; the walk isn't for everyone, but there's always the water taxi option.

Auckland Rust Meetup: "Building An Omniscient Debugger In Rust"

I gave this month's Auckland Rust Meetup talk: a very high-level overview of Pernosco's architecture and then a dive into some superficial metrics about the project, comments on the third-party crates we use, and some thoughts about the pros and cons of Rust for this project. I apologise for the slides being thrown together in a hurry, and they're probably a bit hard to follow without my commentary.

Stack Write Traffic In Firefox Binaries

For people who like this sort of thing...

I became interested in how much CPU memory write traffic corresponds to "stack writes". For x86-64 this roughly corresponds to writes that use RSP or RBP as a base register (including implicitly via PUSH/CALL). I thought I had pretty good intuitions about x86 machine code, but the results surprised me.

In a Firefox debug build running a (non-media) DOM test (including browser startup/rendering/shutdown), Linux x86-64, non-optimized (in an rr recording, though that shouldn't matter):

Base register	Fraction of written bytes
RAX	0.40%
RCX	0.32%
RDX	0.31%
RBX	0.01%
RSP	53.48%
RBP	44.12%
RSI	0.50%
RDI	0.58%
R8	0.01%
R9	0.00%
R10	0.00%
R11	0.00%
R12	0.00%
R13	0.00%
R14	0.00%
R15	0.00%
RIP	0.00%
RDI (MOVS/STOS)	0.25%
Other	0.00%
RSP/RBP	97.59%

Ooof! I expected stack writes to dominate, since non-opt Firefox builds have lots of trivial function calls and local variables live on the stack, but 97.6% is a lot more dominant than I expected.

You would expect optimized builds to be much less stack-dominated because trivial functions have been inlined and local variables should mostly be in registers. So here's a Firefox optimized build:

Base register	Fraction of written bytes
RAX	1.23%
RCX	0.78%
RDX	0.36%
RBX	2.75%
RSP	75.30%
RBP	8.34%
RSI	0.98%
RDI	4.07%
R8	0.19%
R9	0.06%
R10	0.04%
R11	0.03%
R12	0.40%
R13	0.30%
R14	1.13%
R15	0.36%
RIP	0.14%
RDI (MOVS/STOS)	3.51%
Other	0.03%
RSP/RBP	83.64%

Definitely less stack-dominated than for non-opt builds — but still very stack-dominated! And of course this is not counting indirect writes to the stack, e.g. to out-parameters via pointers held in general-purpose registers. (Note that opt builds could use RBP for non-stack purposes, but Firefox builds with -fno-omit-frame-pointer so only in leaf functions, and even then, probably not.)

It would be interesting to compare the absolute number of written bytes between opt and non-opt builds but I don't have traces running the same test immediately at hand. Non-opt builds certainly do a lot more writes.

Winter Tramp: Waihohonu Hut To Tama Lakes

New Zealand's Tongariro National Park is one of my favourite places. We had a three-day weekend so I drove three friends and family down for a two-night stay at Waihohonu Hut, surely the grandest public hut in New Zealand, and we enjoyed the park in a wintry setting ... an interesting change from our previous visits.

We left Auckland around 7am on Saturday to avoid traffic — often hordes of people leave Auckland for long weekends — but there was no congestion. After stopping for lunch in Turangi we reached the trailhead on the Desert Road shortly before 1pm. The wind was cold and there was thick low-lying cloud, but it wasn't snowing ... yet. From there the walk to Waihohonu Hut is easy in less than two hours, on a good quality track with a very gentle upward slope. Much of the route is very exposed but the wind wasn't as high as forecast and we were well equipped. Towards the end it started snowing gently, but that was fun and we got to the hut in high spirits before 3pm. The hut is well insulated and other trampers had arrived before us and got the fire going, and the LED lighting was on, so it was cosy. We talked to them, made popcorn, watched the snow fall, played some card games and enjoyed the rest of the afternoon and evening as more trampers trickled in.

I had wondered how full the hut would get. There are 28 bunks, but it's off-season so they can't be booked, and given the public holiday potentially a lot of people could have turned up. As it happened about 35 people ended up there on Saturday night — many people tramping in from the Desert Road just to visit Waihohonu, like us, but also quite a few doing round trips from Whakapapa or even doing the Tongariro Northern Circuit (which requires alpine skills at this time of year). People grabbed bunks as they arrived, and the rest slept on spare mattresses in the common room, which was actually a fine option. The only problem with sleeping in the common room is people staying up late and (probably other) people coming in early for breakfast. Even though it was technically overfull, Waihohonu Hut's common areas are so spacious that at no time did it ever feel crowded.

On Sunday morning there was a bit more snow on the ground, some low cloud and light snow falling. I was hoping to walk west from the hut to the Tama Saddle, which separates Mt Ruapehu to the south from Mts Tongariro and Ngauruhoe to the north, and visit the Lower Tama Lake just north of the saddle. It was unclear what conditions were going to be like but the forecast had predicted snow would stop falling in the morning, and we were well equipped, so we decided to give it a go. The expected walking time was about six and a half hours and we left before 9am so we had plenty of time. In the end it worked out very well. The cloud lifted quickly, except for some tufts around Ruapehu, and the snow did stop falling, so we had stunning views of the mountains the whole day. We were the first walkers heading west that day so we walked through fresh snow, broke the ice of frozen puddles and streams, and saw the footprints of rabbits and other animals, and relished the pristine wintry environment. It's the first time I've done a long-ish walk in the snow in the wilderness like this, and it was magnificent! I'm so grateful we had the chance to be there and that the weather turned out well.

As we got close to the saddle the snow was thicker, up to our knees in a few places, and the wind got stronger, and at the Lower Tama Lake it was quite cold indeed and blowing steadily from the east. I was a bit worried about having to walk back into that wind, and there was still the possibility of a change in the weather, so even though we were ahead of schedule I decided after lunch above the lake we should head back to Waihohonu rather than carrying on up to Upper Tama Lake (where no doubt the views would have been even better, but the wind would have been even colder!). Interestingly though, we were far from alone; many people, mostly foreign tourists, had walked to the lakes from Whakapapa (on the western side of Ruapehu), a shorter walk, and even headed up the ridge to the upper lake. As it turned out, our walk back was pretty easy. The wind mostly died away and the sun even came out.

We got back to Waihohonu about 3:30pm and once again relaxed at the hut for the rest of the afternoon, catching up with the trampers who were staying both nights and meeting new arrivals. That night the hut was again overfull but only by a couple of people, and again that wasn't a problem.

This morning (Monday) the sky was completely clear, giving magnificent views of snow-covered Ngauruhoe and Ruapehu through the hut's huge picture windows. A thick frost on the ground combined with the snow to form a delightfully crunchy surface for our walk back to the car park. I for one kept turning around to take in the incredible views. It was a very pleasant drive back in the sun through the heart of the North Island, but I can't want to go tramping again!