Static Customization Of Function Signatures In Rust

Sometimes I have a big function that does a lot, and in new code I need to do almost the same thing, but slightly differently. Often the best approach is to call the same function but add parameters (often described as "options") to select specific variations. This can get ugly when the function has optional outputs — results that are only produced when certain options were passed in — because typically there is the possibility of an error when code looks for an output (e.g. unwraps a Rust Option) at a site that did not request it. It would be great if the compiler could check that you only use an output when you passed in the option to enable it. Fortunately, some simple Rust coding patterns let us achieve this.

Here's an example simplified from some Pernosco code. We have a function that pretty-prints a data structure at a specific moment in time, which takes an optional parameter specifying another time to render the change in values between the two times. If that optional parameter is specified, the function returns an additional result — whether or not there was any difference in the two values. The naive signature would look something like this:

fn print(&self, moment: Moment, other: Option<Moment>)
-> (String, Option<Difference>) {
  ...
  let difference = other.map(|moment| ...);
  ...
  (..., difference)
}
...
let (string1, difference1) = value.print(moment, None);
let (string2, difference2) = value.print(moment, Some(other_moment));
println!("{:?}", difference1.unwrap()); // PANIC
println!("{:?}", difference2.unwrap());

It is possible to misuse this function by passing None in other and then expecting to find a meaningful value in the second part of the result pair. We'd probably catch it in tests, but it would be good to catch it at compile time. There is also a small efficiency issue: passing None for other is a bit less efficient than calling a customized version of print that has been optimized to remove other and the Difference result.

Here's a way to avoid those problems:

trait PrintOptionalMoment {
  type PrintOptionalDifference;
  fn map<F>(&self, closure: F) -> Self::PrintOptionalDifference
     where F: FnOnce(Moment) -> Difference;
}
impl PrintOptionalMoment for () {
  type PrintOptionalDifference = ();
  fn map<F>(&self, closure: F) -> Self::PrintOptionalDifference
     where F: FnOnce(Moment) -> Difference {}
}
impl PrintOptionalMoment for Moment {
  type PrintOptionalDifference = Difference;
  fn map<F>(&self, closure: F) -> Self::PrintOptionalDifference
     where F: FnOnce(Moment) -> Difference { closure(*self) }
}
fn print<Opt: PrintOptionalMoment>(&self, moment: Moment, other: Opt)
-> (String, Opt::PrintOptionalDifference) {
  ...
  let difference = other.map(|moment| ...);
  ...
  (..., difference)
}
let (string1, ()) = value.print(moment, ());
let (string2, difference2) = value.print(moment, other_moment);
println!("{:?}", difference2);

Rust playground link.

This cleans up the call sites nicely. When you don't pass other, the "difference" result has type (), so you can't misuse it or cause a panic trying to unwrap it. When you do pass other, the "difference" result is not an Option, so you don't need to unwrap it. The implementation of print is basically unchanged, but now Rust will generate two versions of the function, and the version that doesn't take other should be optimized about as well as a handwritten function that removed other. (Unlike in C++, in Rust a () value does not take any space in a struct or tuple.)

If you're not familiar with Rust, the intuition here is that we define a trait PrintOptionalMoment that we use to mean "a type that is either nothing, or a Moment", and we declare that the "void" type () and the type Moment both satisfy PrintOptionalMoment. Then we make print generic over type Opt, which can be either of those. The PrintOptionalMoment trait defines an associated type PrintOptionalDifference which is the result type associated with each Opt that satisfies PrintOptionalMoment.

This approach easily extends to cover more complicated relationships between options and input and output types. In some situations it might be more trouble than it's worth, or the generated code duplication is undesirable, but I think it's a good tool to have.

Updating Pernosco To Rust Futures 0.3

The Pernosco debugger engine is written in Rust and makes extensive use of async code. We had been using futures-preview 0.2; sooner or later we had to update to "new futures" 0.3, and I thought the sooner we did it the easier it would be, so we just did it. The changes were quite extensive:

103 files changed, 3610 insertions(+), 3865 deletions(-)

That took about five days of actual work. The changes were not just mechanical; here are a few thoughts about the process.

The biggest change is that Future and Stream now have a single Output/Item instead of Item and Error, so if you need errors, you have to use an explicit Result. Fixing that was tedious, but I think it's a clear improvement. It encouraged me to reconsider whether we really needed to return errors at all in places where the error type was (), and determine that many of those results can in fact be infallible. Also we had places where the error type was Never but we still had to write unwrap() or similar, which are now cleaned up.

I mostly resisted rewriting code to use async/await, but futures 0.3 omits loop_fn, for the excellent reason that code using it is horrible, so I rewrote our uses of loop_fn with async/await. That code is far easier to read (and write) now. Now that the overall conversion has landed we can incrementally adopt async/await as needed.

It took me a little while to get my head around Pin. Pin adds unfortunate complexity, especially how it bifurcates the world of futures into "pinned" and "unpinned" futures, but I accept that there is no obviously better approach for Rust. The pin-project crate was really useful for porting our Future/Stream combinators without writing unsafe code.

A surprisingly annoying pain point: you can't assign an explicit return type to an async block, and it's difficult to work around. (I often wanted this to set an explicit error type when using ? inside the block.) Async closures would make it easy to work around, but they aren't stabilized yet. Typically I had to work around it by adding explicit type parameters to functions inside the block (e.g. Err).

Outside the main debugger engine we still have quite a lot of code that uses external crates with tokio and futures 0.1. We don't need to update that right now so we'll keep putting it off and hopefully the ecosystem will have evolved by the time we get to it. When the time comes we should be able to do that update more incrementally using the 0.1 ⟷ 0.3 compatibility features.

The really good news is that once I got everything to build, we had only two regressions in our (fairly good) test suite. One was because I had dropped a ! while editing some code. The other was because I had converted a warning into a fatal error to simplify some code, and it turns out that due to an existing bug we were already hitting that situation. Given this was a 4K line patch of nontrivial complexity, I think that's a remarkable testament to the power of Rust.

A Risk Of Transactional Politics For Christians

I've subscribed to Christianity Today for a long time. I was surprised by the editorial advocating Trump's removal from office, but it makes sense to me.

NZ politics is very different from American politics, for which I am profoundly grateful, but it's worth thinking about what to do in a situation where the candidate I strongly preferred on policy was thoroughly immoral. I think one of the biggest issues is the potential corrupting effect of supporting such a candidate, to my own soul and the souls of Christians around me. I think I could easily start out supporting the candidate for policy reasons while declaring their behavior detestable, but there is a great risk that over time that separation breaks down because tribalism, and a tendency to avoid cognitive dissonance, lead to defending or minimizing more and more hitherto unacceptable behavior (Isaiah 5:20). I think it's clear this happens to a lot of people. I believe that kind of personal corruption is an extremely great evil; it would be better for Christians to suffer in all kinds of ways than to fall prey to it (Matthew 16:24-26). This is all even before we consider the impact on our proclamation of the gospel message.

So a good test would be: can I resist the temptation to defend the flaws in this hypothetical flawed candidate? If not, then it is very dangerous to support them. Even if I have adequate mental discipline, there's also the question of whether they are likely to corrupt others in this way.

Nelson Lakes Tramping: Lessons And Observations

Overall, we were really lucky with the weather. We arrived just after a big storm, so there was a lot of water in the streams and significant track damage, but we had very little rain in all ten days. On the Travers-Sabine circuit we had excellent conditions the entire time. I'm so grateful for that.

Because of track damage we had to abandon our plans to visit Blue Lake and significantly alter our route and schedule accordingly. It was actually fun to make up plans on the fly; it felt empowering to know we had the supplies to be relatively self-sufficient for several days and could go wherever we wanted within that time. We appreciatead that it's a feature that the huts in the area (apart from Angelus Hut) don't require or allow advance booking.

Apart from the storm-induced damage, the tracks in the area were very well maintained and easier than I expected.

As always, one of the great things about hut tramping is meeting lots of people from all over the world. On the more demanding tracks, we tended to meet people who were in New Zealand for several months (whereas on the Great Walks you often meet people who are only here for a few weeks). I asked a few people how they first got the idea of visiting New Zealand; more than once I heard they simply looked at the map and saw that New Zealand was about as far away as they could go.

On the Travers-Sabine circuit we ate mostly store-bought dehydrated food for dinner, and I simply don't like any brands that we've tried. I need to figure out how to pack and cook for a week-long tramp food that actually tastes good.

As I noted in a previous post, the water quality in the park was excellent, perhaps aided by the earlier storm. We drank untreated water from streams and rivers everywhere with no ill effects. Even standing pools of water were crystal clear.

We play a lot of card games in huts when tramping. This trip we brought playing cards, Citadels, Bang! (with expansions), and San Juan (with expansion). They're good games but they still got a bit boring, so we need to branch out.

Perhaps the most valuable thing we learned on this trip was how our fitness developed. As we walked day after day we experienced competing effects: we got more tired, but we also got fitter. About half-way through the Travers-Sabine circuit was probably the low point; after that, fitness started to win out. As I noted in the previous post, on day five of the circuit we ascended Speargrass valley to Robert Ridge considerably faster than we had done the same track at the start of our trip. At the end of the circuit everyone in our group agreed that we were feeling great and would have been quite happy to continue tramping, time and supplies permitting. That's an encouraging lesson for all of us.

It was great that we got to attend the "Chapel By the Lake" in St Arnaud twice. It seems like a good congregation, and it's always fun and helpful to see how God works in other churches. Thanks!

Overall it was a great holiday and I thoroughly detoxed from computer overexposure. Now I need to get back to work mode!

Nelson Lakes Tramping: Travers-Sabine Circuit

Following our trip to Angelus and a rest day on Sunday, on Monday we embarked on six-day walk of the Travers-Sabine Circuit, "clockwise". Our plan was to take a water taxi across Lake Rotoiti, walk up the Travers Valley to stay at Upper Travers Hut, and cross the Travers Saddle to West Sabine Hut on the second day. Then we'd leave the circuit proper for two days, walking to Blue Lake Hut and back to West Sabine. Then we'd spend two days finishing the circuit, walking back to St Arnaud with an overnight stay at Sabine Hut/Lake Rotoroa. Things did not go according to plan.

We had heard the previous week's big storm had caused numerous track washouts and changed the course of the Travers River. Department of Conservation people told us to take the Coldwater Track up the west side of the Travers Valley, because the parallel Lakehead Track on the east side was more badly washed out. (They were certainly right about that; we later met trampers who had had a miserable time on the Lakehead Track, and one very tough Polish woman who'd had to turn back.) When our water taxi arrived at Coldwater Hut to drop us off at the start of the track, the water around the jetty was completely clogged with rafts of floating logs and other vegetation! Presumably they'd been washed down in the storm. The skipper, who must have been 18 or 19 but seemed very capable, eventually managed to get us to the jetty after we helped maneuver some of the logs out of the way. We ended up having a very pleasant walk up the valley with just one significant washout on the Coldwater side forcing us to bush-bash off-track for about twenty minutes. It took us about eight hours of walking to get all the way to Upper Travers Hut but we all felt good. Upper Travers is a lovely area at the edge of the treeline below the Travers Saddle. Here we met a number of people — Kiwis, Minnesotans, and people from various European countries — we'd see again at West Sabine Hut and some at Sabine Hut too.

The next day we crossed Travers Saddle in perfect conditions. There were great views of Upper Travers valley, the East and West Sabine valleys, Mount Travers next to us and other mountains nearby. Then there was a long steep descent to the East Sabine river, first over boulder fields and then a rooty, steppy track through the forest — hard on the knees! Spectacular country, but we were glad to get to West Sabine Hut in the early afternoon. West Sabine Hut is pretty cramped but we managed to get those Europeans interested in playing Bang!, so we squeezed around the table and played eight-player Bang! all afternoon and early evening.

Late in the day a couple of DoC volunteers arrived to check hut tickets and update us on track news. The track to Blue Lake was known to be difficult due to washouts — a couple had gone up that afternoon and turned back — but some Te Araroa through-hikers had gone up and not come back so it seemed to be passable. The DoC volunteers were planning to try to get through the next day, as were we and a couple of others at the hut.

So on Wednesday we headed along the track by the West Sabine river towards Blue Lake. Much of it was washed out — replaced by new fast-flowing streams in many places — but we steadily made our way through the bush. Things got tricky when we reached a field of avalanche debris running down into the river. Previously there had been a track etched out among the scree and rocks, but that was gone. Worse, there was a big boulder at the edge of the river with water flowing fast around it, and the hillside above the boulder was a constant steep slope of fine scree. Some people ahead of us had successfully picked their way over the scree, and a couple of people in our group did too, but when I tried to follow I lost my footing and slid down the slope. My son behind me managed to grab my pack and haul me back so we both slid down into a shallow pool by the river ... otherwise I would have gone off the boulder into the deep end of the river. I scraped my elbow pretty well but things could have been worse. After that escape, we decided to give up on Blue Lake and turned back. On the way back we met the DoC volunteers heading up and gave them a full report. (We later learned that they made it to Blue Lake in six and half hours (usually three and a half hours), and were recommending a very wide/high detour around the avalanche field.)

With Blue Lake abandoned, we had to decide what to do with the two days we'd allocated to walk there and back. We had no desire to finish the walk earlier. We settled on lengthening our exit by staying at Sabine Hut and Speargrass Hut, then walking out to the car park but instead of going back to town, going up Mt Robert to stay at Bushline Hut for the last night, which has great views of Lake Rotoiti, St Arnaud and the surrounding valleys and mountains.

The walk from West Sabine Hut to Sabine Hut at Lake Rotoroa was pretty noneventful, just a few short track washouts to navigate around, though in many places the track has been seriously undermined and will need to be rerouted. Sabine Hut is in a beautiful location right at the lakeside, with great views across the lake. Probably not coincidentally, it also had by far the most sandflies outside of any hut on this trip. I had a nice swim there to clean off some of my accumulated grime.

The next day's walk to Speargrass Hut was lovely, but also uneventful. There's a boardwalk section through a swamp, and near the ridge over Speargrass there's a lookout with great views west to mountains. This was our shortest day of walking at just five hours. Speargrass Hut was interesting; there was a party of six Australian women plus another tramper, so the hut was full before evening. Then a couple of deer hunters showed up, and then a couple of very late-arriving European visitors, so they had to sleep on the floor. Speargrass Hut isn't big, so this was the fullest hut I've ever been in so far, but it wasn't a problem.

On the way to Speargrass I had come up with a new plan: instead of walking to the car park along the Speargrass Track we'd already traversed twice in our Angelus tramp, and then up to Bushline Hut, we could go up Speargrass valley again to the ridge and walk along Robert Ridge to Bushline, which would be new track with much better views. At the ridge above Speargrass Hut I was able to get enough cellphone signal to get a weather forecast which suggested the weather next morning would be excellent, just getting a bit windier and starting to rain in the afternoon.

So on Friday we got up earlier and left the hut around 7am to make the most of the good weather. We went up Speargrass valley again — our third time on this section of track in 9 days, but this time with perfect visibility to the mountain ranges in the west. This time we made it to the ridge in just two and a half hours, a full hour faster than our first ascent eight days earlier. There was some wind at the top but the conditions were indeed excellent. We also went pretty fast along Robert Ridge, getting to Bushline Hut in just another three and a half hours including a lunch stop. The views up there in all directions were, as advertised, spectacular.

We arrived at Bushline Hut around 1pm and played card games all afternoon. It was a great place for an extended visit, even though it did get cloudy and rainy in the afternoon. It's on the excellent Mt Robert loop track day walk, so we had numerous groups of day walkers sheltering in the hut during showers, who were fun to talk to. Drifting clouds and rain meant the view was constantly changing: one minute we'd have impenetrable fog, next minute a view all the way to the town just under cloud cover, another minute dramatic views of showers sweeping down the valleys across the lake. Because it was Friday night, Bushline Hut got overfull with people driving to the Mt Robert car park after work and walking up to get a head start on a weekend tramp to Angelus Hut. Again, though the hut was crowded, this was fun and not a problem. (Bushline Hut tip: one of the best views down into Lake Rotoiti can only be had from the toilets.)

Saturday was pretty uneventful — just an easy walk down Mt Robert on the extremely well-formed day-walk-standard track, then a walk around the lake back to town, about three hours altogether. Everyone enjoyed hearty meals for the first time in nearly a week.

I have some general observations to make about our Nelson Lakes tramps, but this blog post is already too long so I'll put them in a new one.

Nelson Lakes Tramping: Angelus Basin

On Thursday December 5 I flew down to Nelson with five friends and family members to spend ten days in the Nelson Lakes National Park, split into two tramping trips: a three-day trip to Angelus Hut, followed by a rest day and then a six-day trip around the Travers-Sabine circuit. Nelson Lakes is a mountainous area with lots of tracks: around lakes, along rivers and streams in bush-clad valleys, and some above the treeline over saddles and along ridges. Just before we arrived, the area was hit by a powerful storm that caused flash-flooding in the rivers and streams throughout the park. We just missed the storm itself, but the impact of the flooding was evident everywhere, including a lot of track wash-outs that made travel interesting, as I will describe later!

Upon arriving at the airport we got a shuttle straight to our trailhead at Mt Robert car park (with a brief stop to drop off some gear at the motel). We had intended to spend Thursday night at Bushline Hut near the top of Mt Robert, then on Friday proceed along Robert Ridge to our main destination, Angelus Hut among the peaks of the Travers Range. However relatively high winds (at least 50km/h) were predicted for the ridge on Friday, which combined with foggy and possibly rainy conditions meant we weren't comfortable with that option. Instead we chose the more sheltered Speargrass valley route: from the car park along Speargrass Creek to Speargrass Hut, then on Friday up the Speargrass valley and over the ridge to Angelus.

The Speargrass route worked out well. We didn't have any significant rain, but the track was dripping wet and the rivers and streams were roaring with water. At one point on the way to Speargrass Hut we had to cross a tricky scree pile beside Speargrass creek; the track had been completely eliminated by scree being washed down. Nevertheless we made it to Speargrass Hut in about three hours of pretty easy walking. There wasn't much of a view due to low-lying cloud, but the weather was otherwise clear. We spent the rest of the afternoon lazing around the hut, except for one person who discovered they really enjoyed chopping wood. Six other people joined us for the night (they were doing a one-night trip to Speargrass), making the 12-bunk hut exactly full. It was a bit cold so we got a fire going and had a great evening. Because this was only a three-day trip we had decent food — sausages and bread for dinner.

The next day, about 9am, we headed up the valley route to Angelus. It was fairly easy going, though the track crosses the creek many times, and the creek was quite high so we had to get wet feet. Again there wasn't much of a view but the cloud and mist had their own dramatic effect. As expected we were sheltered from the wind until we reached the ridge at the top of the valley, where indeed it was quite uncomfortable, but from there it was only a half-hour walk to Angelus Hut — crossing a couple of patches of snow, which was fun.

Angelus Hut is in an amazing spot, between a couple of lakes at the bottom of Angelus Basin, ringed by a ridge that still has a lot of snowy patches this early in the summer. It was a bit too cold to enjoy the outside deck, and unfortunately it was also quite cold inside the hut because they had just about run out of firewood so there was no fire. Having arrived in time for lunch, around 1pm, we had lunch and made endless hot chocolates to keep warm. Visibility was generally poor due to drifting fog, but late in the afternoon the clouds lifted to give us a clear view of the surrounding mountains. We took a walk around the smaller lake to enjoy the view and the alpine environment — absolutely magical.

During the day quite a number of other trampers arrived, mostly arriving via the Robert Ridge which had turned to out to be okay on the day. For some reason a lot of them were Israeli; apparently word about Angelus Hut has spread in the Israeli travel community. There was also a volunteer hut warden present, who gave a good hut talk in the evening while I cooked pasta, salami and tomato sauce for dinner.

I was bemused by the "BOIL WATER BEFORE DRINKING" signs in Angelus Hut (and in many other huts on our trip). I have almost never treated hut water anywhere in New Zealand and Angelus seemed like an unlikely place for the hut water to require treatment. Most huts I've visited have less strident warnings, along the lines of "the water is untreated but generally safe to drink; you may wish to treat it before drinking". We ended up drinking hut water — and flowing stream and river water — untreated everywhere during our trip, to no ill effects. I hope this is not a new trend towards more risk-averse — and less informative — signage.

It was a beautiful moonlight night in the Angelus Basin, though not clear enough to see many stars unfortunately, and I got up and stood outside in the cold for a while to soak up the atmosphere and enjoy the majesty of God's creation.

We were hoping to take the Robert Ridge route to Mt Robert and then back to town on Saturday, but the warden got a weather forecast in the morning that made that sound inadvisable. So we returned via the same Speargrass route we came up, walking all the way back to the car park and then following the road back into town to check into our motel. It was a longish day but not too hard and we had an excellent dinner at a local restaurant. On Sunday one of our group return to Auckland while the rest of us rested and prepared for the next leg of our trip — which deserves its own blog post.

All in all, a little disappointing that we didn't get to walk the Robert Ridge, but a great tramp regardless.

Your Debugger Sucks

Author's note: Unfortunately, my tweets and blogs on old-hat themes like "C++ sucks, LOL" get lots of traffic, while my messages about Pernosco, which I think is much more interesting and important, have relatively little exposure. So, it's troll time.

TL;DR Debuggers suck, not using a debugger sucks, and you suck.

If you don't use an interactive debugger then you probably debug by adding logging code and rebuilding/rerunning the program. That gives you a view of what happens over time, but it's slow, can take many iterations, and you're limited to dumping some easily accessible state at certain program points. That sucks.

If you use a traditional interactive debugger, it sucks in different ways. You spend a lot of time trying to reproduce bugs locally so you can attach your debugger, even though in many cases those bugs have already been reproduced by other people or in CI test suites. You have to reproduce the problem many times as you iteratively narrow down the cause. Often the debugger interferes with the code under test so the problem doesn't show up, or not the way you expect. The debugger lets you inspect the current state of the program and stop at selected program points, but doesn't track data or control flow or remember much about what happened in the past. You're pretty much stuck debugging on your own; there's no real support for collaboration or recording what you've discovered.

If you use a cutting-edge record and replay debugger like rr, it sucks less. You only have to reproduce the bug once and the recording process is probably less invasive. You can reverse-execute for a much more natural debugging experience. However, it's still hard to collaborate, interactive response can be slow, and the feature set is mostly limited to the interface of a traditional debugger even though there's much more information available under the hood. Frankly, it still sucks.

Software developers and companies everywhere should be very sad about all this. If there's a better way to debug, then we're leaving lots of productivity — therefore money — on the table, not to mention making developers miserable, because (as I mentioned) debugging sucks.

If debugging is so important, why haven't people built better tools? I have a few theories, but I think the biggest reason is that developers suck. In particular, developer culture is that developers don't pay for tools, especially not debuggers. They have always been free, and therefore no-one wants to pay for them, even if they would credibly save far more money than they cost. I have lost count of the number of people who have told me "you'll never make money selling a debugger", and I'm not sure they're wrong. Therefore, no-one wants to invest in them, and indeed, historically, investment in debugging tools has been extremely low. As far as I know, the only way to fix this situation is by building tools so much better than the free tools that the absurdity of refusing to pay for them is overwhelming, and expectations shift.

Another important factor is that the stagnation of debugging technology has stunted the imagination of developers and tool builders. Most people have still never even heard of anything better than the traditional stop-and-inspect debugger, so of course they're not interested in new debugging technology when they expect it to be no better than that. Again, the only cure I see here is to push harder: promulgation of better tools can raise expectations.

That's a cathartic rant, but of course my ultimate point is that we are doing something positive! Pernosco tackles all those debugger pitfalls I mentioned; it is our attempt to build that absurdly better tool that changes culture and expectations. I want everyone to know about Pernosco, not just to attract the customers we need for sustainable debugging investment, but so that developers everywhere wake up to the awful state of debugging and rise up to demand an end to it.