Tuesday, 4 June 2019

Winter Tramp: Waihohonu Hut To Tama Lakes

New Zealand's Tongariro National Park is one of my favourite places. We had a three-day weekend so I drove three friends and family down for a two-night stay at Waihohonu Hut, surely the grandest public hut in New Zealand, and we enjoyed the park in a wintry setting ... an interesting change from our previous visits.

We left Auckland around 7am on Saturday to avoid traffic — often hordes of people leave Auckland for long weekends — but there was no congestion. After stopping for lunch in Turangi we reached the trailhead on the Desert Road shortly before 1pm. The wind was cold and there was thick low-lying cloud, but it wasn't snowing ... yet. From there the walk to Waihohonu Hut is easy in less than two hours, on a good quality track with a very gentle upward slope. Much of the route is very exposed but the wind wasn't as high as forecast and we were well equipped. Towards the end it started snowing gently, but that was fun and we got to the hut in high spirits before 3pm. The hut is well insulated and other trampers had arrived before us and got the fire going, and the LED lighting was on, so it was cosy. We talked to them, made popcorn, watched the snow fall, played some card games and enjoyed the rest of the afternoon and evening as more trampers trickled in.

I had wondered how full the hut would get. There are 28 bunks, but it's off-season so they can't be booked, and given the public holiday potentially a lot of people could have turned up. As it happened about 35 people ended up there on Saturday night — many people tramping in from the Desert Road just to visit Waihohonu, like us, but also quite a few doing round trips from Whakapapa or even doing the Tongariro Northern Circuit (which requires alpine skills at this time of year). People grabbed bunks as they arrived, and the rest slept on spare mattresses in the common room, which was actually a fine option. The only problem with sleeping in the common room is people staying up late and (probably other) people coming in early for breakfast. Even though it was technically overfull, Waihohonu Hut's common areas are so spacious that at no time did it ever feel crowded.

On Sunday morning there was a bit more snow on the ground, some low cloud and light snow falling. I was hoping to walk west from the hut to the Tama Saddle, which separates Mt Ruapehu to the south from Mts Tongariro and Ngauruhoe to the north, and visit the Lower Tama Lake just north of the saddle. It was unclear what conditions were going to be like but the forecast had predicted snow would stop falling in the morning, and we were well equipped, so we decided to give it a go. The expected walking time was about six and a half hours and we left before 9am so we had plenty of time. In the end it worked out very well. The cloud lifted quickly, except for some tufts around Ruapehu, and the snow did stop falling, so we had stunning views of the mountains the whole day. We were the first walkers heading west that day so we walked through fresh snow, broke the ice of frozen puddles and streams, and saw the footprints of rabbits and other animals, and relished the pristine wintry environment. It's the first time I've done a long-ish walk in the snow in the wilderness like this, and it was magnificent! I'm so grateful we had the chance to be there and that the weather turned out well.

As we got close to the saddle the snow was thicker, up to our knees in a few places, and the wind got stronger, and at the Lower Tama Lake it was quite cold indeed and blowing steadily from the east. I was a bit worried about having to walk back into that wind, and there was still the possibility of a change in the weather, so even though we were ahead of schedule I decided after lunch above the lake we should head back to Waihohonu rather than carrying on up to Upper Tama Lake (where no doubt the views would have been even better, but the wind would have been even colder!). Interestingly though, we were far from alone; many people, mostly foreign tourists, had walked to the lakes from Whakapapa (on the western side of Ruapehu), a shorter walk, and even headed up the ridge to the upper lake. As it turned out, our walk back was pretty easy. The wind mostly died away and the sun even came out.

We got back to Waihohonu about 3:30pm and once again relaxed at the hut for the rest of the afternoon, catching up with the trampers who were staying both nights and meeting new arrivals. That night the hut was again overfull but only by a couple of people, and again that wasn't a problem.

This morning (Monday) the sky was completely clear, giving magnificent views of snow-covered Ngauruhoe and Ruapehu through the hut's huge picture windows. A thick frost on the ground combined with the snow to form a delightfully crunchy surface for our walk back to the car park. I for one kept turning around to take in the incredible views. It was a very pleasant drive back in the sun through the heart of the North Island, but I can't want to go tramping again!

Wednesday, 29 May 2019

A Few Comments On "Sparse Record And Replay With Controlled Scheduling"

This upcoming PLDI paper is cool. One thing I like about it is that it does a detailed comparison against rr, and a fair comparison too. The problem of reproducing race bugs using randomized scheduling in a record-and-replay setting is important, and the paper has interesting quantitative results.

It's unfortunate that the paper doesn't mention rr's chaos mode, which is our attempt to tackle roughly the same problem. It would be very interesting to compare chaos mode to the approach in this paper on the same or similar benchmarks.

I'm quite surprised that the PLDI reviewers accepted this paper. I don't mean that the paper is poor, because I think it's actually quite good. We submitted papers about rr to several conferences including PLDI (until USENIX ATC accepted it), and we consistently got quite strong negative review comments that it wasn't clear enough which programs rr would record and replay successfully, and what properties of the execution were guaranteed to be preserved during the replay. We described many steps we had to take to get applications to record efficiently in rr in practice, and many reviewers seemed to perceive rr as just a collection of hacks and thus not publishable. Yet it seems to me this "sparse replay" approach is considerably more vague than rr about what it can handle and what gets preserved during replay. I do not see any principled reason why the critical reviewers of our rr paper would not have criticised this paper even harder. I wonder what led to a different outcome.

Perhaps making the idea of "sparse replay" (i.e., record only some subset of behaviour that's necessary and sufficient for a particular application) a focus of the paper effectively lampshaded the problem, or just sufficiently reduced expectations by not claiming to be a general-purpose tool.

I also suspect it's partly just "luck of the draw" in reviewer assignment. It is an unfortunate fact that paper review outcomes can be pretty random. As both a submitter and reviewer, I've seen that scores from different reviewers often differ wildly — it's not uncommon for a paper to get both A and D reviews on an A-to-D scale. When a paper gets both A and D, it typically gets a lot more scrutiny from the review committee to reach a decision, but one should also expect that there are many (un)lucky papers that just happen to avoid a D reviewer or fail to connect with an A reviewer. Given how important publications are to many people (fortunately, not to me), it's not a great system. Though, like democracy, maybe it's better than the others.

Saturday, 25 May 2019

Microsoft's Azure Time-Travel Debugging

This looks pretty cool. The video is also instructive.

It's not totally clear to me how this works under the hood, but apparently they have hooked up the Nirvana TTD to the .NET runtime so that it will enable TTD recording of invocations of particular methods. That means you can inspect the control flow, the state of registers (i.e. local variables), and any memory read by the method or its callees, at any point during the method invocation. It's not clear what happens if you inspect memory outside the scope of the method (e.g. global variables) or if you inspect memory that was modified concurrently by other threads. Plus there are performance and other issues listed in the blog post.

This seems like a good idea but somewhat more limited than a full-fledged record-everything debugger like rr or WinDbg-TTD. I suspect they're pushing this limited-scope debugging as a way to reduce run-time overhead. Various people have told me that WinDbg-TTD has almost unusably high overhead for Firefox ... though other people have told me they found it tolerable for their work on Chrome, so data is mixed.

One interesting issue here is that if I was designing a Nirvana-style multithread-capable recorder for .NET applications — i.e., one that records all memory reads in some fashion via code instrumentation — I would try building it into the .NET VM itself, like Chronon for Java. That way you avoid recording stuff like GC (noted as a problem for this Azure debugger), and the JIT compiler can optimize your instrumentation. I guess Microsoft people were looking for a way to deploy TTD more widely and decided this was the best option. That would be reasonable, but it would be a "solution-driven" approach to the problem, which I have strong feelings about.

Monday, 20 May 2019

Don't Call Socially Conservative Politicial Parties "Christian"

There is talk about starting a new "Christian" (or "Christian values") political party in New Zealand. The party might be a good idea, but if it's really a "social conservative" party, don't call it "Christian".

Audrey Young writes:

The issues that would galvanise the party are the three big social issues before Parliament at present and likely to be so in election year as well: making abortions easier to get, legalising euthanasia, and legalising recreational cannabis.

None of those issues are specifically Christian. None of them are mentioned directly in the New Testament. I even think Christians can be for some version of all of them (though it makes sense to me that most Christians would oppose the first two at least). Therefore "social conservative" is a much more accurate label than "Christian" for a party focused on opposing those changes.

A truly Christian party's key issues would include reminding the voting public that we all sinners against God, in need of repentance and forgiveness that comes through Jesus. The party would proclaim to voters "how hard it is for the rich to enter the kingdom of God" and warn against storing up treasures on earth instead of heaven. It would insist on policies that support "the least of these". It would find a way to denounce universally popular sins such as greed, gluttony and heterosexual extra-marital sex, and advocate policies that reduce their harm, while visibly observing Paul's dictum "What business is it of mine to judge those outside the church? Are you not to judge those inside?" A Christian party would follow Jesus' warning against "those who for a show make lengthy prayers" and downplay their own piety. It would put extraordinary emphasis on honouring the name of Christ by avoiding any sort of lies, corruption or scandal. Its members would show love for their enemies and not retaliate when attacked. If they fail in public, they would confess and repent in public.

That sounds pretty difficult, but it's what Jesus deserves from any party that claims his name.

I'm all for Christians being involved in politics and applying their Christian worldview to politics, if they can succeed without making moral compromises. But it's incredibly important that any Christian who publicly connects Christ with politics takes into account how that will shape unbelievers' view of Christianity. If they lead people to believe that Christianity is about being socially conservative and avoiding certain hot-button sins, with the gospel nowhere in sight, then they point people towards Hell and betray Jesus and his message.

Monday, 6 May 2019

Debugging Talk At Auckland Rust Meetup

I gave a talk about "debugging techniques for Rust" at tonight's Auckland Rust Meetup. There was many good questions and I had a good time. It wasn't recorded. Thanks to the organiser and sponsors!

I'm also going to give a talk at the next meetup in June!

Monday, 29 April 2019

Goodbye Mozilla IRC

I've been connected to Mozilla IRC for about 20 years. When I first started hanging out on Mozilla IRC I was a grad student at CMU. It's how I got to know a lot of Mozilla people. I was never an IRC op or power user, but when #mozilla was getting overwhelmed with browser user chat I was the one who created #developers. RIP.

I'll be sad to see it go, but I understand the decision. Technologies have best-before dates. I hope that Mozilla chooses a replacement that sucks less. I hope they don't choose Slack. Slack deliberately treats non-Chrome browsers as second-class — in particular, Slack Calls don't work in Firefox. That's obviously a problem for Mozilla users, and it would send a bad message if Mozilla says that sort of attitude is fine with them.

I look forward to finding out what the new venue is. I hope it will be friendly to non-Mozilla-staff and the community can move over more or less intact.

Friday, 26 April 2019

Update To rr Master To Debug Firefox Trunk

A few days ago Firefox started using LMDB (via rkv) to store some startup info. LMDB relies on file descriptor I/O being coherent with memory-maps in a way that rr didn't support, so people have had trouble debugging Firefox in rr, and Pernosco's CI test failure reproducer also broke. We have checked in a fix to rr master and are in the process of updating the Pernosco pipeline.

The issue is that LMDB opens a file, maps it into memory MAP_SHARED, and then opens the file again and writes to it through the new file descriptor, and requires that the written data be immediately reflected in the shared memory mapping. (This behavior is not guaranteed by POSIX but is guaranteed by Linux.) rr needs to observe these writes and record the necessary memory changes, otherwise they won't happen during replay (because writes to files don't happen during replay) and replay will fail. rr already handled the case when the application write to the file descriptor (technically, the file description) that was used to map the file — Chromium has needed this for a while. The LMDB case is harder to handle. To fix LMDB, whenever the application opens a file for writing, we have to check to see if any shared mapping of that file exists and if so, mark that file description so writes to it have their shared-memory effects recorded. Unfortunately this adds overhead to writable file opens, but hopefully it doesn't matter much since in many workloads most file opens are read-only. (If it turns out to be a problem there are ways we can optimize further.) While fixing this, we also added support for the case where the application opens a file (possibly multiple times with different file descriptions) and then creates a shared mapping of one of them. To handle that, when creating a shared mapping we have to scan all open files to see if any of them refer to the mapped file, and if so, mark them so the effects of their writes are recorded.

Update Actually, at least this commit is required.