Thursday, 12 September 2019

Dissatisfied With Docker

I am not satisfied with Docker.

Untrusted users should be able to run their own container instances. Running a basic container instance means forking a process, putting it in the right kernel namespaces, and setting up mounts and virtual network interfaces, and those can all be done without privileges. Unfortunately, in Docker, access to the Docker daemon is tantamount to being root. Fixing this would improve both convenience and security.

In fact, a global system daemon should not be needed. Either users should be able to run their own daemons or container management should avoid having a daemon at all, by storing container state in a shared database.

Docker container builds are too slow. Installing a container image requires piping massive amounts of image data to the daemon over a socket, which is ridiculous. This could be avoided by passing a file descriptor to the container manager ... or you might even get rid of container image management and start containers by pointing directly to the files comprising the image.

Docker container instances start too slowly. It shouldn't take seconds to start or stop a simple small container. The system calls required to start a simple container run in milliseconds.

No doubt these observations are old news, so I assume there are better container implementations out there. Which one do I want?

Wednesday, 10 July 2019

Cape Brett 2019

It's the school holidays so I took one of my children and one of my friends (a young adult) for a tramping trip to Cape Brett Hut on Monday and Tuesday. It's nominally an eight-hour walk each way. The hut used to be a lighthouse-keeper's house and is in a spectacular setting right at the end of the Cape, on a grassy slope with the ocean on three sides. The walking track is through lovely bush with great views north to the Bay of Islands and south along the Tutukaka coast. It's an excellent trip, just a few hours drive from Auckland and because it's up north and coastal, it's good to do during the winter. The hut is almost never fully booked, perhaps because the walk is quite long and arduous (compared to other walks in the area).

I went there eight years ago with a group. That time we took a water taxi to the hut and a water taxi carried our packs out while we walked back along the track. This time I wanted to do it "properly": walking with our supplies both ways. The nature of the group was also quite different; eight years ago the group was much larger and with a greater variation of fitness, while this time there was just the three of us and we're all pretty fit (but I'm eight years older than last time!).

We stayed at Whangaruru Beachfront Camp on Sunday night so we could get to the trailhead early on Monday and start walking around 7:45am, not long after sunrise at 7:30am. The days are short and I wanted to reach the hut with plenty of daylight to enjoy the destination. We ended up being pretty fast and got to the hut in about six hours, just before 2pm! It's a tough track, with lots of steep uphills and downhills, and we all felt a fair bit of soreness in our legs. Nevertheless I was pretty pleased with our speed and my ability to keep up with the younger people. The weather was great both days — mild, mostly cloudy, and a light breeze in places — and the scenery was brilliant. We had time to rest in the sun and explore the end of the Cape before it got dark, then we cooked a tasty meal.

Around 8pm, when it had been fully dark for a while, a couple arrived at the hut. They told us they'd walked to the hut in just five hours, most of that in the dark. That deflated my pride a bit!

During the night the skies cleared and the moon set, giving an excellent view of the stars through the windows next to my bunk.

On Tuesday we again got up pretty early, around 7am. The sun hadn't risen but we'd already been in our bunks for ten hours. We got to see a lovely sunrise over the ocean. We left the hut at 8:20am and this time finished the walk out in just five hours and twenty minutes. Perhaps surprisingly, I felt a lot better on the second day than the first, and so did the others. Our packs were a little lighter, but I think the previous day's workout had made us all a bit fitter. I found it exhilarating grinding up steep hills without pausing and then stretching the legs for a fast walk along the flat or slightly downhill, and I also felt more agile on the steep downhill sections.

This was a great trip and I really feel thankful to God for the privilege of being able to it. I look forward to doing it again with other people; the walk isn't for everyone, but there's always the water taxi option.

Monday, 1 July 2019

Auckland Rust Meetup: "Building An Omniscient Debugger In Rust"

I gave this month's Auckland Rust Meetup talk: a very high-level overview of Pernosco's architecture and then a dive into some superficial metrics about the project, comments on the third-party crates we use, and some thoughts about the pros and cons of Rust for this project. I apologise for the slides being thrown together in a hurry, and they're probably a bit hard to follow without my commentary.

Thursday, 20 June 2019

Stack Write Traffic In Firefox Binaries

For people who like this sort of thing...

I became interested in how much CPU memory write traffic corresponds to "stack writes". For x86-64 this roughly corresponds to writes that use RSP or RBP as a base register (including implicitly via PUSH/CALL). I thought I had pretty good intuitions about x86 machine code, but the results surprised me.

In a Firefox debug build running a (non-media) DOM test (including browser startup/rendering/shutdown), Linux x86-64, non-optimized (in an rr recording, though that shouldn't matter):

Base registerFraction of written bytes
RAX0.40%
RCX0.32%
RDX0.31%
RBX0.01%
RSP53.48%
RBP44.12%
RSI0.50%
RDI0.58%
R80.01%
R90.00%
R100.00%
R110.00%
R120.00%
R130.00%
R140.00%
R150.00%
RIP0.00%
RDI (MOVS/STOS)0.25%
Other0.00%
RSP/RBP97.59%

Ooof! I expected stack writes to dominate, since non-opt Firefox builds have lots of trivial function calls and local variables live on the stack, but 97.6% is a lot more dominant than I expected.

You would expect optimized builds to be much less stack-dominated because trivial functions have been inlined and local variables should mostly be in registers. So here's a Firefox optimized build:

Base registerFraction of written bytes
RAX1.23%
RCX0.78%
RDX0.36%
RBX2.75%
RSP75.30%
RBP8.34%
RSI0.98%
RDI4.07%
R80.19%
R90.06%
R100.04%
R110.03%
R120.40%
R130.30%
R141.13%
R150.36%
RIP0.14%
RDI (MOVS/STOS)3.51%
Other0.03%
RSP/RBP83.64%

Definitely less stack-dominated than for non-opt builds — but still very stack-dominated! And of course this is not counting indirect writes to the stack, e.g. to out-parameters via pointers held in general-purpose registers. (Note that opt builds could use RBP for non-stack purposes, but Firefox builds with -fno-omit-frame-pointer so only in leaf functions, and even then, probably not.)

It would be interesting to compare the absolute number of written bytes between opt and non-opt builds but I don't have traces running the same test immediately at hand. Non-opt builds certainly do a lot more writes.

Tuesday, 4 June 2019

Winter Tramp: Waihohonu Hut To Tama Lakes

New Zealand's Tongariro National Park is one of my favourite places. We had a three-day weekend so I drove three friends and family down for a two-night stay at Waihohonu Hut, surely the grandest public hut in New Zealand, and we enjoyed the park in a wintry setting ... an interesting change from our previous visits.

We left Auckland around 7am on Saturday to avoid traffic — often hordes of people leave Auckland for long weekends — but there was no congestion. After stopping for lunch in Turangi we reached the trailhead on the Desert Road shortly before 1pm. The wind was cold and there was thick low-lying cloud, but it wasn't snowing ... yet. From there the walk to Waihohonu Hut is easy in less than two hours, on a good quality track with a very gentle upward slope. Much of the route is very exposed but the wind wasn't as high as forecast and we were well equipped. Towards the end it started snowing gently, but that was fun and we got to the hut in high spirits before 3pm. The hut is well insulated and other trampers had arrived before us and got the fire going, and the LED lighting was on, so it was cosy. We talked to them, made popcorn, watched the snow fall, played some card games and enjoyed the rest of the afternoon and evening as more trampers trickled in.

I had wondered how full the hut would get. There are 28 bunks, but it's off-season so they can't be booked, and given the public holiday potentially a lot of people could have turned up. As it happened about 35 people ended up there on Saturday night — many people tramping in from the Desert Road just to visit Waihohonu, like us, but also quite a few doing round trips from Whakapapa or even doing the Tongariro Northern Circuit (which requires alpine skills at this time of year). People grabbed bunks as they arrived, and the rest slept on spare mattresses in the common room, which was actually a fine option. The only problem with sleeping in the common room is people staying up late and (probably other) people coming in early for breakfast. Even though it was technically overfull, Waihohonu Hut's common areas are so spacious that at no time did it ever feel crowded.

On Sunday morning there was a bit more snow on the ground, some low cloud and light snow falling. I was hoping to walk west from the hut to the Tama Saddle, which separates Mt Ruapehu to the south from Mts Tongariro and Ngauruhoe to the north, and visit the Lower Tama Lake just north of the saddle. It was unclear what conditions were going to be like but the forecast had predicted snow would stop falling in the morning, and we were well equipped, so we decided to give it a go. The expected walking time was about six and a half hours and we left before 9am so we had plenty of time. In the end it worked out very well. The cloud lifted quickly, except for some tufts around Ruapehu, and the snow did stop falling, so we had stunning views of the mountains the whole day. We were the first walkers heading west that day so we walked through fresh snow, broke the ice of frozen puddles and streams, and saw the footprints of rabbits and other animals, and relished the pristine wintry environment. It's the first time I've done a long-ish walk in the snow in the wilderness like this, and it was magnificent! I'm so grateful we had the chance to be there and that the weather turned out well.

As we got close to the saddle the snow was thicker, up to our knees in a few places, and the wind got stronger, and at the Lower Tama Lake it was quite cold indeed and blowing steadily from the east. I was a bit worried about having to walk back into that wind, and there was still the possibility of a change in the weather, so even though we were ahead of schedule I decided after lunch above the lake we should head back to Waihohonu rather than carrying on up to Upper Tama Lake (where no doubt the views would have been even better, but the wind would have been even colder!). Interestingly though, we were far from alone; many people, mostly foreign tourists, had walked to the lakes from Whakapapa (on the western side of Ruapehu), a shorter walk, and even headed up the ridge to the upper lake. As it turned out, our walk back was pretty easy. The wind mostly died away and the sun even came out.

We got back to Waihohonu about 3:30pm and once again relaxed at the hut for the rest of the afternoon, catching up with the trampers who were staying both nights and meeting new arrivals. That night the hut was again overfull but only by a couple of people, and again that wasn't a problem.

This morning (Monday) the sky was completely clear, giving magnificent views of snow-covered Ngauruhoe and Ruapehu through the hut's huge picture windows. A thick frost on the ground combined with the snow to form a delightfully crunchy surface for our walk back to the car park. I for one kept turning around to take in the incredible views. It was a very pleasant drive back in the sun through the heart of the North Island, but I can't want to go tramping again!

Wednesday, 29 May 2019

A Few Comments On "Sparse Record And Replay With Controlled Scheduling"

This upcoming PLDI paper is cool. One thing I like about it is that it does a detailed comparison against rr, and a fair comparison too. The problem of reproducing race bugs using randomized scheduling in a record-and-replay setting is important, and the paper has interesting quantitative results.

It's unfortunate that the paper doesn't mention rr's chaos mode, which is our attempt to tackle roughly the same problem. It would be very interesting to compare chaos mode to the approach in this paper on the same or similar benchmarks.

I'm quite surprised that the PLDI reviewers accepted this paper. I don't mean that the paper is poor, because I think it's actually quite good. We submitted papers about rr to several conferences including PLDI (until USENIX ATC accepted it), and we consistently got quite strong negative review comments that it wasn't clear enough which programs rr would record and replay successfully, and what properties of the execution were guaranteed to be preserved during the replay. We described many steps we had to take to get applications to record efficiently in rr in practice, and many reviewers seemed to perceive rr as just a collection of hacks and thus not publishable. Yet it seems to me this "sparse replay" approach is considerably more vague than rr about what it can handle and what gets preserved during replay. I do not see any principled reason why the critical reviewers of our rr paper would not have criticised this paper even harder. I wonder what led to a different outcome.

Perhaps making the idea of "sparse replay" (i.e., record only some subset of behaviour that's necessary and sufficient for a particular application) a focus of the paper effectively lampshaded the problem, or just sufficiently reduced expectations by not claiming to be a general-purpose tool.

I also suspect it's partly just "luck of the draw" in reviewer assignment. It is an unfortunate fact that paper review outcomes can be pretty random. As both a submitter and reviewer, I've seen that scores from different reviewers often differ wildly — it's not uncommon for a paper to get both A and D reviews on an A-to-D scale. When a paper gets both A and D, it typically gets a lot more scrutiny from the review committee to reach a decision, but one should also expect that there are many (un)lucky papers that just happen to avoid a D reviewer or fail to connect with an A reviewer. Given how important publications are to many people (fortunately, not to me), it's not a great system. Though, like democracy, maybe it's better than the others.

Saturday, 25 May 2019

Microsoft's Azure Time-Travel Debugging

This looks pretty cool. The video is also instructive.

It's not totally clear to me how this works under the hood, but apparently they have hooked up the Nirvana TTD to the .NET runtime so that it will enable TTD recording of invocations of particular methods. That means you can inspect the control flow, the state of registers (i.e. local variables), and any memory read by the method or its callees, at any point during the method invocation. It's not clear what happens if you inspect memory outside the scope of the method (e.g. global variables) or if you inspect memory that was modified concurrently by other threads. Plus there are performance and other issues listed in the blog post.

This seems like a good idea but somewhat more limited than a full-fledged record-everything debugger like rr or WinDbg-TTD. I suspect they're pushing this limited-scope debugging as a way to reduce run-time overhead. Various people have told me that WinDbg-TTD has almost unusably high overhead for Firefox ... though other people have told me they found it tolerable for their work on Chrome, so data is mixed.

One interesting issue here is that if I was designing a Nirvana-style multithread-capable recorder for .NET applications — i.e., one that records all memory reads in some fashion via code instrumentation — I would try building it into the .NET VM itself, like Chronon for Java. That way you avoid recording stuff like GC (noted as a problem for this Azure debugger), and the JIT compiler can optimize your instrumentation. I guess Microsoft people were looking for a way to deploy TTD more widely and decided this was the best option. That would be reasonable, but it would be a "solution-driven" approach to the problem, which I have strong feelings about.