Feelings Versus Facts At Christmas

Yesterday I had the honour of leading the Christmas Day service for our congregation. I talked briefly about "the feelings of Christmas" vs "the facts of Christmas".

Growing up in New Zealand, Christmas was (and is) an wonderful mix of traditions --- fake snow under the summer sun, Santa suits by the beach, hearty puddings under blossoming pohutukawa trees, family gatherings, gifts, carols, swimming and sunscreen. These traditions produce a palpable feeling of Christmas. In my youth I devalued such feelings, on the grounds that Mr Spock was right and emotion should never displace reason as the sole regulator of one's mental state. (Yes, I was a twerp.)

Later I became a Christian, and my perspective changed. I found that emotions have their proper place --- as a response to truth, not a determiner of it. If Jesus lives, joy and awe are the proper response. More importantly, Christmas is not just tradition and feelings: it has facts behind it, world-changing facts that make Christmas Christmas no matter how we feel about it. God himself became present in the world, in a child, to reestablish a relationship between us and himself. Christmas carols produce Christmassy feelings, but they also recite facts. We can sing them and mean them, and the meaning justifies the feelings.

This is especially relevant during hard times. Suffering doesn't cease during Christmas, and Christmas feelings may be absent, but the facts of Christmas persist no matter what. Thank God for that.

Abel Tasman Track

A week ago I spent four days walking the Abel Tasman track with some friends and family. As always we had a very good time tramping, staying in huts, and enjoying the outdoors. The weather was excellent. The track winds along the coast of Abel Tasman National Park, mostly granite promontories separted by beautiful beaches of coarse golden sand, with thick bush running up into the hills and a few estuaries to cross. This area is busier than other tramping tracks because there are many campsites and most of it is easily accessible by kayak and water taxis --- and by road in a few places. However, in the late afternoons and evenings there's no-one around except for the trampers. Three of the four days were four hours of walking or less, so we had plenty of time relaxing around the huts, often playing Bang.

Despite the relative lack of remoteness, we saw a lot of interesting wildlife on this trip ... a duck with ten ducklings, oystercatchers with chicks, seagulls, shags, herons, kereru, weka, estuary denizens such as fish, crabs and shellfish, and seals on a side trip to Separation Point.

One unexpected highlight was finding a fancy cafe at Awaroa Lodge, nestled in the bush, far from any road. We had an excellent dinner there --- eye-wateringly expensive, but no more than you'd expect for a fine meal in the middle of nowhere.

After finishing the track we stayed the night at Takaka; on Monday the children and I flew home via Wellington. Our flight to Wellington was on a tiny Piper aeroplane: six seats, including the pilot's, the whole aircraft about the size of a large car with wings attached. I got to sit in the front next to the pilot, with duplicate controls, which I did not need reminding to stay clear of! It was a wonderful flight, since we flew low back across Abel Tasman National Park and then the Marlborough Sounds. Turbulence near Wellington was a bit scary --- I don't like rollercoasters and the little plane weaved and dipped a lot --- but not as bad as my last flight into Wellington.

CppCast rr Podcast

This week I took part in a podcast interview with the folks of CppCast, about rr. It was a lot of fun... Check it out.

Even More rr Replay Performance Improvements!

While writing my last blog post I realized I should try to eliminate no-op reschedule events from rr traces. The patch turned out to be very easy, and the results are impressive:

Now replay is faster than recording in all the benchmarks, and for Mochitest is about as fast as normal execution. (As discussed in my previous post, this is probably because the replay excludes some code that runs during normal execution: the test harness and the HTTP server.) Hopefully this turns into real productivity gains for rr users.

rr Replay Performance Improvements

I've been spending a lot of time using rr, as have some other Mozilla developers, and it occurred to me a small investment in speeding up the debugging experience could pay off in improved productivity quite quickly. Until recently no-one had ever really done any work to speed up replay, so there was some low-hanging fruit.

During recording we avoid trapping from tracees to the rr process for common syscalls (read, clock_gettime and the like) with an optimization we call "syscall buffering". The basic idea is that the tracee performs the syscall "untraced", we use a seccomp-bpf predicate to detect that the syscall should not cause a ptrace trap, and when the syscall completes the tracee copies its results to a log buffer. During replay we do not use seccomp-bpf; we were using PTRACE_SYSEMU to generate a ptrace trap for every syscall and then emulating the results of all syscalls from the rr process. The obvious major performance improvement is to avoid generating ptrace traps for buffered syscalls during replay, just as we do during recording.

This was tricky to do while preserving our desired invariants that control flow is identical between recording and replay, and data values (in application memory and registers) are identical at all times. For example consider the recvmsg system call, which takes an in/out msg parameter. During recording syscall wrappers in the tracee would copy msg to the syscall log buffer, perform the system call, then copy the data from the log buffer back to msg. Hitherto, during replay we would trap on the system call and copy the saved buffer contents for that system call to the tracee buffer, whereupon the tracee syscall wrappers would copy the data out to msg. To avoid trapping to rr for a sequence of such syscalls we need to copy the entire syscall log buffer to the tracee before replaying them, but then the syscall wrapper for recvmsg would overwrite the saved output when it copies msg to the buffer! I solved this, and some other related problems, by introducing a few functions that behave differently during recording and replay while preserving control flow and making sure that register values only diverge temporarily and only in a few registers. For this recvmsg case I introduced a function memcpy_input_parameter which behaves like memcpy during recording but is a noop during replay: it reads a global is_replay flag and then does a conditional move to set the source address to the destination address during replay.

Another interesting problem is recapturing control of the tracee after it has run a set of buffered syscalls. We need to trigger some kind of ptrace trap after reaching a certain point in the syscall log buffer, without altering the control flow of the tracee. I handled this by generating a large array of stub functions (each only one byte, a RET instruction) and after processing the log buffer entry ending at offset O, we call stub function number O/8 (each log record is at least 8 bytes long). rr identifies the last log entry after which it wants to stop the tracee, and sets a breakpoint at the appropriate stub function.

It took a few late nights and a couple of half-days of debugging but it works now and I landed it on master. (Though I expect there may be a few latent bugs to shake out.) The results are good:

This shows much improved replay overhead for Mochitest and Reftest, though not much improvement on Octane. Mochitest and Reftest are quite system-call intensive so our optimization gives big wins there. Mochitests spend a significant amount of time in the HTTP server, which is not recorded by rr, and therefore zero-overhead replay could actually run significantly faster than normal execution, so it's not surprising we're already getting close to parity there. Octane replay is dominated by SCHED context-switch events, each one of which we replay using relatively expensive trickery to context-switch at exactly the right moment.

For rr cognoscenti: as part of eliminating traps for replay of buffered syscalls, I also eliminated the traps for the ioctls that arm/disarm the deschedule-notification events. That was relatively easy (just replace those syscalls with noops during replay) and actually simplified code since we don't have to write those events to the trace and can wholly ignore them during replay.

There's definitely more that can be squeezed out of replay, and probably recording as well. E.g. currently we record a SCHED event every time we try to context-switch, even if we end up rescheduling the thread that was already running (which is common). We don't need to do that, and eliminating those events would reduce syscallbuf flushing and also the number of ptrace traps taken during replay. This should hugely benefit Octane. I'm trying to focus on easy rr improvements with big wins that are likely to pay off for Mozilla developers in the short term; it's difficult to know whether any given improvement is in that category, but I think SCHED elision during recording probably is. (We used to elide recorded SCHED events during replay, but that added significant complexity to reverse execution so I took it out.)

Debugging Leaks With rr

Last week I looked into a couple of Gecko leaks using rr. One fun aspect of using rr is discovering new debugging techniques. "What's the best way to use a record-and-replay debugger to track down memory leaks?" is a very interesting question. Maybe no-one has tried to do this before.

XPCOM_MEM_LEAK_LOG uses instrumentation to count object constructors and destructors and report any mismatches. That tells us immediately which classes of objects leaked. My next step was to identify the address of a leaked object, preferably one that is the root of an ownership tree of leaked objects. In my particular case I chose to investigate a leaked nsWindow, because only one was leaked and I guessed (correctly, as it turns out) that it was most likely to be the root of the leaked object tree.

Finding the address of the leaked nsWindow was easy: run the program with breakpoints on the nsWindow constructor and destructor, set to log the object address and continue. The constructor with no matching destructor has the address of the leaked object. At process exit I could inspect the window object and see that indeed it still existed with refcount 1.

My next move was to use a similar technique to log addref/release operations on that window. This is similar to what you'd get with Gecko's built-in trace-refcount instrumentation, but produced a lot of data that was hard to analyze. I realized the leaked reference to the window probably corresponds to a pointer to the window in memory when the process exits. So I added a real-tid command to rr to print the real thread-id of a replaying process, and wrote a small program to walk /proc/.../maps and /proc/.../mem to search all process memory for the pointer value. This worked very well and showed just two addresses where the pointer occurred.

The next problem was to figure out what those addresses belong to. I futzed around a bit before finding the correct approach: set hardware write watchpoints on those addresses and reverse-execute from the end of the process to find the writes that stored those addresses. This told me immediately where and why each reference was being stored, and if it was a strong reference. This made the bug obvious.

A lot of this could be more automated. It would be particularly interesting to grab the XPCOM_MEM_LEAK_LOG output and process it to automatically output the "leak roots" --- leaking references that are not contained in objects that themselves leaked. If there are no leak roots then instead there must be at least one cycle of strong references, which we can output.

TPPA Protest

On Saturday I participated in a protest march for the first time ever --- against the TPPA. I support lowering trade barriers, but the TPPA has a lot of baggage that I strongly dislike, such committing member nations to a dysfunctional United States-esque intellectual property regime, and sovereignty-eroding "investor-state dispute settlement". The biggest prize for New Zealand would have been free access to foreign dairy markets, but that was mostly not realized, so it seems like a bad deal for us.

Unsurprisingly there were a lot of different sorts of people involved. Many of them espoused themes I don't agree with --- knee-jerk anti-Americanism, "Socialist Aotearoa", general opposition to free-market economics. That made it more fun and interesting :-). I think it's very important that people who disagree about a lot of things can still work together on issues they do agree about.

Perfection In Imperfection

I like this quote from Ken Liu (translator of Cixin Liu's The Three-Body Problem into English):

The best translations into English do not, in fact, read as if they were originally written in English. The English words are arranged in such a way that the reader sees a glimpse of another culture's patterns of thinking, hears an echo of another language's rhythms and cadences, and feels a tremor of another people's gestures and movements.

This principle definitely applies to his translation of The Three-Body Problem. It contributed a lot to my enjoyment of the book. (Overall I thought it was good, though a little disappointing for a Hugo award winner.)

It occurred to me that I've never noticed this principle at work while reading the Bible in English. I think that's because the Bible is its own genre --- there's almost nothing written originally in English to compare it to. And then, anything remotely comparable originating in English would almost certainly been deeply influenced by the Bible.

rr In VMWare: Solved!

Well, sort of.

I previously explained how rr running in a VMWare guest encounters a VMWare bug where performance counters measuring conditional branches executed fail to count a conditional branch executed between a pair of CPUID instructions. Today a helpful VWMare developer confirmed that this is an issue caused by clever optimizations in the virtual machine monitor. Better still, he provided an easy way to work around it! Just add

monitor_control.disable_hvsim_clusters = true

to your .vmx file. This may impact guest performance a little bit, but it will make rr work reliably in a VMWare guest, which should be useful to a lot of people. Excellent!

An Extraordinary Sunday

Sunday was extraordinary. I got up at 4am, took the ferry to Devonport, gathered with a few thousand runners to watch the All Blacks winning the Rugby World Cup final, ran the Auckland half marathon barefoot, had a quick shower at home, made it to church on time to worship the Creator of the universe, taught Sunday school (on the transfiguration of Jesus), ate tasty Vietnamese food at Hansan, took my son to his water polo game, then to his cousin's pool party, returned home to rewatch the final, cooked dinner, tried to figure out an rr bug, and collapsed into bed.

I'd hardly done any training for the half marathon, initially because I had had a sore knee that I was afraid of aggravating and later because I was too busy with tramping trips and other activities. Yesterday I had intended to just try to finish in under two hours, but I started a bit too quickly and by the time I realized it, there was nothing to be done but push on. I finished in 1:48, five minutes faster than my previous best (four net). I paid for it though; keeping going was mentally harder than anything I've done before.

It was a fantastic day altogether. The experience of being in Devonport at dawn watching the rugby before the run was something I'll always remember.

Research Projects That Should NOT Be Funded

"Is there any feasible way to create a small black hole?"

"Is There A Better Route To Fusion?", slide 25.

KPMG Gets It Totally Wrong About Pittsburgh And Auckland

I lived in Pittsburgh for seven years (1994-2001) and have visited it multiple times since, as recently as last year. I've lived in Auckland for the last ten years. I got a PhD in computer science at CMU while I lived there, and of course I'm still working in tech in Auckland. So I was very interested to read KPMG's report on the lessons Pittsburgh holds for New Zealand ... but I was shocked by how different its conclusions are from my experiences.

Pittsburgh was a great place to live as a CMU student, mainly because the population and economy was so down that everything was cheap, especially rent. It's true that things have improved a bit since then in some ways --- Google opening an office there, for example --- but in many other ways things have gotten worse. City finances have been so bad over the last decade that bus services were being cut back to totally unreasonable levels. Multiple efforts to revitalize the downtown --- lauded by KPMG --- have basically failed; it's still a wasteland, and the Macy's that was the core of one of the more recent efforts closed down this year. The roads are unbelievably bad. I find Auckland by far a nicer city.

One thing Pittsburgh does have going for it is CMU and the tech companies that have grown up or built offices around it. Auckland has nothing like that. But here's what KPMG are recommending:

Simon Hunter says Pittsburgh provides a blueprint for Auckland thinking: "There's courage and focus on a specific group of young technical wealth creators; there's a massive contribution from the universities working together and from commercial entities - and there's urban renewal leveraging the rivers."

Almost none of that makes any sense to me. Auckland's geography is entirely different and with the exception of the wharves downtown, its waterfronts don't need "urban renewal". As for the universities and "commercial entities": Google, Intel, Apple etc are there entirely because CMU is there. CMU is there for historical reasons. (Andrews Carnegie and Mellon put it there, it rode a wave of DOD-funded computer science research, and accumulated enough gravitational pull to make it permanently one of the top four CS universities in the USA.) I like Auckland University but the gulf between it and CMU is vast and probably unbridgeable; at the very least, you would need to spend billions. There have been many, many extremely well-funded efforts to replicate CMU and its peers --- in Asia and the Middle East, for example --- and they haven't succeeded yet.

I do think we could do better at building an environment in Auckland or elsewhere in New Zealand that attracts outposts of the good big tech companies. I have some specific policy ideas. But aping Pittsburgh and CMU isn't going to work.

So in summary: Auckland beats Pittsburgh in most ways, and where Pittsburgh has advantages they don't translate here.

rr 4.0 Released With Reverse Execution

I finally got around to releasing rr 4.0!

rr is Mozilla's low overhead record-and-replay debugging tool for large C++ applications like Firefox. For more information see the rr project page, or watch my Technion talk:

rr 4.0 is the first stable release with reverse-execution enabled. This means gdb's reverse-continue, reverse-step, reverse-next and reverse-finish commands work under rr. Not only do they work, but under rr they're quite efficient and have unlimited scope. It's completely reasonable to reverse-continue from the end of a Firefox run all the way back to the beginning. Furthermore, breakpoints and hardware data watchpoints work with reverse execution. Suppose we're debugging Firefox layout:

Breakpoint 1, nsCanvasFrame::BuildDisplayList (this=0x2aaadd7dbeb0, aBuilder=0x7fffffffaaa0, aDirtyRect=..., aLists=...)
    at /home/roc/mozilla-inbound/layout/generic/nsCanvasFrame.cpp:460
460   if (GetPrevInFlow()) {
(gdp) p mRect.width
12000

We happen to know that that value is wrong. We want to find out where it was set. rr makes that quick and easy.

(gdb) watch -l mRect.width
(gdb) reverse-cont
Continuing.
Hardware watchpoint 2: -location mRect.width
Old value = 12000
New value = 11220
0x00002aaab100c0fd in nsIFrame::SetRect (this=0x2aaadd7dbeb0, aRect=...)
    at /home/roc/mozilla-inbound/layout/base/../generic/nsIFrame.h:718
718       mRect = aRect;

This is extremely powerful. rr's regular users inside (and outside) Mozilla have been enjoying this for a while on rr master, so we're glad to be able to get it into a stable release.

This release is about as stable as rr has ever been --- meaning it seems to be generally working for people and very few new issues have been coming in. However, rr depends on gnarly details of the kernel, system libraries and CPU so I expect as the number of rr users grows we'll need to keep fixing bugs and updating rr regularly.

By the way, Mozilla is interested in hiring people to work on rr-related technology, more closely integrated with the browser. I'd like to hear from people who're interested!

Hobbiton

On Sunday afternoon I visited Hobbiton with a few Mozillians. It's a two-hour drive from Auckland to the "Lord of the Rings"/"Hobbit" Hobbiton movie set near Matamata ... a pleasant, if slightly tedious drive through country that looks increasingly Shire-like as you approach the destination. We arrived ahead of schedule and had plenty of time to browse the amusingly profuse gift shop before our tour started.

I'm an enthusiastic fan of The Lord Of The Rings and easily seduced by the rural fantasy of the Shire, but I enjoyed the tour even more than I expected. It's spring and the weather was fine, and Hobbiton is, among other things, a wonderful garden; it looked and smelled gorgeous. The plants, the gently worn footpaths, the spacing of the hobbit holes, the versimilitude of the props, all artfully designed to suggest artlessness --- it all powerfully suggests the place could be real, would be delightful to live in if only it was real, that it should be real. Adding to that the mythology of the movies and Tolkien's books, it was an intoxicating collision between fantasy and reality that kept me slightly off balance mentally.

After the tour proper we entered the Green Dragon Inn for a drink and dinner. The Inn continues the combination of fantasy and reality: carefully designed Shire-props next to "fire exit" signs, real drinks in pottery mugs. The highlight was the dinner feast: food piled on tables, looking like an Epicurean fantasy but actually real, copious and delicious. It's very memorable, great fun, and a bit silly in the best way.

After the dinner there was a walk back through Hobbiton in the dark, with lanterns, admiring the hobbit holes lit to suggest candles and firelight. It was gorgeous again.

Hobbiton is apparently a popular place to propose marriage. One happened on our tour, and another couple in our tour had done so two days before; they couldn't remember the rest of the tour after the proposal, so were repeating it.

The dinner tour is ridiculously expensive, but I thought it was very special and well worth it.

Heaphy Track

October 1-4 I walked the Heaphy Track with family. The weather was wet on the first day but cleared up nicely and as always we had a great time.

The Heaphy runs from the east end of Kahurangi National Park, near Golden Bay on the northern end of the South Island, west through the hills to the west coast of the South Island, and then south along the coast to the northern end of the road. (It can also be walked in the other direction.) Our first day was steady walking uphill, but not steep. The second day was up and down through the Gouland Downs at an elevation of around 800 metres. The third day is steady downhill to the coast, and the fourth day is along the coast, up and down a bit as you cross small ridges. The Heaphy is the longest Great Walk (about 80km) and we walked 5-6 hours each day. Since we generally set out from a hut at 8:30am or so, on the second and third days we arrived at our destination by 2 or 3pm, leaving plenty of time for card games, short walks and socializing!

One of the neat things about the Heaphy is the variety. Each day you're traveling through quite different terrain, all of it interesting. You really appreciate how the environment changes from the east side of Kahurangi to the west coast. I particularly enjoyed the coastal walk on the last day; that coast is truly wild, and parts can be walked off the track, on the beach. At times there is no sign of human presence whatsoever. The sand, surf, thick palm forest and mist work together to provide a feeling of isolation and timelessness.

Going into the walk I had a bit of a niggle in my right knee. That turned out to be no problem at all --- I didn't feel it since we started the walk, and it seems to be cured! On the second and third days I developed some severe ankle chafing against my boots --- skin worn through, with bleeding. So on the fourth day I walked barefoot the whole way. There wasn't as much time on sand as I thought, and the track was often rocky, so it wasn't particularly comfortable, but I found it surprisingly easy. My feet have learned to distinguish pain from damage; going barefoot was more painful than the ankle chafing, but there was no damage at all apart from a small cut on my big toe. That very evening my feet felt fine, but my ankles are still touchy today. Ironically the most painful part of the walk was the last five minutes where someone had helpfully laid gravel on the track.

Apple's Next Frontier: Fusion

Apple has more than two hundred billion dollars in cash that they obviously have no idea what to do with. Impressing investors after the success of the iPhone is very difficult. Apple needs to make a huge investment in something truly world-changing that could be immensely lucrative. Cars and smart watches aren't going to be that. Fusion energy could be! Currently, investment in this area is relatively low compared to the potential payoff. Revolutionizing humanity's energy economy is a challenge worthy of Apple (and Apple's cash). It's risky, but Apple can afford to make a few big bets. And if not Apple, then who?

Booting Fedora 22 On A Lenovo ThinkCentre M53

We just bought a new computer for family use. I want my children to master computers, not just consume, so we got a desktop computer running Linux. I settled on a Lenovo M53, which is similar to a Mac Mini but cheaper --- just 450 NZD, probably because it's a two-year-old model. It comes with 4GB RAM (upgradeable to 8GB), quad-core Celeron, 500GB disk, Wifi, keyboard, mouse, and Windows 8.1. Moore's law always thrills me. When I started my PhD in 1994 we were just at the end of the period when one of the advantages of a top university was that they had access to better computers that let you get more work done. Nowadays any first-world family or school can buy enough computing power for almost any task, and that's a very good thing. I guess it's only a decade or two before everyone else catches up too.

Getting Fedora installed on the disk was easy by booting an installation image from a USB memory stick, but it was very hard to get the unit to actually boot anything other than Windows from the disk. The main problem is that the BIOS seems to reset UEFI boot order every time you boot, always putting Windows first (maybe because it's the first UEFI entry?). I couldn't find any way to prevent this. The BIOS Setup purports to let you change boot order (I'm not sure exactly how that relates to the UEFI boot order; it seems to pull some but not all UEFI boot entries), but no matter how hard I tried I couldn't get those order changes to stick; after "save changes and exit" it would go ahead and boot Windows and reentering BIOS Setup would show the changes were lost. Even entering BIOS Setup is hard; by default there are no on-screen prompts telling you which key to press, and you have only about a half-second window to press F1 during startup. You can however press F12 during startup and select booting from the Fedora partition. Many online comments indicate that Lenovo BIOSes have a problem with setting changes failing to be saved, though it's hard to tell exactly what the problems are.

In the end, I solved the problem by removing the Windows entry (and some other junk entries) from the UEFI boot list using efibootmgr. That was enough for the BIOS to finally give up on booting Windows. I can still boot Windows from the Fedora Grub menu. Presumably I wouldn't have had any of these problems had I not tried to keep Windows available for dual booting...

Hooray For WebExtensions

Many of the comments on WebExtensions miss an important point: basing WebExtensions on the Chrome extension API does not imply limiting WebExtensions to the Chrome API. For example, Bill already made it clear that we want to support addons like Tree Style Tabs and NoScript, which Chrome does not support. So Firefox addons will continue to be able to do things you can't do in Chrome (though there will be some things you can hack into Firefox's XUL today that won't be supported by WebExtensions, for sure).

WebExtensions is something we need to do and should have done many years ago, before Chrome even existed. It's what Jetpack should have been. But better late than never!

Parenting

For a long time I was unenthusiastic about the idea of having children. After being a Christian for a while, I knew raising children would be a good thing to do, that I should do it, and that given the opportunity I would do it ... but I never thought of myself as someone who'd enjoy being a father or be good at it. (At heart I feel more comfortable with technology than with people.) I'll wait for God's judgement on the latter, but on the former, by God's grace, I was totally wrong. I definitely, thoroughly enjoy raising my children. More than that, my children quite often give me moments of sublime joy.

For someone who thinks a lot about mind and machines, watching children develop from helpless babies into the full flower of human physical and mental ability is truly awe-inspiring, and helping to guide that development is a great privilege and responsibility. It's all the dreams for AI coming true before one's own eyes! They learn language, they learn to read, they develop empathy and morals, they learn from what they read, they create new ideas, they pursue their own goals by creating and executing complex plans, and before you know it they're full-fledged people who surprise you, charm you, spar with you and teach you.

There's joy when I see one of my children teaching another. There's joy when my children use words I don't know from books I haven't read. There's joy when one of my children cracks a good, original joke. There's joy when my children see something that needs doing and do it without being told. There's joy when I give a vague instruction and my children work out how to do it and do it well. There's joy when I do something wrong and my children correct me. There's joy when I leave them with my parents for days and get a good report on their behavior. There's joy when my children choose to do something they don't want to do, just because it's right. There's joy when one of my children beats me in a board game I'm pretty good at. It's true that some of these joys don't come along very often :-).

As a Christian, I'm led to understand that God has some of these feelings towards us. He lets us take risks with ourselves and others --- not so that we can be independent of him or surpass him --- but so that we can be full-fledged people whom he can delight in and who can love him in return.

Two Reverse-Execution Optimizations

rr 4.0 will be the first rr release where reverse execution is reliable. I want to release it soon, but some users still find situations where reverse executions that seems to hang. I've implemented a mitigation by allowing ctrl-C to interrupt reverse execution (though I think there are still bugs where sometimes it fails to interrupt cleanly), but there are also underlying issues that can cause reverse execution to be very slow, and I want to fix as many of those as I can. I've recently fixed the two major fixable issues that I currently know of, and they're interesting.

For the following discussion, keep in mind that we implement reverse execution by restoring the program state to a previous checkpoint, then executing forwards and noting which breakpoints/watchpoints get hit. Once we reach our original starting point, we know which breakpoint/watchpoint firing is the "next" one in the reverse direction; we restore the program state to a previous checkpoint again, and execute forward again until we reach that target breakpoint/watchpoint.

The first issue involves reverse execution over conditional breakpoints/watchpoints that are hit frequently but whose condition almost always returns false. Following the above procedure, rr would reverse-continue to the most recent triggering of the breakpoint (or watchpoint), then stop and signal gdb; gdb would read program state via rr to evaluate the breakpoint, and then signal rr to continue with reverse execution. Thus, every time the breakpoint is hit, rr must perform at least two clonings of checkpoint state plus some amount of forward execution. That makes reverse execution unbearably slow if the breakpoint is hit hundreds of times.

Fortunately, gdb has an obscure feature that lets it offload evaluation of simple conditions to the remote target (in this case, rr) by expressing them as bytecode. So we add to rr a little interpreter for gdb's bytecode; then rr can implement reverse execution by executing forwards over some time interval, and every time a breakpoint is hit, evaluating its condition and ignoring the hit if the condition evaluates to false. If all goes well we only have to reconstruct program state for debugger stops where the user actually wanted to stop.

Unfortunately not all goes well. Some breakpoint expressions, e.g. those involving function calls, are too complex for gdb to express with its bytecode; in such cases, pathological slowdown of reverse execution is inevitable. So try to avoid using complex breakpoint conditions on breakpoints that are frequently hit during reverse execution. Moreover it turns out gdb has a bad bug in code generation, which requires an even worse workaround. I submitted a gdb fix which has landed upstream so hopefully one day we can get rid of the workaround.

Another case which is difficult to deal with is an unconditional breakpoint being hit very many times during the interval we're reverse-executing over. In this case just executing forward and stopping thousands times to find the last hit can be prohibitively expensive. Ideally we could turn off breakpoints, run forward until we're close to where reverse execution started, then enable breakpoints and continue forward again, hopefully only hitting a few occurrences of the breakpoint before we reach the last one. We can approximate this by detecting that we're hitting "too many" breakpoints while executing forward, and responding by cutting the remaining execution interval roughly in half, advancing to the start of the second half with breakpoints disabled, and resuming normal forward execution with breakpoints enabled --- possibly having to repeat the subdivision process if we're still in dense region of breakpoint hits.

I just landed these fixes on master. I'm very interested in hearing about any remaining cases where reverse execution takes a pathologically long time.

Midwinter Road Trip

It being the school holidays, I took a few days off for a short road trip around the central North Island with my and my brother's family --- two nights in Waitomo and two nights in Taupo. It is the middle of winter and we had a bit of a cold snap --- frosts every night --- but the weather was clear, fine and sunny. It was a thoroughly excellent trip all around: caves, rivers, countryside, mountains, board games, geothermal areas, tasty food, love and laughter. The South Island is more spectacular, but the central North Island remains dearest to my heart as a God-given blessing to all who live in and around it.

One highlight for me was "black-water rafting" at Waitomo: wetsuit, inner tube, helmet and head-lamp in an underground river. It's thrilling and a little crazy but not as testing as I'd been led to believe; highly recommended. Beyond the thrills, I will long remember drifting along underground in darkness and silence under the glow-worm-studded roof.

Another highlight was a hastily-planned trip up Mt Pureora between Waitomo and Taupo on Friday. We drove up Link Rd from highway 32 on the east side of the mountain --- it's gravel, but in very good condition --- and reached the trailhead a bit late at 3pm, but had plenty of gear and moved fast. We reached the summit by 4pm (beating the signposted time of 1.5 hours) and after 15 minutes at the top returned to the car at 5pm. There had been snow the previous night (closing the roads around Ruapehu/Tongariro --- otherwise we'd probably have been doing a walk there) and the walk was delightful, starting as a regular North Island bush walk and ascending into a winter wonderland with a light covering of snow and ice, opening at the summit into magnificent views of the central North Island --- Taupo, the Waikato, and the snow-capped volcanic plateau. There are many other tracks in the area, and I'd love to visit again.

rr Talk Video From TCE 2015

The good people at the Technion have uploaded video of the TCE 2015 talks, in particular video of my rr talk. My slides are also available. This is a fairly high-level talk describing the motivation and design of rr, delving into a few interesting details and concluding with a discussion of some of the exciting possibilities that could be enabled by rr and similar technology.

Whistler Hike

I'm at the Mozilla all-hands week in Whistler. Today (Monday) was a travel day, but many of us arrived yesterday, so today I had most of the day free and chose to go on a long time organized by Sebastian --- because I like hiking, but also lots of exercise outside should help me adjust to the time zone. We took a fairly new trail, the Skywalk South trail: starting in the Alpine Meadows settlement at the Rick's Roost trailhead at the end of Alpine Way, walking up to connect with the Flank trail, turning up 19 Mile Creek to wind up through forest to Iceberg lake above the treeline, then south up and over a ridge on the Skywalk South route, connecting with the Rainbow Ridge Loop route, then down through Switchback 27 to finally reach Alta Lake Rd. This took us a bit over 8 hours including stops. We generally hiked quite fast, but some of the terrain was tough, especially the climb up to and over the ridge heading south from Iceberg Lake, which was more of a rock-climb than a hike in places! We had to get through snow in several places. We had a group of eight, four of us who did the long version and four who did a slightly shorter version by returning from Iceberg Lake the way we came. Though I'm tired, I'm really glad we did this hike the way we did it; the weather was perfect, the scenery was stunning, and we had a good workout. I even went for a dip in Iceberg Lake, which was a little bit crazy and well worth it!

Bug In Newer Linux Kernels Affecting rr

As of kernel 4.0.4 (and probably earlier), writing to /proc/.../mem at a page which is mapped PROT_NONE causes a kernel BUG at mm/memory.c, which kills the writing process. This breaks rr, although I have a workaround which I'm about to land; the workaround temporarily makes the PROT_NONE regions we're writing to PROT_WRITE.

Trevor reported this in the kernel bugzilla already, though I was able to come up with a nice small test program.

Israel, Part 3

Friday June 5 was a relatively quiet day for me in Jerusalem. I slept in, then went for a walk to the nearby shops of Nachlaot to get some food and a replacement mini-suitcase --- my old little suitcase had fallen apart. It was fun to sit and eat pastries and watch the passers-by and wonder what their stories were. I made it back to my hotel before noon to drop off my gear and headed to the Israel Museum to check out their archaeological area for a couple of hours. It was as fascinating as I'd hoped. An unexpected bonus was the huge scale model of pre-Great-Revolt Jerusalem, giving me a really good feel for the scale and magnificence of the place in those times. I spent the rest of the day relaxing in the hotel lobby and later at Nimrod's friend's apartment, working on my laptop and generally taking it easy until we had another excellent dinner out. This was in preparation for a short sleep and a very early start --- 2:30am --- for our trip to climb the ancient mountain-fortress of Masada before dawn.

We left Jerusalem at 3am and reached the park entrance around 4:30am while it was still almost completely dark. Masada is a steep-sided flat-topped mountain about 350m high with our "Snake Path" winding up the eastern side. It was thrilling to climb that rocky path in that utterly barren landscape, in the dim half-light before dawn, following in the footsteps of countless ancients. We got to the top well before dawn and got some great photos of the sunrise over the Jordanian mountains, across the Dead Sea. Over the next four hours we thoroughly explored the extensive palaces, storehouses, cisterns and other ruins, most of which date back 2000 years to Herod the Great. Over those few hours the cool grey shadows gave way to blazing light and heat, and the Judean mountains emerged in their desolate glory. Masada is a breathtaking place, a perfect combination for my interests in hiking and history. The landscape and the ruins are compelling enough, but its greatest feature is the story of the Sicarii, the band of Jewish rebels who held Masada in 73 AD against the Romans, until the fortress finally fell and (as Josephus tells it) the defenders committed communal suicide just before the Romans entered. The remains of the besieging Roman camps are still clearly visible from above.

My last adventure in Israel was a stop at Kalia Beach on the Dead Sea on the way back to Jerusalem. It was a hot day --- 36C --- and a swim was most welcome, though bathing in the Dead Sea can hardly be called "swimming"! You're encouraged to avoid even the slightest amount of water getting on your face, which makes sense, because it's so saturated with salt it feels oily and you can pick up precipitated salt crystals from the bottom. But it's a lot of fun to lie on.

On the way back to Jerusalem I saw camels. Awesome!

The rest of Saturday was a late lunch in Jerusalem (excellent, again), and Nimrod driving me back to the airport for an uneventful flight home. After the sunny deserts of Israel it was a bit of a relief to return to the cool, wet greenery of Auckland!

The trip was even more memorable than I'd expected. Nothing disappointed, everything was fascinating, and the food was unexpectedly sublime. My hosts Eran, Ruth, Nimrod and Nimrod's friends were simply wonderful. Now when I read in the Bible mentions of Jerusalem and other places I have a much better idea what they are --- and were --- really like, and for that and everything else I'm extremely grateful.