Improving Debugging Workflow With Pernosco

One of the key challenges for debuggers is that the traditional interactive debugging workflow — running your program interactively and starting it under the debugger or connecting to it once it's running, and pausing it to inspect its state — doesn't work well for a lot of people anymore. That workflow isn't convenient when the application normally doesn't run locally — e.g. because testing more often happens in CI, or on a phone, or the code you care about runs as part of a big distributed system. It also falls down when pausing the debuggee breaks the system. As software has increasingly moved to the cloud and mobile platforms, this has become a bigger deal and it's no wonder use of interactive debugging has waned. "Remote debugging" helps a bit, but it tends to be painful and although it can bridge gaps between machines, it doesn't bridge gaps in time.

We've published a couple of documents on how Pernosco tackles this, in particular how Pernosco integrates with CI and how Pernosco supports uploads from developers and QA (manual and automatic). A big part of the solution is just record-and-replay (with rr in our case). Being able to record execution on one machine, without stopping the application, and replay execution on another machine at another time, enables a lot of new workflows that mitigate the above problems. However Pernosco goes further in some important ways.

One issue is that just being able to replay execution isn't enough; we also want a good debugging experience during the replay. This means we need to capture compiled debuginfo, source code and other relevant information that aren't strictly necessary for the replay. In many cases that data isn't even available at the recording site, but it might be available somewhere (e.g. a symbol server or build artifact archive) for us to get later. So our debugging infrastructure has to support collecting information at the recording site, harvesting it from various sources later, and actually using it during the debugging session. This is not at all trivial, and Pernosco has a lot of code to handle this sort of thing, some of which needs to be customized for specific customers. For example, Pernosco identifies Firefox binaries built by Mozilla CI and knows how to locate the relevant symbols and sources from Mozilla's archives. For developer and QA-submitted recordings, Pernosco examines the trace to locate relevant debuginfo and source code and upload them. For source code hosted in well-known public repositories (e.g. mozilla-central or Github), we minimize overhead by uploading only local changes and having our debugger client fetch the public changes from the public repository at debugging time.

Note that rr on its own provides trace portability but debugging ported traces is tricky. With rr pack and rr record --disable-cpuid-features, it is generally possible to create rr recordings that can be replayed on other machines. However, when you replay with gdb, locating symbols and source files is problematic when the replay machine filesystem does not exactly match the recording machines. For example when gdb sees the shared-library loader load /home/roc/libfoo.so, that file might not be present at that location on the replay machine (or worse, it might be a different version) so gdb won't load the right symbols. You can try to work around this by populating a "sysroot" directory with the relevant files, copied and renamed from the trace, but figuring out which trace files need to go where is hard (because e.g. it depends on the symlinks present on the recording machine, which rr doesn't capture in the recording, and it's not even clear how you'd do that).

Another important feature for enabling new workflows is just having a cloud-based Web client. We want to minimize the barrier to getting into a debugging session, and it's hard to think of an easier way than publishing a link which the user clicks on to enter a specific debugging session — no installation, no configuration. Those links can be published wherever you already notify users about test failures.

One thing I'm really excited about is that Pernosco enables splitting failure reproduction from debugging. Traditionally, developers had to reproduce a bug locally when they wanted to use an interactive debugger to debug it. Pernosco lets you delegate the reproduction step to other people (or automation). For example, when QA staff find a bug, instead of writing down the steps to reproduce to send to a developer (and inevitably having a back-and-forth discussion about exactly what's required to reproduce the bug, etc), QA can upload a recording to Pernosco and pass the link to the developer. This saves time and money — especially when QA staff are cheaper and/or more scalable then your developer team.

Auckland Half Marathon 2019

I ran the Auckland Half Marathon last Sunday. My time was 1:46:51, a little slower than last year. I didn't quite have the mental endurance I guess. As always I hope I'll do better next year, though I am getting older...

As usual, I ran barefoot. This year I had climbing tape again which definitely makes it a bit easier on my feet at this speed.

People ask why I run barefoot. I didn't start running until I was about 35, and I started running on a beach where I'm usually barefoot. After I got used to that, it never felt right to run in shoes ... they're just weight on my feet. To be honest, I also like being a bit eccentric. I've never had any serious issues with injuries so at this point, I don't feel like changing anything. I did try wearing Vibram 5-Finger foot-gloves for my first half-marathon, but my feet got all sweaty and were still sore at the end so they don't seem to help me much. Sticking small patches of climbing tape to the hardest-wearing parts of my feet (the balls, and my second toes) is plenty of protection when running fast. When running slower (20K training runs around 2:10) I don't even need that.

Pernosco Demo Video

Over the last few years we have kept our work on the Pernosco debugger mostly under wraps, but finally it's time to show the world what we've been working on! So, without further ado, here's an introductory demo video showing Pernosco debugging a real-life bug:

This demo is based on a great gdb tutorial created by Brendan Gregg. If you read his blog post, you'll get more background and be able to compare Pernosco to the gdb experience.

Pernosco makes developers more productive by providing scalable omniscient debugging — accelerating existing debugging strategies and enabling entirely new strategies and features — and by integrating that experience into cloud-based workflows. The latter includes capturing test failures occurring in CI so developers can jump into a debugging session with one click on a URL, separating failure reproduction from debugging so QA staff can record test failures and send debugger URLs to developers, and letting developers collaborate on debugging sessions.

Over the next few weeks we plan to say a lot more about Pernosco and how it benefits software developers, including a detailed breakdown of its approach and features. To see those updates, follow @_pernosco_ or me on Twitter. We're opening up now because we feel ready to serve more customers and we're keen to talk to people who think they might benefit from Pernosco; if that's you, get in touch. (Full disclosure: Pernosco uses rr so for now we're limited x86-64 Linux, and statically compiled languages like C/C++/Rust.)

Food In Auckland 2019

Some places I like these days that are (mostly) pretty close to my house:

• Jade Town (Dominion Road): Uighur food. A bit Chinese, a bit Indian, but not much like either.
• Cypress (Dominion Road): New-ish yum cha (dim sum) place. Some interesting dishes I'd never seen before.
• Viet Kitchen (Dominion Road): More authentic, bit more expensive Vietnamese.
• Hot And Spicy Pot (Dominion Road, city): tasty pay-per-weight stir-fry.
• Barilla (Dominion Road): Decent cheap dumplings and other Chinese food.
• Tombo (Newmarket): Quality Korean-Japanese BBQ/hotpot buffet.
• Hansan (Newmarket, city): Still a favourite cheap Vietnamese-Chinese place.
• Master Dumpling (Newmarket): good dumplings, great sweet potato in melted sugar.
• Faro (Newmarket): Nice Korean lunch combo.
• Momotea (Newmarket): Taiwanese-style cafe with strong frozen drink selection.
• Selera (Newmarket): Malaysian cafe food.
• Sun World (Newmarket): Reliably good yum cha.
• Kimchi Project (city): Kimchi fusion-eseque place. Kimchi carbonara is great.
• BBQ Duck Cafe (city): Good value tasty Hong Kong-style cafe.
• Nol Bu Ne (city): Great value Korean food.
• Uncle Man's (city): Malaysian food — best roti around IMHO.
• Kiin (Mt Eden): Great cheap Thai food.
• Altar (Mt Eden): Good value sandwhich+fries lunch special.
• Corner Burger (Mt Eden): Great burger+fries+shake combo — the shakes are especially good.
• Gangnam Style (Takapuna): Korean BBQ buffet. The worst thing about this place is the name.
• Petra Schwarma (Kingsland): Really nice Jordanian food.
• Mama Rich (Greenlane): Good Malaysian cafe food, a bit cheaper than Selera.
• Chocolate Boutique (Parnell): Not exactly a restaurant but for dessert, a great option.

Pouakai Circuit

Last weekend I went with a couple of people down to Mt Taranaki to tramp the Pouakai Circuit, planning to do it leisurely over three days. We drove down on Saturday morning (about 5 hours from Auckland), had a fine lunch at the Windsor Cafe in Inglewood, and hiked from a carpark up to Holly Hut via the Kokowai Track. (Normally you'd take the Holly Track from the Egmont Visitor's Centre but that track is currently closed due to a slip, though it should be open soon.) The posted time was 4.5 hours but we did it in 3 hours; my companions are very fast, and I'm not too bad myself though the steep uphill with steps made me struggle a bit! The weather was great and we had some excellent views of Mt Taranaki along the way. It's a beautiful mountain with snow cover on a clear winter's day.

Holly Hut is a fine hut. Some generous donors installed LED lighting, which is great during the winter when the days are short. It was the first Saturday of the school holidays but there were just two other people there. Late Saturday afternoon we did the ~1 hour return side trip to Bell Falls, which were beautiful. The weather had gotten cloudy, drizzly and misty and the daylight was fading, but that added to the effect!

On Sunday morning we had a bit of a late start because we thought we'd spend Sunday night at Pouakai Hut, which is nominally 2.5 hours walk away. The track crosses Ahukawakawa Swamp, in a basin between the old Pouakai volcano and the slopes of Mt Taranaki — most picturesque, especially in drizzly foggy weather. We had been hoping to do the side track to the Pouakai summit on the way to the hut, but bursts of rain and the likelihood of seeing nothing but cloud dissuaded us. We got to Pouakai Hut in about 1.5 hours and had an early lunch, then had to think hard: with phone reception at the hut, we got a new weather forecast for Monday, which promised heavy rain, wind, lower temperatures and possible thunderstorms — even less promising than the rain currently beating against the hut. We decided to avoid that weather by finishing the circuit on Sunday and driving back to Auckland late. That plan worked out pretty well; it didn't rain much during the rest of Sunday and although our fast pace left us all feeling a bit tired when we got back to the carpark, we had fun and got back to Auckland before midnight. All in all, a short trip with lots of driving and lots of fast walking, but still a great chance to appreciate God's creation.

I'm looking forward to doing this trip again when the weather's better and the Holly Track is open! It's quite accessible from Auckland, and we could do it with less fit people if we take more time. We could also save some energy by starting at the Egmont Visitor's Centre at the top end of the road, then taking the Holly Track to Holly Hut, then returning via Pouakai Hut to the lower carpark and sending some fitter people up the road to fetch the car.

Is Richard Dawkins A Moral Realist?

An interview with Richard Dawkins in the New Scientist (21 September 2019) contains this exchange:

Graham Lawton: Another chapter in your book looks at progress in moral issues such as gender and racial equality, and you present a very upbeat picture. Do you worry that progress has gone into reverse?
Richard Dawkins: No. It's important to take the long view. I think there's absolutely no doubt that we're getting better as the centuries go by. The moral standards of a 21st century person are significantly different from those of a 20th century person.

Dawkins here seems to assume there are objective moral standards against which human moral opinions can be measured, i.e. moral realism. This surprised me because Dawkins is such a strong advocate for naturalism and it has always seemed obvious to me (including before I became a Christian) that naturalism is incompatible with moral realism — the famous is-ought gap. Sean Carroll, for example, has written about this much better than I could. In fact, Dawkins has apparently written (in River Out of Eden, quoted here):

The universe we observe has precisely the properties we should expect if there is at the bottom, no design, no purpose, no evil and no good.

so unless he's changed his mind about that, it seems Dawkins at least professes to be a moral anti-realist.

This confusion illustrates why I've always found moral anti-realism so deeply unsatisfactory in real life. One can argue that there are no objective moral facts, and that moral claims simply express opinions shaped by evolution and culture etc, and even try to believe those things — but the temptation to think, speak and act as moral realists seems practically irresistible ... so much so that even the most prominent moral anti-realists consistently yield to it, and hardly anyone even notices.

Addendum Arguably the New Scientist quote could be interpreted in other ways, e.g. that by "getting better" Dawkins meant more internally consistent, or more in accordance with his personal subjective moral opinions. However I think it's obvious most people would interpret it as "objectively morally better" and thus if Dawkins meant something else, he needs to work a lot harder at eliminating such misleading language.