Eyes Above The Waves

Robert O'Callahan. Christian. Repatriate Kiwi. Hacker.

Thursday 22 October 2015

rr 4.0 Released With Reverse Execution

I finally got around to releasing rr 4.0!

rr is Mozilla's low overhead record-and-replay debugging tool for large C++ applications like Firefox. For more information see the rr project page, or watch my Technion talk:

rr 4.0 is the first stable release with reverse-execution enabled. This means gdb's reverse-continue, reverse-step, reverse-next and reverse-finish commands work under rr. Not only do they work, but under rr they're quite efficient and have unlimited scope. It's completely reasonable to reverse-continue from the end of a Firefox run all the way back to the beginning. Furthermore, breakpoints and hardware data watchpoints work with reverse execution. Suppose we're debugging Firefox layout:

Breakpoint 1, nsCanvasFrame::BuildDisplayList (this=0x2aaadd7dbeb0, aBuilder=0x7fffffffaaa0, aDirtyRect=..., aLists=...)
    at /home/roc/mozilla-inbound/layout/generic/nsCanvasFrame.cpp:460
460   if (GetPrevInFlow()) {
(gdp) p mRect.width
We happen to know that that value is wrong. We want to find out where it was set. rr makes that quick and easy.
(gdb) watch -l mRect.width
(gdb) reverse-cont
Hardware watchpoint 2: -location mRect.width
Old value = 12000
New value = 11220
0x00002aaab100c0fd in nsIFrame::SetRect (this=0x2aaadd7dbeb0, aRect=...)
    at /home/roc/mozilla-inbound/layout/base/../generic/nsIFrame.h:718
718       mRect = aRect;
This is extremely powerful. rr's regular users inside (and outside) Mozilla have been enjoying this for a while on rr master, so we're glad to be able to get it into a stable release.

This release is about as stable as rr has ever been --- meaning it seems to be generally working for people and very few new issues have been coming in. However, rr depends on gnarly details of the kernel, system libraries and CPU so I expect as the number of rr users grows we'll need to keep fixing bugs and updating rr regularly.

By the way, Mozilla is interested in hiring people to work on rr-related technology, more closely integrated with the browser. I'd like to hear from people who're interested!


Hi Robert, you may be interested in the SHIM profiler. https://github.com/ShimProfiler/SHIM/blob/master/papers/ISCA-SHIM.pdf
This looks interesting... How does it compare to Intel's PIN tool? -Andrew
This looks interesting... How does it compare to Intel's PIN tool? -Andrew
PIN instruments the code intensively. It is good at logging information of the instrumented code, but very bad in term of observing the behavior of the original code because the observer effects. For example, PIN can log the address of executed functions, but doing that would change the icache miss rate significantly. So, if users want to understand the icache behavior, they have to build another model and feed the logged data in. SHIM could observer the icache misses signal and other software signals at high frequency. So, you can get a timeline view about the number of icache misses every hundreds cycles and corresponding software information, such as which functions are executed in this short period. It is very useful in term of observing software's hardware utilization. For example, if you use SHIM to observe firefox on Intel new CPU, you would found that periods in which firefox has very high L0 cache miss (L0 is a small decoded icache, L1i is the icache).
SHIM sounds pretty cool, though it's not directly related to rr.
Thanks. Right, SHIM is not directly related to rr. It focus on performance debugging. By the way, are you or any other firefox developers are interested in improving the CPU utilization of firefox on intel CPU? I am very happy to share some numbers how firefox underutilizes the micro-architecture resources.
I'd be interested to see the numbers! Though I expect we're mostly bounded by the memory system.
How to send you? Would you like to drop me an email? Main is xi.yang@anu.edu.au. By the way, I work with Steve Blackburn, I think you know him. It is interesting that similar with Chrome, it is pretty much front-end bounded, for example, bad instruction cache, poor instruction decoding throughput, etc.
> rr 4.0 is the first stable release with reverse-execution enabled. Mmmm the Debian package for 3.2 has working reverse-execution.
It wasn't stable in 3.2.0.
this tool is awesome! I just used it today to debug a SIGSEGV in the JVM. The ability to watch a memory location and do reverse-continue feels like having super powers when debugging. The only drawback is that it runs the program on a single core and the JVM makes heavy use of multiple threads, so a large part of our trickiest bugs consists of data races :/ still, if a bug manifests itself while being recorded with rr, then one can almost consider the bug solved :)
Thanks! Feel free to add to https://github.com/mozilla/rr/wiki/Testimonials :-). It is possible to catch races in rr if context switches occur at the right times, but it is hard. Figuring out better strategies for context switching to find races is on my TODO list.
Hi, looks nice! Out of curiosity, how does it compare with QIRA (QEMU Interactive Runtime Analyser)? https://github.com/BinaryAnalysisPlatform/qira http://qira.me/
I haven't tried it, but AFAICT QIRA has a similar architecture to Chronomancer: http://robert.ocallahan.org/2007/08/announcing-chronomancer_21.html That implies that the recording overhead will be much higher than with rr.
Isn't QIRA essentially what he talked about in "Omniscient debugging"? And for that reason, QIRA doesn't really compare with rr in terms of performance.
Rob Irving
Hi Robert. I'm the host of CppCast, a podcast focused on C++ development. We recently discussed RR as a news item on our show (http://cppcast.com/2015/10/andrei-alexandrescu/) and were wondering if you'd be interested in coming on the show to discuss the project in more depth. Please let me know if you'd be interested: feedback [at] cppcast [dot] com