Tuesday, 23 June 2015

Whistler Hike

I'm at the Mozilla all-hands week in Whistler. Today (Monday) was a travel day, but many of us arrived yesterday, so today I had most of the day free and chose to go on a long time organized by Sebastian --- because I like hiking, but also lots of exercise outside should help me adjust to the time zone. We took a fairly new trail, the Skywalk South trail: starting in the Alpine Meadows settlement at the Rick's Roost trailhead at the end of Alpine Way, walking up to connect with the Flank trail, turning up 19 Mile Creek to wind up through forest to Iceberg lake above the treeline, then south up and over a ridge on the Skywalk South route, connecting with the Rainbow Ridge Loop route, then down through Switchback 27 to finally reach Alta Lake Rd. This took us a bit over 8 hours including stops. We generally hiked quite fast, but some of the terrain was tough, especially the climb up to and over the ridge heading south from Iceberg Lake, which was more of a rock-climb than a hike in places! We had to get through snow in several places. We had a group of eight, four of us who did the long version and four who did a slightly shorter version by returning from Iceberg Lake the way we came. Though I'm tired, I'm really glad we did this hike the way we did it; the weather was perfect, the scenery was stunning, and we had a good workout. I even went for a dip in Iceberg Lake, which was a little bit crazy and well worth it!

Saturday, 20 June 2015

Bug In Newer Linux Kernels Affecting rr

As of kernel 4.0.4 (and probably earlier), writing to /proc/.../mem at a page which is mapped PROT_NONE causes a kernel BUG at mm/memory.c, which kills the writing process. This breaks rr, although I have a workaround which I'm about to land; the workaround temporarily makes the PROT_NONE regions we're writing to PROT_WRITE.

Trevor reported this in the kernel bugzilla already, though I was able to come up with a nice small test program.

Wednesday, 17 June 2015

Israel, Part 3

Friday June 5 was a relatively quiet day for me in Jerusalem. I slept in, then went for a walk to the nearby shops of Nachlaot to get some food and a replacement mini-suitcase --- my old little suitcase had fallen apart. It was fun to sit and eat pastries and watch the passers-by and wonder what their stories were. I made it back to my hotel before noon to drop off my gear and headed to the Israel Museum to check out their archaeological area for a couple of hours. It was as fascinating as I'd hoped. An unexpected bonus was the huge scale model of pre-Great-Revolt Jerusalem, giving me a really good feel for the scale and magnificence of the place in those times. I spent the rest of the day relaxing in the hotel lobby and later at Nimrod's friend's apartment, working on my laptop and generally taking it easy until we had another excellent dinner out. This was in preparation for a short sleep and a very early start --- 2:30am --- for our trip to climb the ancient mountain-fortress of Masada before dawn.

We left Jerusalem at 3am and reached the park entrance around 4:30am while it was still almost completely dark. Masada is a steep-sided flat-topped mountain about 350m high with our "Snake Path" winding up the eastern side. It was thrilling to climb that rocky path in that utterly barren landscape, in the dim half-light before dawn, following in the footsteps of countless ancients. We got to the top well before dawn and got some great photos of the sunrise over the Jordanian mountains, across the Dead Sea. Over the next four hours we thoroughly explored the extensive palaces, storehouses, cisterns and other ruins, most of which date back 2000 years to Herod the Great. Over those few hours the cool grey shadows gave way to blazing light and heat, and the Judean mountains emerged in their desolate glory. Masada is a breathtaking place, a perfect combination for my interests in hiking and history. The landscape and the ruins are compelling enough, but its greatest feature is the story of the Sicarii, the band of Jewish rebels who held Masada in 73 AD against the Romans, until the fortress finally fell and (as Josephus tells it) the defenders committed communal suicide just before the Romans entered. The remains of the besieging Roman camps are still clearly visible from above.

My last adventure in Israel was a stop at Kalia Beach on the Dead Sea on the way back to Jerusalem. It was a hot day --- 36C --- and a swim was most welcome, though bathing in the Dead Sea can hardly be called "swimming"! You're encouraged to avoid even the slightest amount of water getting on your face, which makes sense, because it's so saturated with salt it feels oily and you can pick up precipitated salt crystals from the bottom. But it's a lot of fun to lie on.

On the way back to Jerusalem I saw camels. Awesome!

The rest of Saturday was a late lunch in Jerusalem (excellent, again), and Nimrod driving me back to the airport for an uneventful flight home. After the sunny deserts of Israel it was a bit of a relief to return to the cool, wet greenery of Auckland!

The trip was even more memorable than I'd expected. Nothing disappointed, everything was fascinating, and the food was unexpectedly sublime. My hosts Eran, Ruth, Nimrod and Nimrod's friends were simply wonderful. Now when I read in the Bible mentions of Jerusalem and other places I have a much better idea what they are --- and were --- really like, and for that and everything else I'm extremely grateful.

Thursday, 11 June 2015

Israel, Part 2

My Technion hosts Eran and Ruth were very eager to ensure I had a good time in Israel, and I'm immensely grateful to them for that. Ruth organised a day trip from Haifa to Nazareth for me (and another conference-related person) on Wednesday, with a Christian Arab guide called Nuha ("mind" in Arabic) who's based in Nazareth. We started at the International Mary of Nazareth Center, which has a beautiful chapel and a good view over Nazareth, and is built over a display of the ruins of a house from around the time of Jesus. The Center has a theme of Christian unity and combines elements of many Christian traditions --- I felt it very uplifting. Almost next door is the Basilica of the Annunciation, a Catholic cathedral celebrating the site where the angel announced to Mary her pregnancy. Parts of the cathedral are decorated with Madonna-and-child images donated by Catholics from many different cultures, and it's heartwarming to see how they've interpreted the scene according to their own cultures! The Basilica is a great example of how history piles up in layers in Israel; there was originally a Byzantine church on the site, Crusaders built on top of the ruins of that, and very recently Catholics built over those ruins. At each stage, some of the older buildings were preserved, so inside the modern cathedral you see parts of the walls of the Crusader church, which enclose even more ancient Byzantine ruins. We also visited the church at the traditional site of Joseph's home, a hundred-year-old cafe, a slightly older house with ceilings painted in the style of Orthodox iconography, the "White Mosque", and a spice shop built in an old milling factory --- all very interesting.

On Thursday I took the train to Tel Aviv to meet Martin Odersky and a guide to take us on a tour of Jerusalem --- again arranged by Ruth. Our guide first took us to look out over the area from a church tower at the Mount of Olives, which still has many olive trees. We parked near the city hall, and walked around the city walls past the Jaffa Gate and the Tower of David to enter the old city at the Zion Gate. The city walls of Jersualem are another place where every era has visibly built on the ruins of what came before. Inside the old city, there's so much history all piled together I can't even remember everything we saw. Buildings are piled atop and against each other in time and space, connected by a maze of roads, alleys, stairs, and tunnels ... and less visibly by the lives of countless generations of pilgrims, families, rulers and servants who frequented them. A few scenes particularly struck me: Jewish men studying Torah at the Tomb of David; the partially excavated "Broad Wall" of Hezekiah, 2,600 years old; joyous bar mitzvahs at the the Western Wall; the Via Dolorosa; the Church of the Holy Sepulchre with its polyglot patchwork of Christian communities worshiping --- and fighting --- together. Along the way we squeezed in a very nice meal at the "Abu-Taher" restaurant in the Muslim Quarter. It may be trite to say so, but the Old City of Jerusalem was the highlight of my whole trip. The place is steeped in history, politics, culture and religion like no other.

When you consider the past and present state of Jerusalem, it's easy to see tragedy, but to me it's also awe-inspiring to think about the generations of pilgrims, worshipers and warriors who have lived, prayed and died there through the millennia. Sin and redemption, destruction and creation are all part of the story of Jerusalem, as in all stories of God's dealings with us. I look forward to a time when God will make all things well, but I think we'll be treasuring those stories into eternity.

After checking into my hotel on Thursday evening, I had some time on my hands so I walked to the World Zionist Organization's gardens on Mount Herzl and then to the Yad Vashem Holocaust memorial. There's a path that physically and thematically connects the two, with commentary drawing the connection between the aftermath of the Holocaust and the establishment of the Israeli state. Even though I was familiar with much of the content, Yad Vashem was a powerful and humbling experience. There's no escaping the fact that --- with some notable exceptions --- Christians generally abetted or inadequately resisted the Holocaust. The question one has to ask oneself is: would I have done better? If another such test comes, will I do better? I hope reminders like Yad Vashem sharpen our resolve.

Tuesday, 9 June 2015

Israel, Part 1

I just got back today from an amazing trip to Israel. I was invited by the Technion to speak at their TCE 2015 conference. So I did that, added a few days for sight-seeing since I don't like to travel halfway around the world for just a couple of days of work, and also had a full day in Hong Kong on the way there.

I've never been outside the airport in Hong Kong before, so I was excited to finally do that. I took the tram up Tai Ping Peak, walked the Lugard Road circuit and back down to the city, then spent the afternoon at Cheung Chau island. It was incredibly hot and humid, and I was so busy looking for drinks and icy desserts that I forgot to eat proper food until the end of the day. I enjoyed the coastal track on Cheung Chau island, through jungle and the back streets of villages, though the Cheng Po Tsai "pirate cave" was a bit disappointing --- comparable sea caves at Piha don't even have names :-).

Checking into my El Al flight I experienced the famous El Al security. The very nice security man asked me endless questions, including topics like what I did in the USA and who I knew there from the Middle East. It all went well, aided by the Technion's letter, but I was nervous and felt a bit of an urge to confess to something (anything!) to make the man happy. Two local security officers with very large guns were watching the whole area and I wondered what happens if you accidentally give a wrong answer...

Early Sunday morning I got a taxi from Ben Gurion airport to Haifa. Excitingly, the taxi driver drove at 130km/h most of the way, with one hand on the steering wheel --- because he only had one arm. I stayed at the Colony Hotel, which was generally excellent and who very generously checked me into a room at 6am! Later in the morning I walked around the waterfront and up the front side of the famous Mount Carmel, then to the Bahai Gardens tour with my friend Nimrod --- amazing. We had a late, long lunch with a couple of other conference people and I spent the rest of the day relaxing and working in my room.

An unexpected blessing of my trip was the food. I've never had much Middle Eastern/Mediterranean food before --- there's not much of it here in Auckland --- so I had a lot of very good food that's new to me. I will never think about hummus the same way again.

The TCE conference was held on Monday and Tuesday. It was an eclectic group of speakers, but most of them were very good at giving talks accessible to a broad audience, and I really enjoyed hearing a broad range of computer science talks like I used to in my grad student days at CMU. There were very interesting talks on complexity theory, databases, architecture, compilers, testing, verification, and more. One of the speakers was my former CMU officemate Bianca --- an unexpected pleasure to meet again in such an exotic location! Having been mostly out of the academic circuit for a long time, I felt many waves of imposter syndrome, and so I was rather nervous leading up to my talk. It all seemed to mysteriously come right as soon as I got started --- I'll chalk that up to God and prayer. Without a clock or a proper practice run, even the timing was perfect; we had time for a couple of questions before a wailing siren cut us off --- a scheduled national-defense drill. I think we were supposed to flee to a shelter but most people just went to get coffee. I had a great time talking to researchers at the conference, and learned a lot of useful information.

On Monday night we had a speakers' dinner at the Technion, on a balcony with a view towards the Golan (and on a clear day, Mount Hermon, apparently). It was sobering to think of the brutal civil war raging just beyond our vision. On Tuesday night I had one of the most extraordinary meals of my life at a Druze village near Haifa --- a dozen large side dishes, most new to me; a huge main dish consisting of kebab meats, rice and cheese, with a layer of bread baked on top; and an irresistable kanafe dessert. I had no desire or need to eat anything the whole of the following day --- which I'll cover in the next post.

Sunday, 7 June 2015

Small Change To rr Behavior

To address issue 1490, I just checked in an rr improvement that changes the behavior of gdb+rr in a user-visible way. When the last thread of the debuggee process exits, rr generates a fake SIGKILL signal before exiting the process. This gives the user a chance to reverse-execute from the end of the debuggee execution, when previously rr would have let the process terminate normally so reverse execution cannot occur. The new behavior is a little bit of a lie since in most cases SIGKILL was never actually sent, but I hope the usefulness of this new behavior outweights any possible confusion.

Saturday, 23 May 2015

rr Performance Update

It's been a while (March 2014 to be precise) since I gathered meaningful rr performance numbers. I'm preparing a talk for the TCE 2015 conference and as part of that I ran some new benchmarks with mozilla-central Firefox. It turned out that numbers had regressed --- unsurprisingly, since we don't have continuous performance tests for rr, and a lot has changed since March 2014. In particular, Firefox has evolved a lot, our tests have changed, we're using x86-64 now instead of x86-32, and rr has changed a lot. Over the last few days I studied the regressions and fixed a number of issues: in particular, during the transition to x86-64 some of the optimizations related to syscall-buffering were lost because we weren't patching some important syscall callsites and we weren't handling the recvfrom syscall, which is common in 64-bit Firefox. I also realized that in some cases we were flushing much more data from the syscallbuf to the trace file than we'd actually recorded in the buffer, massively bloating the traces, and fixed that.

There are still some regressions evident since last March. Octane overhead has increased significantly. Forcing Octane to run on a single core without rr shows a similar overhead; in particular that alone causes one test (Mandreel) to regress by a factor of 10! My guess is that Spidermonkey is using multiple cores much more aggressively that it did last year and because it's carefully tuned for Octane, going back to a single core really hurts performance. Replay overhead on the HTML mochitests has also increased significantly; I think this is partly because we changed rr to disable syscall buffering on writes to standard output. This improves the debugging experience but it results in a lot more overhead during replay.

Overall though, I remain very happy with rr performance, especially recording performance, which is critical when you're trying to capture a test failure under rr. Replay performance is becoming more important since it impacts the debugging experience, especially reverse execution; but doing a lot of work to improve raw replay performance is low priority since I think there are projects that could provide a better improvement in the debugging experience for less work (e.g. the ability to take a checkpoint during a recording and start debugging from there, and implement support for gdb's evaluate-in-target conditional breakpoints).