Mercurial

I've started using Mercurial, and it's been mostly good. I've run into a few issues that we should address:

• Mercurial has its own diff.exe and patch.exe tools. I accidentally had them on my path before the MSYS tools and this seemed to cause problems (like 'patch' converted line endings to CRLF in every file it touched). It would be nice if bsmedberg or someone else rolled Mercurial into the Windows build tools package so that other people don't get the opportunity to make this mistake.
  
• hg diff doesn't seem to offer a way to change the number of lines of context. So I had to enable and configure the extdiff Mercurial extension. That was a bit annoying.
  
• http://hg.mozilla.org/cvs-trunk-mirror has a nice mirror of CVS into Mercurial, updated frequently, with history back to March. This is a nice place to pull from. Unfortunately it doesn't include everything you need to build Firefox; in particular NSPR and NSS must still be pulled from CVS, and extensions/xmlextras was not imported so you need to --disable-xmlextras in .mozconfig (which used to break libxul builds, although I managed to land a fix for that today). There's a client.py script in http://hg.mozilla.org/mozilla-central that can do the CVS pulls of NSS and NSPR, but mozilla-central is not a good place to pull from (it's not updated regularly) so you need to get that script seperately, and then that script has Tamarin stuff in it so I didn't like the look of it... I ended up just pulling NSS and NSPR with manual CVS commands. We need to make it easier to use Mercurial to get a working build. Perhaps someone could embed Mercurial knowlege directly into client.mk so "make -f client.mk pull_all" works with a Mercurial repository just as well as it does with CVS?
  

On the bright side, Mercurial is fast, even on Windows: whole-tree diffs take me just 5-10 seconds, whole-tree revert is even faster, pulls are fast, and all the basic functionality seems to work fine (I haven't done any merging or branching yet, nor have I used MQ). Best of all the underlying model seems quite simple and I think I understand it having read a few chapters of hgbook. I used git with cairo without ever really understanding what git was doing.

It seems to me that if cvs-trunk-mirror (or another Mercurial repository) was set up to make it easier to work with for regular trunk builds, developers could move over right now, except for actually checking in patches. Then we could have a flag day where instead of checking into CVS and having the changes propagated to a Mercurial mirror, we check into a Mercurial repository and the changes get propagated to a CVS mirror. LXR, Bonsai, Tinderbox and other tools could work off the CVS mirror so the switch would be pretty easy. There must be something wrong with this picture because smarter people than me have already said it's harder than that :-).

OOXML Wars

This is a hot topic in New Zealand right now, and an interesting relevant rant about OOXML technical issues just appeared on Slashdot, so I feel like getting my 2c in...

Rod Drury's blog has an interesting comment:

The antagonists seem to change between the spec being either

a) Too long and detailed for people to be able to implement or;

b) Too vague and lacking in specific detail for people to implement.

It's entirely possible for both of these to be true at the same time --- when the thing you're trying to specify is much too complicated, so that a specification with all the necessary details is much too large to be correctly implemented by anyone. That is exactly the situation OOXML is in; it wants to specify every detail of the behaviour of Microsoft Office, millions of lines of code accumulated over 20 years.

People are worried about the cost of migrating documents from Word format to ODF. I don't think anyone should be advocating such a migration; legacy documents, if converted at all, should simply be printed to PDF for archival. The real issue is what format new documents should be produced in. OOXML is clearly a horrible format from a purely technical point of view --- not even its defenders seem to be challenging this --- so there's a strong argument that you do not want to be producing new documents in OOXML if you have a choice. I think having OOXML stamped by ISO as a de jure standard would send the wrong message about that. On the other hand, making it an standard does not help at all with the problem of preserving legacy documents.

The overall most cost-effective and future-proof solution IMHO is for ODF to be the single de jure office document standard, for Microsoft to ship ODF read/write capability in its products, and for people who care about the longevity and openness of their documents to make ODF the default for new documents, while continuing to work on old documents in their existing formats. Presumably Microsoft opposes this approach because it would ever-so-slowly weaken people's dependence on their products.

Someone might wonder whether support for the WHATWG's efforts to standardize "Web-compatible" HTML is consistent with opposition to Microsoft's efforts to standardize "Word-compatible" OOXML. It is, because there are actually many differences. The main difference is that Ian and his gang aren't just specifying "whatever IE does"; they typically look at what a number of implementations do, and choose the behaviour that makes the most sense while remaining "Web compatible" (which is often, but certainly not always, what IE does). This is made easier because HTML and its related technologies originated in the standards world, whereas Microsoft's Office formats have always been wholly their own monstrous babies. (Another major difference, I suspect, is that conversion of HTML documents is a lot less feasible due to greater reliance on embedded scripting to give meaning to pages.)

Announcing Chronomancer

I'm very pleased to announce the release of Chronomancer, an Eclipse-based UI for Chronicle. I'm releasing it under MPL 1.1. Thanks to Mozilla for letting me do this.

As I mentioned in my last post, we successfully used Chronomancer last week to track down a bug in Gecko's cycle collector. I think that means this is the right time to release it in the hope that other people will find it useful or interesting, even though it is still far from complete. Release early, release often and all that. I also hope that people will be interested in contributing, since I certainly don't have enough time to do a lot of work on it.

I have posted a screencast of Chronomancer in action, debugging the cycle collector bug. I apologise in advance for my mumbling and backtracking and the general roughness of the demo; I don't have time to make it great. I hope it communicates the power of the tool.

Chronomancer has the following features:

• A Timeline view that shows events of interest, such as function calls and memory writes; time increases from top to bottom. Function calls display their parameter values. Clicking on an event makes it the "current event".
  
• The Timeline contains two buttons, one to connect to a Chronicle database and another to close the connection.
  
• The Timeline's dropdown menu lets the user add new queries to the timeline. Currently the only available query is to find invocations of a named function.
  
• A Call Stack view that implements history-based stack reconstruction to display a stack for the current event. Each stack frame shows the parameters to the call at the time when the function was invoked. Double clicking on a stack frame adds the start and end of the function call to the timeline, and also adds a "caller" event to the timeline and selects that so you see the context in which the function was called.
  
• A Locals view shows the values of local variables (including parameter values) for the currently-executed function.
  
• A Data view shows the values of chosen variables in memory. Double-clicking on a pointer in any view will add the target of that pointer to the Data pane. Whenever the current event changes the Data pane updates to show the values of the variables at the chosen time.
  
• The context menu for variable values includes "Find Last Write" and "Find Next Write" commands. Currently these commands only work on variables in the Data pane (fixing that isn't too hard but I haven't gotten to it yet).
  
• Chronomancer will open source windows to show the current line, if you create an Eclipse project containing the source files. (You can do this fairly easily for most C/C++ projects, and you don't have to use Eclipse to build or edit your code; see Getting Started in the Chronomancer wiki for more information.) The current line is highlighted in green. Other lines in the function that were executed in the same function invocation are highlighted in amber. For function invocations containing loops, the loop head is highlighted in red and only the annotations for the current (or nearest) iteration are displayed. (It would be easy to add commands to display other iterations but I haven't gotten to that yet.)
  
• The context menu for the vertical ruler in source panes has two new commands. "Show Callees" adds all callees for the executed line to the timeline so you can work with them. Chronomancer automatically descends into the first callee. "Set Current Line" adds the executed line to the timeline and makes it the current event; this is handy for displaying the values of variables at that point. (Stepping --- in either direction --- is obsolete!)
  
• The dropdown menu for all Chronomancer views contains a "Copy Contents" command that copies the view contents to the clipboard as text --- invaluable when you want to copy a stack trace to Bugzilla, for example, although with a bit more work copying a Timeline or Data pane to Bugzilla would also be interesting.
  

There is some interesting stuff under the hood. There's an org.ocallahan.chronicle package which provides a convenient Java binding to the Chronicle engine. It does most of the query wrangling on a worker thread so we can run algorithms like call stack reconstruction without blocking the UI. There's a core org.ocallahan.chronomancer package which manages the core Chronomancer state; most of the views and commands are just extensions that plug into that infrastructure. Chronomancer also offers two new kinds of extension points:

• Queries: The only query currently available is finding calls to a named function, but new types can be plugged in via an Eclipse extension point. This allows for project-specific queries (e.g., "Find XPCOM Assertion Failures").
  
• Type Renderers: Plugins can register renderers for specific C/C++ types. All the current variable rendering is done this way; for example Chronomancer currently includes a custom renderer for PRUnichar* (although this should really be broken out into a Mozilla-specific Chronomancer extension). Type renderers can create arbitrary Eclipse Draw2D widgets, which can offer their own custom UI.
  

There's so much that could be fixed or added; this was the bare minimum I could do to make something I felt was compelling. I won't even bother trying to list the possibilities right now. Maybe later I'll blog about some of the things that could be done.

Obligatory screenshot:

Post Vivem

What a week!

On Sunday I met with Eli and Fantasai to talk about vertical breaking issues. It was quite productive and we scoped out what should be done for 1.9 and beyond. For 1.9 we plan to do something hacky that will fix printing of pages where most of the page is in an abs-pos block that starts on the first page. This will solve most of the abs-pos printing pain. We will probably not be able to deal with abs-pos blocks starting after the first page, unfortunately. That will require changes that don't fit into 1.9. We also agreed that post-1.9 we should eliminate continuation frames for float placeholders; I used to be unsure whether this would be an overall simplification, but now that abs-pos placeholders are in the mix, it's clear this will be a win.

In fact I've become convinced that our entire continuation model is a bad idea. It's nice in some ways that our frame tree corresponds (mostly) to the CSS box tree, so that elements that generate a chain of CSS boxes create a chain of corresponding nsIFrames; but mostly it's just an enormous mountain of pain. Managing the continuations during DOM changes and dynamic layout is a ton of code, it's fragile, and leads to memory safety errors. It also leads to performance problems. For example, if you have a big chunk of text in inlines nested K levels deep, spread over N lines, then there are at least NK CSS boxes and therefore we create NK nsIFrame objects. This is both memory-hungry and slow especially for certain pages. If we just had one "frame" per element and stored the positions of line breaks and some other selected geometry data (like the Y-offset of each line's baseline and each line's overflow area) we would only need O(N + K) storage and I think we could still paint and layout as efficiently as we do now, probably better. Webkit doesn't have this continuation model and I'm sure that helps them a lot (although Webkit's architecture doesn't accomodate advanced cases of vertical breaking so I wouldn't just copy it).

On Monday I went to VMWare to give a talk about Chronicle and speak to people there. I had a great time and learned many interesting things. One thing I learned is that the overheads of VM record and replay are significantly higher than I'd believed --- probably a minimum of 20 or 30 percent. It's still a very interesting technology, though, and there's great potential synergy between VM replay and Chronicle-like tools. I hope that we can do some low effort cooperation there.

Tuesday through Thursday was the Mozilla Corporation all-hands meeting. It was all very good and as usual it was fun to see people I'd worked with but not yet met face to face. Some sessions were particularly interesting. There was a good discussion of Mercurial which pretty much convinced me to start using it via hg.mozilla.org's cvs-trunk-mirror. There was a spirited pair of sessions on post-1.9 planning. There are a number of interesting Gecko features that are well under way but I think will slip out of 1.9, and I'm also doubtful that a meaningful Mozilla2 release can happen next year, so I think it behoves us to ship a 1.9.1 release subject to constraints:

1. New features for 1.9.1 should be substantially done by the time we wrap up 1.9.
   
2. New features for 1.9.1 should not affect compatibility with existing Web pages.
   
3. We should be conservative about fixing bugs for existing features in 1.9.1, i.e., stick to fixing regressions, security bugs, or other critical bugs.
   

Given the first constraint, I think we could just about write down the list of 1.9.1 features right now. That also tells us who will be focused on 1.9.1; the rest of us --- much the majority --- should be focused on Mozilla2. I think that my compositor proposal should not be in 1.9.1, given my third constraint, which lets me focus on Mozilla2.

Doing 1.9.1 has additional advantages, like relieving pressure on 1.9, allowing us to tidy up after 1.9 in a timely manner, adding some dazzle to a 2008 Firefox front-end release, and gratifying contributors. There are of course disadvantages, like the cost of doing a release, and the potential to drain resources from Mozilla2. I trust in the iron fist of our Mozilla overlords to avoid the latter.

I enjoyed the MoCo social events, although I felt subdued and a bit withdrawn on Tuesday and Wednesday. It might have been my natural difficulty adjusting to large groups (unless immersed in a comforting bath of technical discussion).

I worked with David Baron during the week to explore down a cycle collector fault that I captured in a Chronicle trace weeks ago. It went much as I'd hoped: we'd explore with my tools for a little while, then we'd realize the UI needed a new feature, so that night I'd add it and we'd try again the next day. We eventually succeeded in tracking down the issue, so score one for Chronicle! I have prepared a open-source release of the UI (thanks to Mozilla); I will post again soon to announce it. I'm just waiting until I've made a good screencast of the UI in action.

Friday was a fun quiet day at the office. Chris Double and I met with Ian Hickson to discuss stuff. I also had a good chance to chat with Mozilla people in greater detail. Chris amazed all with his Silverlight-alike SVG-with-video demo --- as Vlad commented, "it works shockingly well!"

I flew out on Friday night and arrived home on Sunday morning. We've got new hires joining the Auckland office in next couple of months and I need to find new office space and organize a few other things. I will of course be mostly working on 1.9 blocker bugs, especially text bugs.

During the flights I managed to watch a bunch of movies, thanks to Air New Zealand's somewhat-annoying but mostly great video on demand system:

• Spider-Man 3: okay, more of the same.
  
• Hot Fuzz: Pretty good, a good mix of parody and actual plot.
  
• 300: ahistorical, prurient, philosophically revolting, but very memorable and occasionally intriguing --- worth watching if you can stomach it.
  
• Black Sheep: achieves its aims of being an unsophisticated splatter-fest full of Kiwi in-jokes --- so okay, but could have been a lot better.
  
• Amazing Grace: brilliant acting, moving, inspiring. Recommended.
  
• Next: only moments when it threatened to be more than rubbish. I like Nicholas Cage, but he seems to be in a rut of terrible vehicles.
  
• Shrek 3: okay, more of the same.
  

High And Low

Yesterday I visited Otago University and gave a guest lecture in a software deveopment class there, titled "Developing Firefox". This was a revised version of my "Firefox For CS Researchers" talk, with most of the "future Web developments" and "how this relates to CS research" stuff ripped out and replaced with greater discussion of our development tools, processes and issues. As it turned out I had way too much material for the time available so I also ended up leaving out the discussion of security issues, but that's OK for this version of the talk. I took some interesting questions and talked to some students after the talk for a while.

It's becoming clear that NZ universities steer students towards "safe languages and frameworks" used by IT and Web front/backends, and away from the sort of low-level systems programming required for infrastructure projects such as Gecko. That's understandable --- high-level is where the majority of jobs are, and I'm glad the industry has matured so people no longer write bespoke apps in C and C++ --- but it's also sad, because I think low-level stuff is more interesting in many ways, and tends to have more leverage in its ability to change the world. There is actually a great need worldwide for good low-level engineering and if we have a reservoir of talent here, jobs will flow here too. We don't have to just be another consumer of platforms produced elsewhere. Maybe one NZ university should focus on this while the others feed the IT shops.

(By 'low level' here I don't mean "assembler" or "C" --- I mean low in the software stack: infrastructure and frameworks that provide APIs and languages that other developers build on. For performance, footprint and other reasons this layer is more likely to be written in lower-level languages. I'm well aware of various efforts to write "systems code" in "high-level" languages, and I'm willing to discuss them with the proponents of those languages in comments :-).)

For our recruiting this means the interesting applicants are people who have either done some particular project that required systems programming (sometimes at university, e.g. a Masters thesis), or have just done some on the side for a hobby project. At Otago I exhorted students to consider contributing to suitable open source projects to get this kind of experience. I proposed that such experience opens many interesting career paths, some involving open source projects, but many others as well.

Tomorrow I'm visiting Waikato University to give both the "Developing Firefox" and "Firefox For CS Researchers" talks. It'll be interesting to see how things compare.

Flood Control Dam Number Three

Today the weather was mostly fine with occasional rain, but since Auckland is "who dares, wins" weather-wise we took off with some friends to the Upper Nihotupu Dam in the Waitakere ranges, about 45 minutes drive west of the CBD.

Our family visited the dam a couple of months ago, and at that time it was half-empty --- about ten metres of exposed bank between the high water mark and the waterline. I guess they've had plenty of rain in the Waitakeres recently because today the lake was absolutely full. In fact the spillway was operating, which was quite spectacular! (See below.)

It's about a 40 minute walk each way from the car park (on the Piha road just after it turns off Scenic Drive) to the dam, mostly along a service road. We got showered on lightly at one point but otherwise the weather was excellent, clear and sunny. Recent rain actually made the site particularly attractive --- apart from the operating spillway, the bush was most irriguous and streams and waterfalls were running high.