Anniversary Weekend

Today, Monday, is a public holiday in New Zealand --- "Anniversary Day". The weather during this long weekend has been mostly excellent: warm, sunny, fine except for some rain this morning. Our family made the most of it by spending a lot of time outside each day at various spots around Auckland...

On Saturday we had lunch at Achilles Point looking out over the ocean from St Heliers. Then we went for a walk around Churchill Park, a large open space with cattle on it. Along the western edge someone has constructed a very large private garden with ponds, ducks, and exotic plants. It abuts several houses so it must be some sort of community project. Quite mysterious, but very nice.

On Sunday after church we had lunch with some friends at Crazy Noodle in Newmarket. Wonderful milkshakes as usual. Then we went to the North Shore and ended up going for a walk through Northboro Park, then south along the bridge across the estuary. We saw eels, crabs and fish --- lots of fun. We want all the way past the Bayswater Cemetery to Plymouth Reserve before turning back.

Today we were a bit more organized and went with some friends to the Okura Bush Walkway, about half an hour's drive north of the city centre. This is not a loop track so we dropped a car in Stillwater at the north end and started the walk at the end of Haigh Access Road at the south end. It rained hard last night and we had some more showers early on, but that suited the rainforest feel of the Okura Bush section of the track. The kids enjoyed the mud too :-). After stopping by the Dacre Cottage for a picnic lunch we walked most of the the rest of the way along the beach. I went barefoot and just about managed to leave my shoes behind --- oops!

Altogether it was an enormously fun weekend, and also rather tiring --- in a good way. I'm looking forward to a restful day of Mozilla hacking tomorrow!

Over the last couple of years we've managed to do a lot of walks north, east and west of Auckland, as well as in the city area. I want to check out some areas to the south, especially the Hunua Ranges, 400 square kilometres of forest only an hour from Auckland but as far as I can remember I've never been there. There are just too many places to explore in the Auckland region!

ec06b3461cf0eaf3d3e4d7a2e429bddb

Today's blog post is very interesting, but unfortunately, publishing it now would cause serious harm. On the other hand, at some point in the future I would like to be able to prove that I wrote it today. So, let me just say that its MD5 hash is

ec06b3461cf0eaf3d3e4d7a2e429bddb

:-).

Sharing Sources Across Virtual Machines

As I mentioned in my last post, I would like to have a single set of source trees which I can edit, diff, patch etc in the host OS (Mac OS X) but from which I can easily make and test Mac, Linux or Windows builds. The obvious way to do it is to host the sources in OS X and mount them in the guest VMs using CIFS. Overnight I did some experiments to see how much this will slow down builds in a Linux guest VM, compared to hosting the sources in the Linux VM's own filesystem.

The results are in and they're pretty good! A from-scratch Firefox debug build with -j2 averaged 2647 seconds over four runs with sources in the native filesystem, 3308 seconds over four runs with sources in CIFS --- a 25% slowdown. I tested incremental builds using "touch layout/generic/nsBlockFrame.cpp" and then remaking layout; over four builds it averaged 27 seconds with sources in the native filesystem and 30 seconds with sources in CIFS, about 10% slowdown. (I discarded the first two runs in each configuration to ensure I was only testing warm-cache performance.) The object files in both cases were hosted in the Linux filesystem, of course.

This is definitely good enough to use the CIFS configuration in the Linux VM. Hopefully it will work just as well for Windows.

Babes In Toyland

I have a new Macbook Pro courtesy of the Mozilla Corporation. Somehow I've never grown out of the thrill of unpacking a shiny new machine, celebrating it as "the best yet!" (mostly thanks to Moore's Law), and then setting it up "just right".

Last time I bought a Mac, one of the first things I saw after turning it on was an Apple nag dialog asking me to pay $60 or so to upgrade to Quicktime Pro. Having just shelled out thousands of dollars for the "superior Apple experience", that dialog was most aggrieving. Fortunately that hasn't happened this time. Has Apple figured out what a turn-off that was?

I requested a Macbook mainly because via virtualization I will be able to run Mac, Windows and Linux all in the same machine. That will be very very useful. For too long I'm stumbled along doing all my work in Linux (and before that, all in Windows) and having to beg for help whenever I encounter bugs on other platforms. That was lame, and it's over.

In fact, I've already installed OpenSUSE 10.2 under Parallels. Parallels doesn't have good Linux guest integration so I'm stuck with crappy manual transfer of mouse control from Linux to MacOS. And there's something slightly weird going on with networking; almost everything works, except openssh can't resolve any DNS addresses properly. I wonder what openssh does differently from apps like Firefox. But otherwise it seems to work well, and fast, so I think this is going to be a winner.

One problem with virtualization is communication between the VMs. It's going to be annoying to copy patches or trees around between them. However I've managed to set up file sharing (CIFS shares actually) so that Linux can mount my Mac homedir directly. It would be interesting to try keeping all my trees in the Mac file system and building from them directly in Linux, putting the results in an objdir in the Linux file system. I suspect that the bazillion stat() calls performed by 'make' will be dog-slow over CIFS, making this untenable, but it'd be cool if it wasn't too bad. Then I could edit all my files in native Mac apps and just use Linux and Windows to build and test. And I also wouldn't have to juggle as many trees. I'll give it a go.

Update I "fixed" my openssh address resolution problem by setting "AddressFamily inet" in my ssh_config. I guess the problem was related to IPv6.

Fortress

I spent some time today reading the Fortress manual. Fortress is a new parallel programming language for scientific computing being developed as part of DARPA's high-performance computing initiative (think supercomputers simulating weapons). It's very interesting work, particularly so to me since I did some work on X10, IBM's language contribution to the same project. (Cray has one too, called Chapel). It has been a couple of years since I last looked at X10 and no doubt it's changed considerably during that time.

It's very impressive --- no surprise since Guy Steele is deeply involved. There are a several important ideas:

• Major syntax and Unicode work to make it possible to write code in mathematical notation
  
• Traits-based object system
  
• A units type system for scalars (e.g., "x := 10 kg/s")
  
• Very powerful generics, with static value parameters as well as type parameters, and very free usage of "where" constraints
  
• Explicit component programming mode
  
• Various kinds of built-in parallelism, in particular, "for" loop iterations run in parallel by default
  
• Inter-thread communication via "atomic" blocks, not locks, reduction operators, and the ability to use explicitly spawned threads as futures
  

I think some of these ideas work out better than others. I think the component programming features, where components are added, removed and upgraded in "fortresses", may prove to be a mistake. It can be very difficult to integrate that sort of thing into IDEs, source code control systems, development processes, build systems, etc.

The generics system has some rather unusual properties compared to similar systems for other "real" languages. For example you can write "trait C[T] extends C[E] where E extends T" --- i.e., declare that C is contravariant. But that's just the beginning, it seems you can create any network of constraints you like, with free type variables (e.g. E) not bound in the signature of the declaration. You can also write scary stuff like "object C extends D[T]", i.e., C has every method declared in D for all T, i.e., C has a possibly-infinite number of methods. It'll be very interesting to see what happens when they try to prove decidability of typechecking. It'll be even more interesting to see what kind of error messages the compiler produces.

I think the drive for ubiquitous implicit parallelism plus the fact that by default actions on shared data are not atomic is ... bold. It may work out OK for the scientific programming audience. For wider use I think we really need to restrict thread interleavings and eliminate implicit races. For example, Fortress's "for" loop can execute iterations in parallel and in any order, but for a general purpose language (like oh say Javascript), I would have the iterations appear to execute in any serial order --- i.e., run them in any order but make each iteration atomic.

Their approach to reductions seems really good. E.g. you can write (not real Fortress syntax):

x := 0;
for (i in 0..99) {
x += a[i];
}

They promise to notice that x can be computed efficiently via reduction --- i.e., that you can run the loop iterations in parallel and compute x in parallel too, using a tree-structured computation of logarithmic depth. This actually works by exploiting the type system, which declares via traits that addition over numbers is a monoid (or something). In general the standard libraries encode the rich mathematical structure of the fundamental types via the trait system; very cool.

The drive for mathematical syntax is an interesting experiment. I really wonder how it will work out. I think much depends on their ability to create a good IDE --- not just an editor, but also a debugger, not to mention pretty-printing compiler errors. And then we have to see whether people prefer writing programs that way. They argue that mathematical syntax has evolved over thousands of years, which is true enough, but it evolved under different constraints than we're dealing with here. It will make for awesome demos at least.

There's a nod towards exposing to the programmer the distribution of computation over nodes, via memory "regions" and explicit array distributions, but they don't take it as far as X10, where data location was part of the type system and all non-local references were statically exposed and explicit. It's hard to say right now what the right direction is.

Altogether lots of fun. It'll be great to watch how things play out.

Status

I've been working for the Mozilla Corporation for nearly a week now. I'm working from home because I'm still working on finding office space. Unfortunately I'm stuck on my home computer for now, a 2-year old iMac with OS X 10.3.9, which is absymally slow. Worse still, my Firefox builds don't actually work --- they never show the main browser window. I've been trying to find out if there's any way to build a working Firefox on 10.3.9, but no-one seems to know. Upgrading to 10.4 would probably help, but it costs 260 NZD, which makes me want to exhaust my 10.3.9 options before I splash out for a .1 upgrade that I mostly won't use --- it builds very slowly anyway and I hope to get a shiny new machine soon.

So I've spent my time coding up a really big patch. We have a nexus of problems that tie into a number of important bugs, including:

• Bug 298889: Transparent windows break when they contain IFRAMEs
  
• Bug 343430: Fixed-position elements flicker when scrolling
  
• Smooth scrolling can be annoyingly slow
  
• Bug 316992: onscroll events aren't dispatched in a timely manner
  
• Bug 130078: Chrome UI overlaid on content windows (e.g., the phishing UI) doesn't work
  
• Scrolling and IFRAMEs inside SVG foreignobject being very slow or completely broken
  
• The mechanisms for suppressing and batching visual updates that we currently use are somewhat broken; they tend to suppress update for an entire hierarchy of documents even when we only intend to affect one document
  
• Other flickering problems in certain kinds of scrolled content
  
• Unable to draw some kinds of HTML content over windowed plugins
  

Furthermore, soon we'll want to introduce browser-driven animation (e.g., SMIL). That demands infrastructure for refreshing windows at a fixed rate, coordinated with other kinds of window updates.

A sketch of a solution to all these problems, and more, is written up in the wiki, and I'm taking a stab at implementing it. The basic idea is that there are three kinds of window updates: repainting window contents, scrolling, and changes to the geometry of windowed plugins. We batch requests for those changes together and periodically execute them all at once in a unified "window update". By carefully structuring the code in the window update, we can minimize or eliminate the delay between the different kinds of operations to ameliorate flicker. At the start of the window update we can also ask animations to update their state to the current time.

To make this work properly we have to eliminate the use of child widgets for display other than for windowed plugins. But actually we don't need to fully remove those widgets, which is hard for many reasons; we can give all child widgets size (0,0) and they won't interfere with rendering. The widget hierarchy will still be around so we won't break accessibility or other things that depend on it. (Although eventually those widgets should be removed.)

One nice thing about working at home is that yesterday, when I was thinking about some of these problems, I was able to take a short walk to the top of Mt Eden and stare at the ocean for a while, free from the distractions of my computer. Very nice, very productive.