Perambulations

In previous days I walked down College Hill or through the back streets of St Mary's Bay and then through Victoria Park to the Viaduct Basin where I work. This morning I had some extra time and decided to try Westhaven Marina and the harbour foreshore. In the still, early morning, it was sublime. Words can hardly do it justice. Small things caught my attention: kayakers near a small island, roiling shallows where predators chased bait fish, shags diving, the reflection of clouds in the water. All quiet, all cool, all bathed in early light. Thanks to you, o Lord.

In fact the waterside route appears to be as fast as the inland routes (about 30 minutes), perhaps because it has almost no traffic lights. I'm looking forward to walking it regularly.

Sharing binaries across multiple Mozilla trees with ccache

As I explained earlier, I want to have multiple Mozilla source tree checkouts each with a different set of patches applied, and build Mozilla in each of these trees. Usually the patches are small so most of the build products are the same in each tree. Doing regular builds in each tree is therefore wasted work and disk space.

ccache can partially overcome this problem by caching the results of compiler invocations so that when a compilation is performed that is "identical" to a previous invocation, the resulting object file will simply be fetched from the cache. So when you use ccache with multiple trees, compiling the same source file in different trees will do the compile in the first tree to be built, and then the other trees will simply get that object file from the cache. The theory's good but there are a few problems.

The first problem is that ccache will only reuse an object file when the preprocessed source texts are identical. The preprocessed text includes the paths of all #include files in the compilation unit. Mozilla's build process usually specifies a number of #include files using absolute paths, which of course will differ per tree, causing ccache to treat the compilations as different. Most of the problems are eliminated if you specify --srcdir=.. in your .mozconfig, so that all sources are accessed relative to the build directory. The remaining problem is that currently the NSPR library is accessed via an absolute path; a patch to fix this is in bug 275790. With these changes, ccache works across trees.

Another problem is deeper. The most efficient way to use ccache is to have it hardlink the cache file to the output, so that there's only one file on disk that's used everywhere (instead of having to make separate copies in each tree). Unfortunately, as the ccache documentation explains, hardlinking doesn't work well with multiple trees.

An example will illustrate. Suppose we have a source file F.c, two trees 1 and 2, each of which builds F.c to make F1.o and F2.o respectively. Suppose that we then link F1.o and F2.o to make final binaries B1 and B2. Suppose we modify F.c at 8am, then build tree 1 at 9am to make F1.o and B1. Then suppose we build tree 2 at 10am. Essentially, ccache makes F2.o by creating a hardlink to F1.o. To stop "make" from getting confused, it must update the last-modified-time on F2.o to be later than when F.c was modified. It does this by stamping it with the current time, 10am. But because this is the same file as F1.o, F1.o's last-mod-time is now also 10am. If you go back to tree 1 and run "make" again, it will unnecessarily relink F1.o to make B1 because it looks like F1.o has changed, even though it hasn't. This is documented as a limitation of ccache with multiple trees, and it would be a fairly serious problem with Mozilla trees because unnecessary linking can take a long time.

But I've found a way to resolve this. In the example, note that it's not actually necessary to update F2.o's timestamp because F1.o is already older than F.c, which F2.o depends on. Put it another way, make will be happy as long as ccache ensures the object file's last-mod-time is newer than the last-mod-time of all the files the object depends on. So I've hacked gmake to take a new option --last-dep-mtime; when this option is set, whenever gmake rebuilds a target, it sets the MAKE_LAST_DEP_MTIME environment variable to the timestamp of the most recently modified dependency (in seconds since the epoch). I've also hacked ccache to check for this environment variable, and when it's present, just ensure that the object file is newer than that time. In most of my cases it won't need to touch the object file at all.

I will try to push these patches upstream to the gmake and ccache maintainers; they're very small code changes.

Asiana

Auckland has turned into a really cosmopolitan city, especially with the noticeable influx of Asians into the central business district, which apparently is at least partly due to growth in English language schools for fee-paying foreign students. For a Sinophile like me, this is a great thing. I've been back a week now and I've already had a chance to sample several sources of Asian food. I'm really happy that my office is close to the heart of downtown and hence all these options...

Cafe Savoy: Hong Kong style cafe on Albert St, one of those weird "Chinese-style Western food" places. Ever wonder what the Chinese version of General Tso's Chicken is? It's here. Consider, for example, "cheese, creamed corn and shredded chicken on spaghetti", or "potatoes and chicken in a creamy curry sauce on rice, baked". Really, it's lovely.

Food Asia: This is a food court actually. They poisoned my mother the first time I went there but I love it. Lots of cheap classic (and off the wall) dishes from a variety of little stalls. So far I've enjoyed the juicy dumplings and lamb tikka, and I love the preserved plum smoothie.

Food Alley: A slightly cheaper and downmarket version of Food Asia, if such a thing seems possible. Good chicken laksa and great frozen Malaysian ice dessert (with condensed milk, lychees and longans).

Sunshine: I'm looking into an abyss here: a yumcha restaurant (dimsum for you Yanks) directly downstairs from my office, of sublime quality. The basics are superb --- the ha gow (shrimp dumplings) slightly crisp, the desserts melt-in-your-mouth, the chatsu bau are tastier and less sweet. But they also have items I simply haven't seen in New York. Manhattan's excellent Jing Fong had just in its latest incarnation introduced custard dumplings in sticky translucent wrappers (what is that substance?), but Sunshine has those, but also the same thing with red bean filling. Then there were little smoked fishes, jelly rolls, almond cubes, and more ... The flaky pastry pork pies were as good as they should be, and amusingly designed like little Chinese sausage rolls. Best of all the service is relentless. Our four-man team was overwhelmed by a barrage of carts and capitulated after just twenty minutes. I have to repeat, this is all after 1pm on a weekday, and it's directly below my office. May God have mercy upon me.

Continental Noodle House: An old Singaporean/Malaysian favourite of mine back in the early 90s, I'm glad to see it's still operating. Janet and the kids and I had a good meal there but they seem to have cut out the interesting stuff like the peanut pancakes. On the bright side they've cleaned the carpet in the last decade (something I don't think was true back in 1993).

Empress Garden: This is not actually in town, it's a few blocks down the road from where we're staying in Herne Bay. Very nice Peking duck though, as always. Our cold appetizers in sesame sauce were great too.

I haven't even been to Grand Harbour yet...

Forestry

Over the years I've been hacking on Mozilla, I've wasted a lot of time grappling with the limitations of CVS. There are a few main issues for me:

• Frequently I need to collect all the changes I've made to a tree into a patch file, by diffing my tree against the trunk. CVS contacts the server and takes a long time.
  
• CVS updates are slow.
  
• Managing many trees is painful, especially creating a tree with an initial checkout (very slow), building it, and keeping the trees up to date. This means that I tend to use only a few trees, usually just one. This is bad because various pieces of work get mixed up in a tree, and my patches or even checkins sometimes contain fragments from logically separate changes. Ugh.
  

Since Mozilla is my full time job now, I'm investing some energy in tackling these problems. I looked at various "CVS replacement" programs, especially the version control systems that claim to interoperate with CVS and provide local branching. But for various reasons they're not suitable; the main reason is that none of them have been proven on codebases as large as Mozilla.

So I've decided to use rsync to maintain a mirror of the entire Mozilla CVS repository on my local machine. Checking out and updating many trees from it is very fast. Likewise, doing CVS diffs is now a local and very fast operation. But how costly is it to maintain a synchronized copy of the entire repository, given that it records all the changes ever made to 5 million lines of code over six years? Surprisingly, it's not bad at all. It's using about 2.6GB of disk space. It took about 30 minutes to pull down the first time, using rsync with compression. My office Internet link is pretty good, but I am in New Zealand. It takes about 20 seconds to resynchronize on a day when there aren't many checkins. Altogether I'm extremely pleased. My only fear is that a lot of people will start doing it and mozilla.org will have to restrict rsync access...

Another thing I'm doing is using ccache to speed up builds and share build products across trees. Since in most trees only a few files are modified, building in each tree will mostly produce the same object files. ccache does a good job of recognizing when the same file is being compiled in different places and just copying the object file from its cache into the output directory. In fact you can do better just by hardlinking the file from the cache to the output, so you only have one copy of the output file shared by all your trees. There is one problem with ccache currently that I believe I have fixed... I'll write about it soon, once I've verified that my fix works.

Branded

Things have been a bit stressful as the members of our family have taken turns hosting a virus and as we go through the processes of settling down. However, things are great. The weather has been fantastic --- especially by New York standards --- and it's incredible to be able to walk out the door, two blocks past sweet-smelling jacaranda trees and you're at the water's edge. Not to mention Erin Pt Park just a little further away, and downtown Auckland is sensational too, So much to talk about!

One odd thing I've just noticed is how deeply ingrained is my memory of New Zealand brands. At the supermarket I feel misty seeing the likes of Craig's strawberry jam, Tip Top ice cream, and even Diamond spaghetti. It's ridiculous --- and frightening --- just how much of my mental real estate they must have been squatting on all these years.

Well, I'm Back

Arrived 6:15am Wednesday morning, New Zealand time.

I took great pleasure in filling out the immigration form, under "New Zealand Resident", ticking "Returning ... permanently". The official asked "Have you been away long?" "Ten years." "Welcome back." "It's good to be back."

The Gloomocracy

One thing that always rankles me about New Zealand is the way New Zealanders emphasise the negative side of everything. Sometimes it's because complaining will push someone's selfish agenda. Most of the time I think it's just a national habit of whining, and reporting on whining. It's intensely annoying. It's also actually harmful to the country and its people.

I just saw a particularly good example. The Heritage Foundation and the Wall Street Journal did their 11th annual ranking of countries' "economic freedom". Roughly they're measuring how much government intervention there is in the economy. (It's a conservative think tank, so more is necessarily bad.) New Zealand was ranked fifth most economically free in the world, behind only Hong Kong, Singapore, Luxembourg and Estonia, and hence ahead of countries such as the United States, Australia, Japan, etc etc. So what was the TV headline for this impressive performance?

NZ SLIPS IN ECONOMIC FREEDOM

Because last year we were third in the world. Sigh.

Counting Down

It's late on Sunday, January 2. The rest of my family leaves on Wednesday, less than three days from now, and I leave on the following Monday. There are still a number of loose ends to take care of, but I feel that we're on track and ready to go. Praise the Lord...

Over the last few weeks I've felt a mixture of excitement and sadness. Even though overall I'm really happy about our move, that doesn't eliminate the disappointment and pain of leaving behind friends, places and work that I love dearly. (Perhaps I should feel the same way about death.) We've just today had a wonderful bittersweet farewell party with some Christian friends, which drives home the point. But the sadness is rooted in God's blessings to me over my ten-year American sojourn; as C.S. Lewis put it, "the pain now is part of the happiness then".

I've only gradually come to terms with full impact of this move; I've felt too distracted by kids and busywork to think deeply about it. Today we removed some large pieces of furniture from the house, and I think that more than anything gives me the feeling that we ARE going. Honestly, I can't survive long like this!