Rediscovering Auckland

Over the last month or so our lives have started to settle down and we've had time --- and great weather --- to get out, visit friends and relatives, and see Auckland again after ten years away.

Strangely, I feel a much stronger desire to really know this city than I did before. For example, there are dozens of volcanic cones in the city, most of which have been turned into parks, but I've only been to a handful of them. So yesterday we went to Mount St John for the first time.

It's quite thrilling to walk through this upper-class suburb in the middle of everything, walk up a strip of reserve between houses and pop out above an amazing natural amphitheatre with amazing views over the city. I wonder how I managed to pass close by hundreds of times --- my old high school is perhaps a mile away --- being only barely aware it was there. Perhaps it's because Mount St John has larger and more famous neighbours --- Mount Hobson, Mount Eden and One Tree Hill.

Here's another shot of the view from the top. Auckland aficionados may wish to identify the four volcanic cones visible in this picture.

This Easter long weekend we also went to the Royal Easter Show, walked around One Tree Hill and Albert Park, and had lunch with friends at Grand Harbour and T-Mark. T-Mark is a rather obscure Taiwanese cafe in Newmarket --- very good. I'm exhausted and looking forward to a relaxing day at work tomorrow!

Gecko 1.8 For Web Developers: Collapsing Margins And The 'Clear' Property

Every Gecko release we spend lots of time and energy fixing bugs to make us more standards compliant and, where necessary, more Web-compliant. One area where I did some significant work last year was fixing how we handle content that combines floats, collapsing margins, and the 'clear' property.

This was particularly important because apparently IE has a bug where a block that contains a float automatically has its height extended to include the bottom of the float. In fact, the float children of a block should not normally have any direct effect on the height of the block. But what if you do want a block to extend at least as far as the bottom of its floats, either to match this IE bug, or for your own reasons? CSS 2.1 provides a way. You simply write something like this:

<div>
<div style="float:left">...</div>
<div style="clear:both"></div>
</div>

The element with 'clear' is forced below the float, and because that element is in the normal flow, it forces its container to be at least that high. The only problem is that this usually doesn't work in Gecko 1.7/FF 1.0. Basically we were treating the space induced by 'clear' as margin space and then collapsing it away. Fixing this required a major overhaul of how clearance and margin collapsing were handled, so that right now I believe we're more standards-compliant in this area than any other browser.

I could go into lots of gory details --- this is an egregiously complex area of the CSS spec --- but I won't. Suffice it to say that the above trick for vertically sizing blocks does work in Gecko 1.8.

Gecko 1.8 For Web Developers: Columns

One of the big things I've been working on for Gecko 1.8 is support for multicolumn layout. Our implementation follows the CSS3 draft pretty closely. Right now we support the properties -moz-column-count, -moz-column-width, -moz-column-gap and (soon) -moz-column-rule.

Basically this lets you set up multiple columns like a newspaper, so that content flows from the end of one column to the top of the next column automatically. The big win is that you can use the full width of a large screen without making lines excessively long. This very blog is using columns on most entries --- check it out with a Firefox trunk build.

The draft specifies 'balanced' columns. This means that we automatically find a minimal height for a set of columns that balances the content as evenly as possible across the columns while fitting into the available width of the page. This is very powerful and you have to see it in action to appreciate what it does for you. However, balanced columns are inherently a little slow. Also, sometimes you want the columns to be a fixed height but extend horizontally as far as necessary to fit the content. We support that by extending the draft slightly; if the 'height' property on the column set is 'auto', we balance the columns and horizontal overflow doesn't happen, otherwise we do not balance the columns and horizontal overflow can happen.

Our implementation has a few limitations that cannot be fixed for 1.8:

• Tables don't break across column boundaries.
  
• Absolutely positioned elements don't break across column boundaries, even if the absolutes' container does.
  
• We don't always find the minimum height for a balanced column set, for example when the column set contains blocks with margin, borders or padding things can go wrong.
  
• We don't support the CSS properties for page break control, so you can get widows and orphans (a paragraph breaking just after its first line or just before its last line). And you can't prevent breaks from happening without using hacks.
  
• There is no way to make content in one column span multiple columns. In the future we may allow content in a column to flow around overflowing floats in previous columns, but we're not going to get that in this release.
  

Of course there are still some bugs but it's already in much better shape than it was a few weeks ago. In particular some huge issues involving floats breaking across columns have been fixed. Since we use the same code for page layout as for columns, this also means a lot of problems with printing (and print previewing) pages with floats have been fixed.

I'm really interested in having Web developers play with this feature, in particular to shake out any critical bugs. Some tips for using columns:

• Watch out for content that horizontally overflows the column it's in. The next column will draw right over it.
  
• Thus, it might be a good idea to put a DIV inside a columns element and give it a background color, and put the rest of your content inside that.
  
• Column balancing can be slow. Unbalanced columns give faster layout.
  
• People have always thought of elements as being rectangular, thus we have Javascript properties like clientX, clientY, clientWidth, clientHeight etc. That was never true for inlines, and now it's not even true for blocks. Those properties will just give you the first rectangle of many rectangles that might make up the block. Beware.
  

Please file Bugzilla bugs when you find we're doing something unexpected or not per spec. If you have other feedback you can email me personally or just blog about it and let me know.

The Great U-Turn

The Herald published Newsweek's annual story about Easter. As usual it tries to please every side ... strange considering what happened on Easter is really the most divisive issue in the universe. But it does raise one of the great questions: how did the disciples recover from Jesus' death --- the apparent crushing of all their beliefs and hopes --- to become potent advocates of his resurrection and take the world by storm? The only credible answer, in my view, is that it really happened.

Gloomocracy III

The Herald published my letter to the editor, which I wrote before the weekend's articles spotlighted in Gloomocracy II. Here's the edited text.

Having just returned from 10 years in Pittsburgh and New York, I find that the Kiwi love of whinging has made people lose touch with reality. One correspondent complains of "constant rain, wind and cold" when in the past two months all but a few days have been gloriously sunny and warm. Other correspondents wail about soaring house prices, perhaps without realising that the large volume of returning expats is a major factor driving those prices.

From my contacts with returnees and many friends still overseas who hope to return, I'm confident that the influx will continue. Meanwhile, for all those who whine that life is much better in Australia, let them move there. Everyone will be much happier.

It reads a bit harsher than I'd intended. Oh well. Other letters expressed similar sentiments. There were also many letters like these:

---

If New Zealand wants to retain its best and brightest and return to the top of the OECD I suggest that we abandon our draconian student loans system and spend more money on basic infrastructure. If the costs of education (including post-secondary), healthcare and public transport were all decreased one suspects that the "brain-drain" would not be as severe.

---

Why would a self-respecting productive person regress to a welfare state that squeezes the fruits of his labour to redistribute to the herd? Wasn't that the reason most expats left in the first place?

---

I think this illustrates how people use NZ-bashing to push their (often contradictory) agendas as solutions without which NEW ZEALAND IS DOOMED.

On a slight tangent, for some reason the Herald chose to highlight this astonishing letter:

We have the highest interest rates in the developed world and this has given us one of the most overvalued currencies in the Western world.

On the other hand our standard of living has fallen from second in the world to the bottom quarter of the OECD countries.

When will our politicians and economists begin to understand cause and effect?

An interesting question from someone who apparently has no grasp of it whatsoever (assuming the Herald didn't edit out a mass of justifying exposition).

The wool boom of the 1950s temporarily launched NZ to the second top position in some international economic table. Ever since, the Gloomocrats have used it as an excuse to set expectations sky-high --- a benchmark against which all subsequent economic achievements are deemed failure.

Gloomocracy II: Gloomocrats United

Before I launch into my main rant, check out this classic Gloomocrat front page headline from the New Zealand Herald yesterday:

STUDY WARNS OF GRIM TOLL IF BIRD FLU HITS NZ

Let's be clear: no-one has detected any sign of the Asian bird flu in New Zealand. Furthermore, NZ has about the best biosecurity of any country in the world, aided by the fact that it's a thousand miles to the nearest land mass. So why is this the most important news of the day? Beats me. Next week, STUDY WARNS OF GRIM TOLL IF VOLCANIC EXPLOSION DESTROYS AUCKLAND.

Now on to the main topic. New Zealand has record low unemployment, the lowest in the Western world, and has a shortage of skilled workers. The Government announced they plan to recruit NZ expatriates to come home to help fill the gap. Naturally, this provoked a storm of derision from NZers who cannot see why any expatriates would want to come back.

Some of the letters to the editor were genuinely absurd. One writer complained of "constant rain, wind and cold", apparently unaware that over the last two months all but a few days have been pleasantly warm and sunny (but not too hot or humid, mind you). Others complain about soaring house prices without realizing that it's actually returning expats who are contributing to those prices. And no-one bothered to point out that large numbers of expats have indeed been coming back and continue to do so. Every single person in the Novell Auckland office is a returned expat or recent immigrant. And while in the late 90s most of my friends from university were overseas, now most of those have already returned.

I wrote a letter to the editor pointing all this out, but I expect it's not negative enough for publication.

The Herald did a clever thing and attached an online question form to the online version of the article about the government's plan, to canvas expats for their reactions to the proposal. Here are the questions:

1. How long have you been overseas?
   
2. Do you intend to return to NZ definitely soon/definitely
   eventually/possibly/never? Why?
   
3. If you're in the "possibly" category, what are the key factors you
   will consider in deciding whether to return?
   
4. How important are NZ pay rates as a factor in your decision?
   
5. How do your current income and living costs compare with your
   potential income and living costs in NZ?
   
6. Would a 5 per cent general pay increase in NZ this year, and perhaps further increases in future years, make a difference to your decision?
   

Note that the questions about specifics are focused on financial issues, which are definitely not New Zealand's strongest suit. That's a nice Gloomocratic slant.

Here are the related articles that appeared today:

• Quarter of NZ's brightest are gone
  
• Expats have wary eye on home...and don't like all they see
  
• Kiwi, please come home
  

Pretty much what you'd expect given the slant --- mostly money complaints. Of course no-one mentions the horrendous cost of living in New York or London, or the limited recreational options in London, Hong Kong or Singapore, or the terrible weather in all of those places. In each case the best aspect of some other place is pitted against the corresponding aspect of New Zealand, and all other issues are ignored.

The first article "Quarter of NZ's brightest are gone" is especially deceptive. How does the number of NZ grads living overseas compare to historical numbers --- is it rising or falling? How many grads eventually come back vs non-grads? How many of the 77% of expats in Australia are from that graduate population? (I think these numbers may conflate two large groups: graduates who leave NZ and go all over the world and are quite likely to come back, and unskilled people who emigrate to Australia.) And of course the blunt phrasing of the headline suggests that none come back (and the whole article ignores the talented immigrants coming into this country).

And check out this quote from "Kiwi, please come home":

But the size of the wage gap makes most expatriates scoff. Only 21 per cent of those who answered a question about it said the 5 per cent pay rise would affect their decision to return, 6 per cent said "maybe" and 72 per cent said "no".

Collins implies that 72 per cent are "scoffing" at the wage gap, but in fact those "no" votes include every expat for whom NZ wage rates are not a problem (either because they're high enough or because the expat would be financially independent --- in my experience there's lots of both). Collins either made a slip or is deliberately misrepresenting the facts.

I would be interested in seeing a real study like this: sample X% of surviving graduates who graduated in each year 1970-2000 and answer the following questions:

• How many are currently in New Zealand?
  
• For those in New Zealand, what is the distribution of the # of years they have been out of New Zealand?
  
• For those who left New Zealand and came back, why did they come back?
  
• How many returned to New Zealand have subsequently left again, and why?
  

Seeing how these numbers shift across graduation years would be especially interesting.

[Yes, I know this blog entry is a meta-whine. Sorry.]

Slough of Despond

Today may end up being remembered as the sad day when software patents became inevitable in the EU. I feel quite upset about it.

If the US regime holds sway, the rational path to financial success in the software industry must change dramatically. It no longer makes sense to take great ideas and create, sell and support software products. Instead, at modest expense and low risk you can obtain software patents covering your ideas in all their variations. Then just sit back and wait. When someone manages to create and market a successful product that "infringes" one of your patents, wait for them to get established, then move in and demand your cut. Their options are limited: countersuit is impossible, since you are careful not to have any products yourself. Reengineering and rereleasing their software to avoid the patent is likely to be very expensive and difficult, and in any case you can claim damages on everything they've shipped since you made your claim against them. If they have shallow pockets and you ask for little, they'll probably settle. If they choose to fight, law firms or investors will back you if your patents are strong, and eventually you've got a good chance of winning the jackpot. Once you have a war chest, you'll do best suing large profitable companies who are foolish enough to integrate your "invention" into some high-volume product.

So why isn't everyone already doing this in the USA? A lot of people are doing it, and more all the time (see Intellectual Ventures). But to date, most of the smart people who are really good at coming up with new software "inventions" have been busy working in companies that manufacture actual products and are therefore vulnerable to countersuits. The advantages of the pure parasite company are only gradually being realized.

Is it really that easy to come up with new, strong patentable ideas that will cover future products? Sure. In fact, that's the job of academic and industrial researchers in computer science, and I did it for many years. But in fact those researchers are hamstrung by the need to hone their ideas into workable implementations, run experiments, publish reproducible and sound results, discard ideas that don't work, do diligent comparison with related work, communicate it all to their peers, and shepherd their ideas into production systems. The parasite model does away with all that; come up with an idea in the morning, and you can have it in the lawyers' hands by the afternoon. I estimate a good researcher should be able to improve patent production tenfold by shifting to the new model. Certainly not all the patents obtained will prove valuable or valid, but many will.

If this is true, why are the big companies still pushing for extension of the software patent regime? I believe it's institutional inertia in most cases. For example, within IBM many people are sympathetic to the facts as I have presented them here. But the structure of the organization, the reward systems, and the managment outlook are all completely oriented towards "more patents is better". I suspect major trauma will be required to turn that around.

Why is software specially harmed? I think the sheer ease of creation of new "inventions", and the very large number of novel "inventions" required to implement even a modest-sized software product, distinguish software from other fields of endeavour and make it particularly vulnerable to parasites. But I don't deny that patents harm other industries.

Isn't it just because software patent regimes have been badly implemented, but they're OK in principle? Perhaps, but then I think the burden of proof lies with patent proponents to demonstrate a well-implemented regime in the US (or anywhere else) before expanding it to the rest of the world.

Take away this mantra: products are bad, because they make you vulnerable. Only those who produce nothing will prosper.

The other thing to take away is that anyone who talks about their "defensive patent portfolio" is hopelessly out of touch. There is no longer any such thing.

Alright, I do seriously think that's the way we're going. But at this point, when all I see is frustration and anger, Jesus gives me a kick.

Then Jesus said to his disciples: �Therefore I tell you, do not worry about your life, what you will eat; or about your body, what you will wear. Life is more than food, and the body more than clothes. Consider the ravens: They do not sow or reap, they have no storeroom or barn; yet God feeds them. And how much more valuable you are than birds! Who of you by worrying can add a single hour to his life? Since you cannot do this very little thing, why do you worry about the rest?

Tomorrow I'll do my job and let God worry about the big stuff.

Heroes

Mike Connor got me thinking about unsung heroes of Mozilla development. I'm thinking of people who put in long hours for little pay, who toil away endlessly on grungy code, who endure the endless abuse and demands of the community, who patiently fix up regressions and help newcomers, who do work that no-one notices until things go wrong but without which we'd be nowhere, who have more talent than anybody but will never be seen on the WIRED cover or have their own startup, who have been working on the same unsexy code year after year but keep at it because they know that if they don't do the job, nobody will.

To me, no-one exemplifies this dedication more than Boris Zbarsky. My hat's off to you, Boris, and may you one day receive your due.

Visual Regression Tests

Over the last couple of weeks I've been working on automated visual regression testing for Gecko. The idea is to run Gecko through a battery of HTML test files and render each one to an image file. We can then compare the images produced by different Gecko versions; if a new version has different rendering then we've probably either fixed a bug or introduced a new bug (or both!). When layout engineers develop patches, we can see which testcases get changed and hopefully catch any unexpected regressions before checkin. (It's amazing we haven't had this years ago; we've survived until now because we have lots of great volunteers who download nightly builds, test them, and report bugs.)

We already have regression tests based on analyzing the frame tree geometry computed by the layout engine. Visual regression tests have some advantages over comparing those coordinates:

• Visual tests eliminate false positives, because changes in the frame tree that don't affect rendering will not be detected.
  
• Visual tests eliminate false negatives, because they detect changes in rendering caused by z-ordering, image decoding, graphics layer changes, clipping, and so on, not just layout.
  
• Visual test results are easier to understand because we can show visually the difference between two runs. Anyone with HTML+CSS knowledge can run visual tests and analyze the output.
  
• These visual regression tests can run using an optimized build, but our frame tree tests require a debug build, so the visual tests will run 2-3 times faster. Also you'll be able to use standard Mozilla nightly builds to get a baseline, if you want to.
  

I initially tried to do this using Xvfb to create a virtual screen and taking screenshots of it (both using ImageMagick 'import' and reading the Xvfb virtual screen directly through its exported framebuffer). I had horrible problems with synchronization, trying to ensure that the Xvfb server had actually rendered all the commands Gecko sent to it before I took the screenshots, and eventually gave up on that approach late last week.

Instead I just added a real offscreen rendering API to the view manager and added code to nsDocumentViewer to call it when the environment variable MOZ_FORCE_PAINT_AFTER_ONLOAD is set. With this patch, if you set MOZ_FORCE_PAINT_AFTER_ONLOAD=/tmp/foo, then every time we load a document Gecko spits out a message "GECKO: PAINT FORCED AFTER ONLOAD [url] [file] ([status])", where [url] is the loaded URL, [file] is of the form /tmp/foo-NNN which names a file in PPM format, and [status] is OK if we wrote a file or otherwise some barely-descriptive error token. This code should be useful for various hacks, and hopefully we'll be able to make the view manager API visible to Javascript authors for use in the browser UI and extensions.

That solved a lot of problems but I had another set of problems to grapple with this week: nondeterminism. Basically the regression test approach depends on the fact that every time Gecko renders a page in a window of fixed width and height, we get exactly the same image, down to the details of each pixel. But with trunk code we don't and I just spent a few days figuring out exactly why. It boils down to the following issues:

• Animated images :-). These have to be turned off in the profile, using

  user_pref("image.animation.mode", "none");

  
  
• I use Gtk2+Xft builds, and the Freetype2 autohinter seems nondeterministic; it sometimes produces different glyphs for the same character in the same font. Turning off autohinting and antialiasing via ~/.fonts.conf seems to fix the problem.
  
• A bug in painting table collapsing-borders where the BCData::mCornerBevel bit is uninitialized, making Gecko occasionally think that a border should be bevelled when it shouldn't and throwing the painting out of alignment. (I also found that border-collapse painting can vary depending on the damage rect involved, but that's not affecting me right now.)
  
• A bug in nsImageGTK::UpdateCachedImage where it copies image pixels into the image pixmap using gdk_draw_rgb_image_dithalign and specifies a "dither alignment" of rect->x, rect->y. It turns out that GTK already adds alignment to rect->x, rect->y so that we end up with the coordinates double-counted and incorrect alignment. This means when an image gets filled in in one shot we get one dithering, but when the image is filled in in multiple passes (e.g. data loads slowly from the network), the later sections get dithered slightly differently to the one-shot case. So on <24-bit displays, you get slight variations in colouration in some rendered images (not visible to the naked eye), but only when the network is slow.
  
• Freetype2 has a bug where if you ask it to resize a 13x13 face to a 12.5x12.5 face, it decides that they're close enough already and does nothing, even though creating faces from scratch for 13x13 and 12.5x12.5 would give you slightly different font metrics. Because Xft caches faces internally, this means when you're working with fractional font sizes Xft will sometimes give you varying font metrics depending on whether it's reusing a face in the cache (and hitting the Freetype2 bug when it tries to resize the face) or creating one from scratch. This in turn occasionally causes line heights, and sometimes the intrinsic width of text inputs which is computed from font maxAdvance, to be off by one pixel.
  

Dealing with nondeterministic program behaviour is always troublesome and tracking these down was really hellish. I had to instrument the code involved with logging functions, run my 1300 test cases multiple times until I saw two runs with different behaviours, compare the logs to narrow down the cause of the variation, and repeat until I had located the problem. And 1300 testcases with a lot of logging enabled is very unwieldy to analyze.

Anyway after fixing all these issues I now have 100% reproducible rendering for the 1300 testcases in the Mozilla tree. Next week I'll try the 2500 testcases in various online test suites. Hopefully they won't uncover any new issues. Then I'll get this work submitted in Bugzilla and get back to fixing bugs with the help of my shiny new regression test engine.

Currently my script takes about 130 seconds to run 1300 testcases on my 3GHz Xeon dedicated test machine. I have included support for distributing tests across multiple processors and I'm looking forward to seeing how many tests per second I get on my big machine.