The Problem With Counting Browser Features

css3test.com is doing the rounds. I get almost exactly the same results in Firefox trunk and Chrome Nightly (64% vs 63%). This gives me a chance to explain why this sort of testing tends to be bad for the Web, without sounding whiny and bitter :-).

The root of the problem is very simple, and explained right at the top of the page:

Caution: This test checks which CSS3 features the browser recognizes, not whether they are implemented correctly.

Thanks for the disclaimer, but it doesn't eliminate the problem. The problem is that whenever someone counts the number of features supported by a browser and reports that as a nice easy-to-read score, without doing much testing of how well the features work, they encourage browser developers to increase their score by shipping some kind of support for each tested feature. Because we have limited resources, that effectively discourages fixing bugs in existing features and making sure that new features are thoroughly specced, implemented and tested. I think this is bad for Web developers and bad for the Web itself.

If Web authors reward Web browsers for superficial but broken support for features, that's what they'll get.

Instead, I would like to see broad and deep test suites that really test the functionality of features, and people comparing the results of those test suites across browsers. Microsoft does this, but of course they tend to only publish tests that IE passes. We need test suites with lots of tests from multiple vendors, and Web authors too. (Why haven't cross-browser test results for the official W3C CSS 2.1 test suite been widely published?)

I don't want to come across as harsh on css3test.com. The site is lovely, it has that disclaimer, and it goes a lot further than, say, html5test.com in terms of testing the depth of support for a feature. So it actually represents good progress :-).

Mozilla Tree Adventures

Last week we had a few overseas Mozillians stop by the office on their way back from linux.conf.au. We took the opportunity to hold our annual Mozilla Auckland outdoors event, which for only the second time ever did not involve volcanoes. Instead we went to Tree Adventures out in Woodhill Forest. Basically you strap on safety gear, climb ladders into the trees (up to 14 metres above the ground) and conquer a variety of obstacle courses on wires from tree to tree. Flying foxes are also involved. It's tons of fun, unless you have a fear of heights as I do, in which case it's tons of fun mixed with just as much sheer terror. I was glad to be able to overcome my instincts and keep going; that sort of mental discipline is worth practising. I just hope none of my colleagues noticed me whimpering.

We followed up with a nice lunch at Hallertau.

You Know You're In Australia When...

... you take a short walk after dinner and encounter a tribe of kangaroos.

MediaStreams Processing Demos

I'm at linux.conf.au at the moment (until Wednesday) and yesterday I attended the browsers miniconf. It went well, better than I expected. I had a slot to talk about the MediaStreams Processing API proposal to enable advanced audio effects (and much more!) in browsers, which has been my main project for the last several months (see my earlier post here. I worked frantically up to last minute to create demos of some of the most interesting features of the API, and get my implementation into a state where it can run the demos. By the grace of God I was successful :-). Even more graciously, the audio in the conference room worked and even played my stereo effects properly!

I have made available experimental Windows and Mac Firefox builds with most of the MediaStreams Processing API supported. (But the Mac builds are completely untested!) The demos are here. Please try them out! I hope people view the source, modify the demos and play with the API to see what can be done. Comments on the API should go to me or to the W3C Audio Working Group.

I must apologise for the uninspired visual design and extraordinarily naive audio processing algorithms. Audio professionals who view the source of my worker code will just laugh --- and hopefully be inspired to write better replacements :-). Making that easy for anyone to do is one of my goals.

Some of the things I like about this API:

• First-class support for JS-based processing. In particular, JS processing off the main thread, using Workers. This lets people build whatever effects they want and get reasonable performance. Soon we'll have something like Intel's River Trail in browsers and then JS users will be able to get incredible performance.
• Leverages MediaStreams. Ongoing work on WebRTC and elsewhere is introducing MediaStreams as an abstraction of real-time media, and linking them to sources and sinks to form a media graph. I don't think we need another real-time media graph in the Web platform.
• Allows processing of various media types. MediaStreams currently carry both audio and video tracks. At the moment the API only supports processing of the audio because we don't have graphics APIs available in Workers to enable effective video processing, but that will change. Applications will definitely want to process video in real time (e.g. QR code recognizer, motion detection and other "augmented reality" applications). Soon we'll want Kinect depth data and other kinds of real-time sensor data.
• First-class synchronization. Some sources and effects have unbounded latency. We want to make sure we maintain A/V sync in the face of latency or dynamic graph changes. This should be automatic so authors don't have to worry about it.
• Support for streams with different audio sample rates and channel configurations in the same graph. This is important for efficient processing when you have a mix of rates and some of them are low. (All inputs to a ProcessedMediaStream are automatically resampled to the same rate and number of chnanels to simplify effect implementations.)
• No explicit graph or context object. It's not needed.

Most of the features in the proposed spec are implemented. Notable limitations:

• "blockInput" and "blockOutput" are not implemented; there is no way for streams to opt out of being synchronized. For example it would be nice to be able to pipe a media resource into a processing node and if the resource pauses (e.g. due to a network delay), the processing node doesn't block but just treats the paused input as silence. This is probably the trickiest feature not yet implemented.
• No support for "live" streams. Similar to above, if a stream feeds into an output node that is blocked, we sometimes don't want to buffer the input stream. E.g. if the input is a live webcam you often (but not always) want to throw away buffered data so that when the output unblocks it immediately gets the latest video frames.
• There has been very little tuning to optimize throughput and latency, especially across a range of devices. This will be a lot of work.
• In general the API is very lightly tested. I'm sure there are lots of bugs.
• Video elements don't play in sync with streams captured from them. In my demos I worked around this by hiding the source video elements and creating new video elements to play the video via the stream. Fixing this bug would simplify the demos a bit.
• Canvas video sources are not implemented.
• The built in audio resampler is stupendously naive and needs to be replaced.
• Add support multiple audio and video tracks and the MediaStream track API.
• ProcessedMediaStreams using JS workers need to add checks ensuring that all upstream media sources are same-origin.
• The biggest limitation is that it's not shipping in Firefox yet. My giant patch is messy and a lot of cleanup needs to be done. I have a plan to split the patch up, clean up the pieces and land them piecemeal. In particular I need to get some of the infrastructure landed ASAP to help the WebRTC team make progress. (When we ship it, much or all of the API will probably be disabled by default, behind a hidden pref, until the standards situation is resolved.)

  Update Updated the build links to point to new builds with improved performance (faster JS execution in workers due to type inference being turned on; fewer control loop wakeups due to more intelligent buffering decisions for ProcessedMediaStreams).

"Cut The Rope" and HTML5 Audio

Microsoft released an HTML5 version of Cut The Rope which is pretty cool. Unfortunately they use Flash audio by default for Firefox users because, they say, "some Firefox users could have run into an audio problem but will notice we fall back to a flash plugin to ensure that sound effects and music will work." They don't mention specific Firefox bugs (although they do for Chrome), and when I try the HTML5 audio version it works fine for me. So, please try the HTML5 version in Firefox (release or nightly), and if it doesn't work let me know and file bugs! Thanks!

Risk Tolerance

Wise words:

John McGlashan principal Michael Corkery said: "He was doing what every kid should be doing, playing with his friends in a river."

Everybody had done everything right, but Dion still died.

"It was just a tragedy, and tragedies happen. Life sometimes deals out bad luck but we have to get on with it. We'll never forget him."

Too often, people respond to a child's tragic death by setting up a pressure group, foundation or new law to Make Sure This Never Happens Again.

A Case For Non-Fatal Assertions

Jeff Walden is doing awesome work on the foundations of Gecko to unify the lowest-level infrastructure used by JS and the rest of Gecko and to rebase it on modern C/C++ standards. But there is a controversy about whether non-fatal assertions should be part of that infrastructure. This issue came up before, not long ago.

I strongly believe that non-fatal assertions are valuable when used to report the presence of a bug that is not as severe as a browser crash. An example I just pulled out of nsBlockFrame::Reflow:

    ReflowBullet(state, metrics, lineTop);
    NS_ASSERTION(!BulletIsEmpty() || metrics.height == 0,
                 "empty bullet took up space");

If this assertion fails, then we have detected a Gecko bug which should be reported and eventually fixed. If the assertion failure is a regression, and we detect it in time, we will try very hard to fix it before the patch ships in a browser release. (If the regression is triggered by our layout reftest test suite, we will almost certainly detect it on checkin since reftests go orange on new non-fatal assertion failures.) However, if this assertion failure is the only thing that goes wrong, there will almost certainly be no ill-effects beyond the page having a slightly incorrect layout --- maybe. Some pages might trigger the assertion but appear to render correctly. Even if the bug causes a detectable test failure, the assertion helps to narrow down the cause and understand the code.

Using a fatal assertion here would have the same benefits but additional costs. A test run hitting the assertion would abort the suite, meaning we lose the results for the rest of the tests in the suite. This makes fixing test failures slower and more painful than necessary since more runs of the suite will often be required. (It's similar to a compiler always aborting after the first syntax error in a compilation unit.) Another cost is that if you are using a debug build for some reason and you hit this bug while trying to work on something else, your work will be unnecessarily blocked.

At this point some will say "Ah! But assertion failures should always just be fixed. Since non-fatal assertions are more ignorable, they encourage you to leave bugs unfixed."

That statement ignores the reality of bug priorities. An assertion failure is just a bug and needs to be prioritized along with other bugs. If our assertion infrastructure and associated project rules forced us to always prioritize some rare list bullet spacing bug above all bugs that don't trigger assertions, then that infrastructure would be actively damaging our project. We would have to respond by simply removing a lot of our assertions and losing their benefits. It is crucial to be able to ignore unimportant bugs.

Having said all that, fatal assertions certainly have their place. I imagine that in the JS engine almost any bug will lead to a crash sooner or later, so it makes sense for fatal assertions to prevail there because the downside is minimal; you were going to crash anyway, and crashing later might just be confusing.

Addendum: an objection to non-fatal assertions is that libc "assert" and some other assertion mechanisms are fatal, so to call something non-fatal an "assertion" is confusing. That may be true for some people, but it's cultural. The culture in Gecko is that NS_ASSERTION is non-fatal. Maybe renaming our non-fatal assertion mechanism would end up being a net win by some point in the future, but I'm dubious. Although I wouldn't actually mind too much; I just want non-fatal assertions, whatever we call them.