Eyes Above The Waves

As I explained in my last post, Microsoft's PR about "full hardware acceleration" is a myth. But it's true that some graphics benchmarks consistently report better scores for IE9 than for Firefox, so over the last few days I've been looking into that. Below I'll explain the details what I've found about various commonly-cited benchmarks, but the summary is that the performance differences are explained by relatively small bugs in Firefox, bugs in IE9, and bugs in the benchmarks, not due to any major architectural issues in Firefox (as Microsoft would have you believe).

FishIE Tank This is one of many 2D canvas demos that focus on drawing images with various transforms. Firefox 4 RC spends a lot of its time doing security checks for the drawImage call, plus there are some smaller inefficiencies in our drawImage code. I added a simple cache to avoid security checks, and fixed other inefficiencies, and the resulting builds are now slightly faster than IE9 RC1 on my system.(Although since both browser get 60fps for 1000 fish, I had to modify the test to use 2000 fish to see any differences.)

Speed Reading This is also drawImage-heavy but exercises a path that wasn't well optimized in our D2D backend. The above builds include a tweak for that path, and are slightly faster than IE9 RC1 on my system on this test.

GUIMark 2 Flash and HTML5 Some reports show differences between Firefox and IE9 on those benchmarks, but on my laptop their performance is very similar, except for HTML5 Text where Firefox is a bit faster.

Mr Potato Gun, Preschool and HWACCEL These benefit hugely from the improvements in the builds I linked to above, but our score is still lower than IE9's. That's because these tests animate using setTimeout with a delay of zero, and count how may times the animation callback fires per second. In Firefox, delays of zero are clamped to a minimum of 10ms, but in IE9 they're clamped to a minimum of 4ms. In these benchmarks, the actual painting is incredibly fast in Firefox and IE9, so the limiting factor is this timeout delay, and IE9 gets 2.5x of our score. This difference is artificial and does not affect what users see on the screen, because it's impossible to display more than about 60 frames per second under any normal circumstances --- I have previously discussed this on my blog. In these tests the user will just see 60 frames per second in IE9 and Firefox. (HWACCEL actually clamps its reported FPS value to "60+", which is sensible, but some sites are reporting numbers higher than 60 anyway, which is strange; they must have modified the test or be using an old version of it.)
In any case, post-FF4 we will change our timeout clamp to 4ms because that's what the spec says now.

But wait, there's more! My testing shows that in the above demos IE9 consistently clamps to less than 4ms --- about 3.2ms. In my test, if you click on the "run with draw" button you get 1000 callbacks in about 3200ms. If you click on the "run without draw" button you get 1000 callbacks in about 4000ms as expected. The only difference is that "run with draw" draws to a canvas in each iteration, and "run without draw" doesn't! It's definitely actual drawing to a canvas that triggers the reduced clamp; if instead of drawing you do some canvas operations that don't trigger drawing (e.g. a save()/restore() pair), you get the 4ms clamp. This is clearly a bug per spec, and I've reported it. (You won't be able to read the bug report without a Microsoft Connect account.) It's certainly both very strange and very convenient for Microsoft's test scores!

Santa's Workshop Microsoft had a whole blog post about how Firefox's slow performance on this test was due to us not using "full hardware acceleration". But they guessed wrong --- hey, we're open source, they could have checked :-). My profiling shows that we spend less than 8% of the time painting on my laptop; the rest of the time is tied up in script execution and DOM manipulation. In particular our implementation of setAttribute and getAttribute for SVG transform attributes is pretty bad. We have a plan for fixing that. (The test is also quite stupid; it animates transforms using lots of string munging, instead of using the faster and more convenient SVG DOM API.) Anyway, it's nothing to do with hardware acceleration.

PS, please don't bother posting all your test results in comments unless they're particularly interesting. Ta :-).