Eyes Above The Waves

Robert O'Callahan. Christian. Repatriate Kiwi. Hacker.

Friday 11 March 2011

Investigating Performance Differences Between Firefox 4 And IE9

As I explained in my last post, Microsoft's PR about "full hardware acceleration" is a myth. But it's true that some graphics benchmarks consistently report better scores for IE9 than for Firefox, so over the last few days I've been looking into that. Below I'll explain the details what I've found about various commonly-cited benchmarks, but the summary is that the performance differences are explained by relatively small bugs in Firefox, bugs in IE9, and bugs in the benchmarks, not due to any major architectural issues in Firefox (as Microsoft would have you believe).

FishIE Tank This is one of many 2D canvas demos that focus on drawing images with various transforms. Firefox 4 RC spends a lot of its time doing security checks for the drawImage call, plus there are some smaller inefficiencies in our drawImage code. I added a simple cache to avoid security checks, and fixed other inefficiencies, and the resulting builds are now slightly faster than IE9 RC1 on my system.(Although since both browser get 60fps for 1000 fish, I had to modify the test to use 2000 fish to see any differences.)

Speed Reading This is also drawImage-heavy but exercises a path that wasn't well optimized in our D2D backend. The above builds include a tweak for that path, and are slightly faster than IE9 RC1 on my system on this test.

GUIMark 2 Flash and HTML5 Some reports show differences between Firefox and IE9 on those benchmarks, but on my laptop their performance is very similar, except for HTML5 Text where Firefox is a bit faster.

Mr Potato Gun, Preschool and HWACCEL These benefit hugely from the improvements in the builds I linked to above, but our score is still lower than IE9's. That's because these tests animate using setTimeout with a delay of zero, and count how may times the animation callback fires per second. In Firefox, delays of zero are clamped to a minimum of 10ms, but in IE9 they're clamped to a minimum of 4ms. In these benchmarks, the actual painting is incredibly fast in Firefox and IE9, so the limiting factor is this timeout delay, and IE9 gets 2.5x of our score. This difference is artificial and does not affect what users see on the screen, because it's impossible to display more than about 60 frames per second under any normal circumstances --- I have previously discussed this on my blog. In these tests the user will just see 60 frames per second in IE9 and Firefox. (HWACCEL actually clamps its reported FPS value to "60+", which is sensible, but some sites are reporting numbers higher than 60 anyway, which is strange; they must have modified the test or be using an old version of it.)
In any case, post-FF4 we will change our timeout clamp to 4ms because that's what the spec says now.

But wait, there's more! My testing shows that in the above demos IE9 consistently clamps to less than 4ms --- about 3.2ms. In my test, if you click on the "run with draw" button you get 1000 callbacks in about 3200ms. If you click on the "run without draw" button you get 1000 callbacks in about 4000ms as expected. The only difference is that "run with draw" draws to a canvas in each iteration, and "run without draw" doesn't! It's definitely actual drawing to a canvas that triggers the reduced clamp; if instead of drawing you do some canvas operations that don't trigger drawing (e.g. a save()/restore() pair), you get the 4ms clamp. This is clearly a bug per spec, and I've reported it. (You won't be able to read the bug report without a Microsoft Connect account.) It's certainly both very strange and very convenient for Microsoft's test scores!

Santa's Workshop Microsoft had a whole blog post about how Firefox's slow performance on this test was due to us not using "full hardware acceleration". But they guessed wrong --- hey, we're open source, they could have checked :-). My profiling shows that we spend less than 8% of the time painting on my laptop; the rest of the time is tied up in script execution and DOM manipulation. In particular our implementation of setAttribute and getAttribute for SVG transform attributes is pretty bad. We have a plan for fixing that. (The test is also quite stupid; it animates transforms using lots of string munging, instead of using the faster and more convenient SVG DOM API.) Anyway, it's nothing to do with hardware acceleration.

PS, please don't bother posting all your test results in comments unless they're particularly interesting. Ta :-).



Comments

Omega X
What's the status on relevant bugs landing? Fx 4.0.1?
Richard Ayotte
This is great, thanks for the explanation. What about the Linux results, any thoughts on that?
JK
Thanks Robert for doing this work. You are the turn-to guy for cutting through the benchmark bullshit. I hope you stay with FireFox and the good guys for a loooong time!
Justin Dolske
Tangentially related, but since you mention FishIE it's worth plugging Jeff's blog post again... http://muizelaar.blogspot.com/2011/02/drawing-sprites-canvas-2d-vs-webgl.html
Devon Govett
Thank you! This is what I like to see: improvements based on competition rather than just bashing. It's great that you took the time to get to the bottom of these issues and even fix some of them! Gold star. :-)
Matt
Too bad these fixes couldn't be included in time for FF4.
I've heard that Mozilla intends to speed up releases afterwards. Could these be included in FF5 (or 4.1?) in just a few months? Or will it be closer to the end of the year before they come out?
Nicholas
Thanks for the follow-up post. Re-reading my original comment I think it was a little snarky, I didn't mean it that way. Glad you could spell out the differences-- I can't wait to see what the future holds for all of the major browser vendors.
aa
Some of the test are a bit unrealistic, but faster drawImage and other optimizations are really nice! =).
geeknik
IE9 is faster than Firefox 4. I hate to disagree, but it is. Maybe that will change with Firefox 4.1 or 5.0. =) The benchmarks show it's faster and it just feels faster with scrolling, etc.
Test system: Windows 7 Pro x86 SP1, i7-920 2.8ghz, 6GB DDR3, 2x ATI HD 5770s in Crossfire. (Latest drivers, etc)
Latest nightly of 4.0b13pre:
FishIE: 18FPS (1000 Fish)
SpeedReading: 43FPS
GUIMark HTML5: 12FPS
HWACCEL: 60+FPS
Preschool: 60FPS 10ms draw time (NO AUDIO)
MrPotatoGun: 60FPS (NO AUDIO)
IE9 RC:
FishIE: 38FPS (1000 Fish)
SpeedReading: 60FPS
GUIMark HTML: 14FPS
HWACCEL: 60+FPS
Preschool: 60FPS 0ms draw time (WITH AUDIO)
MrPotatoGun: 60FPS (WITH AUDIO)
Sid
BTW, from my experience there, Microsoft's generally a little paranoid about employees reading copyleft FOSS code.
Anonymous
Robert quick question...
I've been following the whole hardware acceleration debate since IE9 first appeared in beta. After recently installing Firefox 4 RC I still find it hard to understand which parts of the product are HWAcceling SVG?
Can you give the lowdown on what's not quite firing on all four in Firefox with SVG?
Canvas/JS in comparison is at least as fast in Firefox as it is in ie9.
Is there still work to be done with SVG in firefox?
Martin
simon
that preschool thing actually crashed ff4 completely, making it give a black screen (tab preview still worked, but everything else was black)
is this just my machine?
Thinus
Are you kidding me? Are you people that stupid and ignorant. Microsoft is back and in the next week they are going to turn on their FUD machine big time and you know what, people are going to believe them.
Seeing is believing, they know that if they spew untruths, the web will hold them accountable. They will also use their demos and their performance in those demos to drive their marketing point home. In their world, IE9 is the best out there and in every message they will prove it. Worst still, you can run independently those test and you will agree with them.
I am simply amazed that you have not picked up on these things earlier and get the updates into Firefox 4. IE9 message is standards, every other browser will have to do those demos just as well...
Simply put, when Firefox beta 1 came out, it was pretty cool, all these shining new things. By the time Firefox 4 ship, most of that features would be present in other browsers and Firefox appear to be slower to boot!
If Mozilla does not get more aggressive with performance and more polished features with every update, I am sorry to say, you might as well give up your market share right now.
Updates containing only security fixes is no longer good enough. You want to fight Microsoft and Google, you better need to step up. Holding big improvements till the next version is no longer going to cut it.
Martijn
I must be doing something terribly wrong, but i'm getting extremely bad results on my macbook pro (core i7 2.66 Ghz, 4GB ram, NVIDIA GeForce GT 330M).
For example when i do the HWACCEL test I get no more than 4 fps.
When I go to about:support i get the following output:
Adapter Description 0x22600,0x20400
Vendor ID0000
Device ID0000
Adapter RAM
Adapter Drivers
Driver Version
Driver Date
Direct2D Enabled false
DirectWrite Enabled false
WebGL Renderer NVIDIA Corporation -- NVIDIA GeForce GT 330M OpenGL Engine -- 2.1 NVIDIA-1.6.26
GPU Accelerated Windows1/1 OpenGL
Is this expected behaviour Robert?
Rob
This is all very pertinent, as i'm trying to work out the reasons for performance difference between FF3 and FF4 at the moment on a project. FF4 is performing way, way slower than FF4 (which is for us performing worst of all browsers). Robert, could i share a link with you privately for you to look at? We'd be very interested in Mozilla's take on what we may be doing wrong...
Rob.
default
What about improving hardware acceleration on the other platforms? i.e. OS X and Unix/Linux.
Derek
Great post. I really appreciate the analysis.
Sascha
i can't help myself but there still a huge bug in the OSX version.... the right click menu for organising the bookmark does not appear in the folders of the bookmark bar.... under windows no prob... help !
Alfred Kayser
I think the 10ms to 4ms fix should be put into FF4.0 as soon as possible. IE9 will be released real soon as well as FF4. FF4 should not trail behind IE9 on this. With this simple fix, IE9 won't have much of an speed advantage over FF4.
Alfred Kayser
Are there any bugs number on these fixes?
Tim
Any insight into why the new demos meant to show off graphics in FF4 run much MUCH slower on my system than either the demo videos, or Chrome 10 on the same box? In particular: https://mozillademos.org/demos/shadows/demo.html the letter movement is EXTREMELY choppy on FF4 (RC1) vs Chrome. No extensions are enabled, just stock RC1. About:support output is as follows (running WinXP SP3):
Adapter Description: Mobile Intel(R) 4 Series Express Chipset Family
Vendor ID: 8086
Device ID: 2a42
Adapter RAM: Unknown
Adapter Drivers: igxprd32
Driver Version: 6.14.10.5303
Driver Date: 9-21-2010
Direct2D Enabled: false
DirectWrite Enabled: false (0.0.0.0, font cache n/a)
WebGL Renderer: Google Inc. -- ANGLE -- OpenGL ES 2.0 (ANGLE 0.0.0.541)
GPU Accelerated Windows: 1/1 Direct3D 9
Note, I'm seeing the same behavior on my home desktop system. On that one, it's a RADEON 5700HD with the very latest drivers. FF4 RC1 VERY choppy (nothing like the Youtube video shown on the demo page), Chrome 10, perfectly fine. On that one Direct 2D and DirectWrite are both enabled as well, and the Direct3D is 10 (running Win7 SP1).
Should a performance bug be opened?
...........
Speedreading scores & average draw duration:
Chrome 11 Canary: 19sec, 18ms
IE 9 RC: 9sec, 10ms
FF 4 RC: 586sec, 29ms !!!!!!!
Nearly 10 minutes.
gordon
I don't really care about benchmarks. IE9 feels much faster than Firefox 4 in my day to day browsing, and the UI is way more responsive. That, I care about.
Wolfgang
Rob, thanks for the link to our site. We do actually quite a bit of benchmark testing and run as many benchmarks as our time allows - and include benchmarks from independent sources, Mozilla, Microsoft and Google. It is difficult to pinpoint the performance with no standard benchmark and our best option is to take an approach of an average result across a wide array of benchmarks (our standard run, which will be be updates as soon as you publish FF4, consists of 248 different tests (JS, HWAcceleration, offline page loading, Flash). The last time we did the test, FF4B10 was slightly ahead of IE9, but way behind Chrome, on average. It appears to us that plain code processing in offline page loading is much faster in Chrome than in any other browser.
Gary
To really appreciate the speed increase ROC is talking about try his build of Fx4:
http://ftp.mozilla.org/pub/mozilla.org/firefox/tryserver-builds/rocallahan@mozilla.com-16644bf2dbd2/
Then change 'dom.min_timeout_value' in about:config to 1.
Boris
geeknik, I don't understand what you're disagreeing with, exactly. Did you _read_ Robert's post? Say the second sentence of it?
Martin, SVG performance involves a lot more than drawing. There's also the manipulation of the actual SVG objects. This last is fairly slow in Gecko; we are working on fixing that. Robert did say so in his discussion of Santa's Workshop.
Thinus, stuffing more stuff into Firefox 4 won't get it out there faster. It just needs to ship. Now. All the fixes Robert talks about here will be shipping within three months after that.
Martijn, that's expected behavior: there is no 2d hardware acceleration on Mac yet. So you're getting image compositing in canvas happening on the CPU.
Rob, please report a bug on whatever issues you're seeing? Cc ":bz" and I'd be happy to take a look at things.
default, it's being worked on!
Alfred, dropping the timer to 4ms is just too high-risk at this point. And even if we did it wouldn't work right on Windows without even riskier code changes. We should have done that back in the summer, yes. But there were other things on our plates then. Again, this is something that we will be shipping 3 months from now!
Tim, your problem is the "Direct2D Enabled: false" bit. That means no 2d graphics acceleration, so canvas is not accelerated. For the system where Direct2d is enabled, if you're seeing problems like that please do file a bug!
Style Thing
Yep, this build really improves the results.
But WHY oh why you did not include patches used for that build into the main tree? so anyone can see that Firefox beats IE in it's own tests :)
James Gentile
What about say Maze Solver, FF 4 is much slower than IE9 RC, the page says that's mostly a CSS test. In addition, FF UI slows down while running this test.
IE9 is also faster at HTML5 Soduku, about 50% faster. The page says that's a javascript stressing test.
IE9 is faster at Map Zooming test as well, though the there is no FPS or score, it just looks faster on my machine (i7 965, gtx 580).
IE9 is about 100% faster than FF 4 at Psychedelic browsing test.
Additionally, IE has been sand boxed for about 4 years now, while still no sign of sand boxing from FF. What happened to the million eyes/coders thing? Aren't you guys supposed to be FASTER, not 4 years SLOWER?
Chris
While I don't have FF installed on this computer, I decided to test IE9 RC1 and Opera 11.01 on a Quad Core 2.66Ghz, 8GB DDR2, ATI HD 3450, computer.
Results:
IE RC1: 13 seconds
Opera : 2392 seconds (39 minutes)
another_sam
awesome, awesome findings and post. can't wait for these patches to land. thank you!!
keldorn
So why aren't this fixes already landed on the trunk (before it was closed for RC)? You obviously know why Firefox is slow, what parts are badly implemented and have fixes for some of them, so why is Mozilla purposely not fixing the performance problems? IE9 tests have been around for quite some time and this is poor timing on Mozilla's part to make excuses on why Firefox is slower then IE9 in them.
Chris
This article is pathetic.
IE9 is now the fastest browser because of a massive refactoring that addressed not only the graphic rendering pipeline but also the JS engine and overall parser. Just face it.
You can spend time trying to convince people that FF4 does this or that better. You are just part of this FUD machine you're complaining about.
Now, just focus on your stuff and make FF4 better.
End of line.
Chris
This article is pathetic.
IE9 is now the fastest browser because of a massive refactoring that addressed not only the graphic rendering pipeline but also the JS engine and overall parser. Just face it.
You can spend time trying to convince people that FF4 does this or that better. You are just part of this FUD machine you're complaining about.
Now, just focus on your stuff and make FF4 better.
End of line.
Dessimat0r
I'm getting 10578ms with draw on your test, 10604ms without draw. This is running your build, so something's clearly wrong? I'm not sure. The other tests are super-fast as expected compared to IE9.
Zeir
Your latest builds have already "expired" on the tryserver and have been moved to the 'old' subdir.
Also, look into the "maze solver" test on the IE site. It says it tests CSS layout performance.
IE9 RC gets 17 seconds, Firefox 116, Chrome 12 133, Opera 11.10 (2039) embarrasingly beats all with just 9.1 seconds without any kind of hardware accel. On 30x30.
The funny thing is, on your build, if I ctrl-tab away from it immediately, it speeds up, getting 19 seconds. You have to refresh the page to make it solve a 'new' one I think, but tabbing away makes it use much less CPU as well, for some reason. (The same thing does not appear to benefit Chrome any.)
If Mozilla can get basic rendering and speed issues sorted out (drawing/rendering times are essentially more important for the web at large than javascript speed, below a certain threshold), and get IPC working, then I'd be relatively happy, I think.
Firefox's "old" javascript bug with the UI makes things extremely unresponsive if any webpages start to hog for any reason, which gets worse the larger number of tabs you have open (eventually almost anything will start hogging the CPU just because you have a lot of tabs).
Even the "lowly" IE9 is going to start drawing power users away, if FF continues to be wholly unusable (for basic performance reasons) with a large number of tabs, and where a single hog can essentially deadlock the browser.
(Unrelated security note, there seems to be no way to simply block 'mixed content' from loading on secure pages? The warning doesn't stop it.)
Meh
Meh, I use Opera 11 anway, well away from this pissing contest and faster than FF4 and IE9 in the real world.
firedfox
Instead of bragging about useless synthetic benchmarks, why Mozilla doesn't concentrate its efforts first on the basics such as proper font rendering?
Look at the issues reported here:
https://bugzilla.mozilla.org/show_bug.cgi?id=635490
https://bugzilla.mozilla.org/show_bug.cgi?id=626293
I really think Firefox jeopardize its future by ignoring such fundamental bugs.
Boris
Style Thing and keldorn: roc wrote these patches last week, after we were already frozen for RC. They're not in Firefox 4 because they're not 100% risk-free and it's very late in the endgame. So that was not an acceptable time to land the patches; they will be landing as soon as we reopen for development after shipping Firefox 4.
James Gentile, we have bugs on some of those tests, yes. Some of those are JS performance issues. Maze solver is about half painting and half layout (on Mac; on Windows the painting may well be faster with d2d). The maze testcase is mostly a stress-test for handling lots of floats; I filed https://bugzilla.mozilla.org/show_bug.cgi?id=641340 on that.
Zeir, it makes sense that if you don't focus that tab while running the maze it goes faster: since it's not showing we don't have to do any painting or layout on those intermediate stages that you can't see anyway. So it obviously goes much faster!
Sid
Firefox 4 is definitely cool but still it hogs a lot of memory. Whatever happened to the cool guys a Mozilla.
I write for http://hacktivist.tumblr.com/
It is a lot about browsers. Especially Firefox.
Anonymous
More on the timout clamp : it's quite easy to see on http://ie.microsoft.com/testdrive/HTML5/PowerMeter/Default.html
Just move the mouse and see the timer drop to 3.2 (and sometimes 2.0, I don't know why nor when exactly) ...
Funny thing is they (MS) drop the limit from 15ms in PP6 to 4ms in Final (but thay may limit to 15ms on battery, didn't test) ...
Jon
Thanks for the clarification. I tried your build, and after updating my graphics driver I did indeed see majorly improved performance, roughly on a par with IE9 in the tests you listed.
However, although you may have disproved Microsoft's FUD about GPU usage, the fact remains that, regardless of the actual reason you ARE slower without these improvements, and since FF4 will ship without them it will be less performant than IE9, which is a real shame.
It seems MS and Mozilla have both done a similar job of enabling GPU usage, but MS have been more aggressive about optimising other parts of their renderer to really squeeze out performance. Maybe it's a false impression, and they've just been careful about how they code their demos, but if not then it seems like Mozilla could benefit from more focus on optimisation.
Gary McAllister
"relatively small bugs in Firefox, bugs in IE9, and bugs in the benchmarks"
You only really elaborate on your bugs but don't go into any details about IE.
Is this just speculation? Also, not sure if it is deliberate (or subconscious) but your language seems to portray that IE has bigger (Cockroach vs Tarantula) bugs than you.
Anyway, keep going, I really enjoy these posts.
Boris
Jon, they've most likely just coded the demos, then optimized to them. This is pretty clear in a lot of their SVG demos, which use the same 2 or 3 things over and over again, and those are the things IE9 makes fast, while ignoring other parts of SVG.
Nothing wrong with that, but the only way to match that in general is to optimize for exactly the edge cases their demos use. We'll end up doing that, of course, but the value of that is unclear in general.
Robert O'Callahan
I can't go into details about IE bugs because I don't have IE source code.
Tim
Boris, just an FYI that, as requested, I've opened the following bug about the Letter-Heads demo still being very choppy even with DirectWrite/2D enabled using a Radeon 5700HD:
https://bugzilla.mozilla.org/show_bug.cgi?id=641758
Thanks.
Martin
Very interesting reading!
Tom
Hi, I'm an average user and must say that those miliseconds don't do anything to me, but prefer Firefox for all it's addons and customization, my looks nothing like original. And this is a BIG difference comparing to IE. Also I never had a crash and every page opened I wanted. All that for free and for anyone to see (the source code).
Ben Curly
Well, SpeedReading test doesn't work for me on Firefox 4 RC at all. The page just doesn't load. I see the blue background and that's it, the letters never come up and I can't run the test ;/
I'm on Windows 7 64-bit.
Nathaniel Tucker
This is really cool. I was wondering why there were differences in some of these benchmarks.
I was wondering, will your fixes for some of these problems be included in a 'stability' release of Firefox 4, or Firefox 5?
Javier
@Robert
You can't give us details about IE bugs because you don't have IE code. Then, how do you know there are these bugs?
Robert O'Callahan
Because I can see IE9 behaving incorrectly. You can run the testcase yourself in IE9 and observe that it doesn't follow the spec.