Friday, 11 March 2011

Investigating Performance Differences Between Firefox 4 And IE9

As I explained in my last post, Microsoft's PR about "full hardware acceleration" is a myth. But it's true that some graphics benchmarks consistently report better scores for IE9 than for Firefox, so over the last few days I've been looking into that. Below I'll explain the details what I've found about various commonly-cited benchmarks, but the summary is that the performance differences are explained by relatively small bugs in Firefox, bugs in IE9, and bugs in the benchmarks, not due to any major architectural issues in Firefox (as Microsoft would have you believe).

FishIE Tank This is one of many 2D canvas demos that focus on drawing images with various transforms. Firefox 4 RC spends a lot of its time doing security checks for the drawImage call, plus there are some smaller inefficiencies in our drawImage code. I added a simple cache to avoid security checks, and fixed other inefficiencies, and the resulting builds are now slightly faster than IE9 RC1 on my system.(Although since both browser get 60fps for 1000 fish, I had to modify the test to use 2000 fish to see any differences.)

Speed Reading This is also drawImage-heavy but exercises a path that wasn't well optimized in our D2D backend. The above builds include a tweak for that path, and are slightly faster than IE9 RC1 on my system on this test.

GUIMark 2 Flash and HTML5 Some reports show differences between Firefox and IE9 on those benchmarks, but on my laptop their performance is very similar, except for HTML5 Text where Firefox is a bit faster.

Mr Potato Gun, Preschool and HWACCEL These benefit hugely from the improvements in the builds I linked to above, but our score is still lower than IE9's. That's because these tests animate using setTimeout with a delay of zero, and count how may times the animation callback fires per second. In Firefox, delays of zero are clamped to a minimum of 10ms, but in IE9 they're clamped to a minimum of 4ms. In these benchmarks, the actual painting is incredibly fast in Firefox and IE9, so the limiting factor is this timeout delay, and IE9 gets 2.5x of our score. This difference is artificial and does not affect what users see on the screen, because it's impossible to display more than about 60 frames per second under any normal circumstances --- I have previously discussed this on my blog. In these tests the user will just see 60 frames per second in IE9 and Firefox. (HWACCEL actually clamps its reported FPS value to "60+", which is sensible, but some sites are reporting numbers higher than 60 anyway, which is strange; they must have modified the test or be using an old version of it.)
In any case, post-FF4 we will change our timeout clamp to 4ms because that's what the spec says now.

But wait, there's more! My testing shows that in the above demos IE9 consistently clamps to less than 4ms --- about 3.2ms. In my test, if you click on the "run with draw" button you get 1000 callbacks in about 3200ms. If you click on the "run without draw" button you get 1000 callbacks in about 4000ms as expected. The only difference is that "run with draw" draws to a canvas in each iteration, and "run without draw" doesn't! It's definitely actual drawing to a canvas that triggers the reduced clamp; if instead of drawing you do some canvas operations that don't trigger drawing (e.g. a save()/restore() pair), you get the 4ms clamp. This is clearly a bug per spec, and I've reported it. (You won't be able to read the bug report without a Microsoft Connect account.) It's certainly both very strange and very convenient for Microsoft's test scores!

Santa's Workshop Microsoft had a whole blog post about how Firefox's slow performance on this test was due to us not using "full hardware acceleration". But they guessed wrong --- hey, we're open source, they could have checked :-). My profiling shows that we spend less than 8% of the time painting on my laptop; the rest of the time is tied up in script execution and DOM manipulation. In particular our implementation of setAttribute and getAttribute for SVG transform attributes is pretty bad. We have a plan for fixing that. (The test is also quite stupid; it animates transforms using lots of string munging, instead of using the faster and more convenient SVG DOM API.) Anyway, it's nothing to do with hardware acceleration.

PS, please don't bother posting all your test results in comments unless they're particularly interesting. Ta :-).



51 comments:

  1. What's the status on relevant bugs landing? Fx 4.0.1?

    ReplyDelete
  2. This is great, thanks for the explanation. What about the Linux results, any thoughts on that?

    ReplyDelete
  3. Thanks Robert for doing this work. You are the turn-to guy for cutting through the benchmark bullshit. I hope you stay with FireFox and the good guys for a loooong time!

    ReplyDelete
  4. Tangentially related, but since you mention FishIE it's worth plugging Jeff's blog post again... http://muizelaar.blogspot.com/2011/02/drawing-sprites-canvas-2d-vs-webgl.html

    ReplyDelete
  5. Thank you! This is what I like to see: improvements based on competition rather than just bashing. It's great that you took the time to get to the bottom of these issues and even fix some of them! Gold star. :-)

    ReplyDelete
  6. Too bad these fixes couldn't be included in time for FF4.
    I've heard that Mozilla intends to speed up releases afterwards. Could these be included in FF5 (or 4.1?) in just a few months? Or will it be closer to the end of the year before they come out?

    ReplyDelete
  7. Thanks for the follow-up post. Re-reading my original comment I think it was a little snarky, I didn't mean it that way. Glad you could spell out the differences-- I can't wait to see what the future holds for all of the major browser vendors.

    ReplyDelete
  8. Some of the test are a bit unrealistic, but faster drawImage and other optimizations are really nice! =).

    ReplyDelete
  9. IE9 is faster than Firefox 4. I hate to disagree, but it is. Maybe that will change with Firefox 4.1 or 5.0. =) The benchmarks show it's faster and it just feels faster with scrolling, etc.
    Test system: Windows 7 Pro x86 SP1, i7-920 2.8ghz, 6GB DDR3, 2x ATI HD 5770s in Crossfire. (Latest drivers, etc)
    Latest nightly of 4.0b13pre:
    FishIE: 18FPS (1000 Fish)
    SpeedReading: 43FPS
    GUIMark HTML5: 12FPS
    HWACCEL: 60+FPS
    Preschool: 60FPS 10ms draw time (NO AUDIO)
    MrPotatoGun: 60FPS (NO AUDIO)
    IE9 RC:
    FishIE: 38FPS (1000 Fish)
    SpeedReading: 60FPS
    GUIMark HTML: 14FPS
    HWACCEL: 60+FPS
    Preschool: 60FPS 0ms draw time (WITH AUDIO)
    MrPotatoGun: 60FPS (WITH AUDIO)

    ReplyDelete
  10. BTW, from my experience there, Microsoft's generally a little paranoid about employees reading copyleft FOSS code.

    ReplyDelete
  11. Robert quick question...
    I've been following the whole hardware acceleration debate since IE9 first appeared in beta. After recently installing Firefox 4 RC I still find it hard to understand which parts of the product are HWAcceling SVG?
    Can you give the lowdown on what's not quite firing on all four in Firefox with SVG?
    Canvas/JS in comparison is at least as fast in Firefox as it is in ie9.
    Is there still work to be done with SVG in firefox?
    Martin

    ReplyDelete
  12. that preschool thing actually crashed ff4 completely, making it give a black screen (tab preview still worked, but everything else was black)
    is this just my machine?

    ReplyDelete
  13. Are you kidding me? Are you people that stupid and ignorant. Microsoft is back and in the next week they are going to turn on their FUD machine big time and you know what, people are going to believe them.
    Seeing is believing, they know that if they spew untruths, the web will hold them accountable. They will also use their demos and their performance in those demos to drive their marketing point home. In their world, IE9 is the best out there and in every message they will prove it. Worst still, you can run independently those test and you will agree with them.
    I am simply amazed that you have not picked up on these things earlier and get the updates into Firefox 4. IE9 message is standards, every other browser will have to do those demos just as well...
    Simply put, when Firefox beta 1 came out, it was pretty cool, all these shining new things. By the time Firefox 4 ship, most of that features would be present in other browsers and Firefox appear to be slower to boot!
    If Mozilla does not get more aggressive with performance and more polished features with every update, I am sorry to say, you might as well give up your market share right now.
    Updates containing only security fixes is no longer good enough. You want to fight Microsoft and Google, you better need to step up. Holding big improvements till the next version is no longer going to cut it.

    ReplyDelete
  14. I must be doing something terribly wrong, but i'm getting extremely bad results on my macbook pro (core i7 2.66 Ghz, 4GB ram, NVIDIA GeForce GT 330M).
    For example when i do the HWACCEL test I get no more than 4 fps.
    When I go to about:support i get the following output:
    Adapter Description 0x22600,0x20400
    Vendor ID0000
    Device ID0000
    Adapter RAM
    Adapter Drivers
    Driver Version
    Driver Date
    Direct2D Enabled false
    DirectWrite Enabled false
    WebGL Renderer NVIDIA Corporation -- NVIDIA GeForce GT 330M OpenGL Engine -- 2.1 NVIDIA-1.6.26
    GPU Accelerated Windows1/1 OpenGL
    Is this expected behaviour Robert?

    ReplyDelete
  15. This is all very pertinent, as i'm trying to work out the reasons for performance difference between FF3 and FF4 at the moment on a project. FF4 is performing way, way slower than FF4 (which is for us performing worst of all browsers). Robert, could i share a link with you privately for you to look at? We'd be very interested in Mozilla's take on what we may be doing wrong...
    Rob.

    ReplyDelete
  16. What about improving hardware acceleration on the other platforms? i.e. OS X and Unix/Linux.

    ReplyDelete
  17. Great post. I really appreciate the analysis.

    ReplyDelete
  18. i can't help myself but there still a huge bug in the OSX version.... the right click menu for organising the bookmark does not appear in the folders of the bookmark bar.... under windows no prob... help !

    ReplyDelete
  19. I think the 10ms to 4ms fix should be put into FF4.0 as soon as possible. IE9 will be released real soon as well as FF4. FF4 should not trail behind IE9 on this. With this simple fix, IE9 won't have much of an speed advantage over FF4.

    ReplyDelete
  20. Are there any bugs number on these fixes?

    ReplyDelete
  21. Any insight into why the new demos meant to show off graphics in FF4 run much MUCH slower on my system than either the demo videos, or Chrome 10 on the same box? In particular: https://mozillademos.org/demos/shadows/demo.html the letter movement is EXTREMELY choppy on FF4 (RC1) vs Chrome. No extensions are enabled, just stock RC1. About:support output is as follows (running WinXP SP3):
    Adapter Description: Mobile Intel(R) 4 Series Express Chipset Family
    Vendor ID: 8086
    Device ID: 2a42
    Adapter RAM: Unknown
    Adapter Drivers: igxprd32
    Driver Version: 6.14.10.5303
    Driver Date: 9-21-2010
    Direct2D Enabled: false
    DirectWrite Enabled: false (0.0.0.0, font cache n/a)
    WebGL Renderer: Google Inc. -- ANGLE -- OpenGL ES 2.0 (ANGLE 0.0.0.541)
    GPU Accelerated Windows: 1/1 Direct3D 9
    Note, I'm seeing the same behavior on my home desktop system. On that one, it's a RADEON 5700HD with the very latest drivers. FF4 RC1 VERY choppy (nothing like the Youtube video shown on the demo page), Chrome 10, perfectly fine. On that one Direct 2D and DirectWrite are both enabled as well, and the Direct3D is 10 (running Win7 SP1).
    Should a performance bug be opened?

    ReplyDelete
  22. Speedreading scores & average draw duration:
    Chrome 11 Canary: 19sec, 18ms
    IE 9 RC: 9sec, 10ms
    FF 4 RC: 586sec, 29ms !!!!!!!
    Nearly 10 minutes.

    ReplyDelete
  23. I don't really care about benchmarks. IE9 feels much faster than Firefox 4 in my day to day browsing, and the UI is way more responsive. That, I care about.

    ReplyDelete
  24. Rob, thanks for the link to our site. We do actually quite a bit of benchmark testing and run as many benchmarks as our time allows - and include benchmarks from independent sources, Mozilla, Microsoft and Google. It is difficult to pinpoint the performance with no standard benchmark and our best option is to take an approach of an average result across a wide array of benchmarks (our standard run, which will be be updates as soon as you publish FF4, consists of 248 different tests (JS, HWAcceleration, offline page loading, Flash). The last time we did the test, FF4B10 was slightly ahead of IE9, but way behind Chrome, on average. It appears to us that plain code processing in offline page loading is much faster in Chrome than in any other browser.

    ReplyDelete
  25. To really appreciate the speed increase ROC is talking about try his build of Fx4:
    http://ftp.mozilla.org/pub/mozilla.org/firefox/tryserver-builds/rocallahan@mozilla.com-16644bf2dbd2/
    Then change 'dom.min_timeout_value' in about:config to 1.

    ReplyDelete
  26. geeknik, I don't understand what you're disagreeing with, exactly. Did you _read_ Robert's post? Say the second sentence of it?
    Martin, SVG performance involves a lot more than drawing. There's also the manipulation of the actual SVG objects. This last is fairly slow in Gecko; we are working on fixing that. Robert did say so in his discussion of Santa's Workshop.
    Thinus, stuffing more stuff into Firefox 4 won't get it out there faster. It just needs to ship. Now. All the fixes Robert talks about here will be shipping within three months after that.
    Martijn, that's expected behavior: there is no 2d hardware acceleration on Mac yet. So you're getting image compositing in canvas happening on the CPU.
    Rob, please report a bug on whatever issues you're seeing? Cc ":bz" and I'd be happy to take a look at things.
    default, it's being worked on!
    Alfred, dropping the timer to 4ms is just too high-risk at this point. And even if we did it wouldn't work right on Windows without even riskier code changes. We should have done that back in the summer, yes. But there were other things on our plates then. Again, this is something that we will be shipping 3 months from now!
    Tim, your problem is the "Direct2D Enabled: false" bit. That means no 2d graphics acceleration, so canvas is not accelerated. For the system where Direct2d is enabled, if you're seeing problems like that please do file a bug!

    ReplyDelete
  27. Yep, this build really improves the results.
    But WHY oh why you did not include patches used for that build into the main tree? so anyone can see that Firefox beats IE in it's own tests :)

    ReplyDelete
  28. What about say Maze Solver, FF 4 is much slower than IE9 RC, the page says that's mostly a CSS test. In addition, FF UI slows down while running this test.
    IE9 is also faster at HTML5 Soduku, about 50% faster. The page says that's a javascript stressing test.
    IE9 is faster at Map Zooming test as well, though the there is no FPS or score, it just looks faster on my machine (i7 965, gtx 580).
    IE9 is about 100% faster than FF 4 at Psychedelic browsing test.
    Additionally, IE has been sand boxed for about 4 years now, while still no sign of sand boxing from FF. What happened to the million eyes/coders thing? Aren't you guys supposed to be FASTER, not 4 years SLOWER?

    ReplyDelete
  29. While I don't have FF installed on this computer, I decided to test IE9 RC1 and Opera 11.01 on a Quad Core 2.66Ghz, 8GB DDR2, ATI HD 3450, computer.
    Results:
    IE RC1: 13 seconds
    Opera : 2392 seconds (39 minutes)

    ReplyDelete
  30. awesome, awesome findings and post. can't wait for these patches to land. thank you!!

    ReplyDelete
  31. So why aren't this fixes already landed on the trunk (before it was closed for RC)? You obviously know why Firefox is slow, what parts are badly implemented and have fixes for some of them, so why is Mozilla purposely not fixing the performance problems? IE9 tests have been around for quite some time and this is poor timing on Mozilla's part to make excuses on why Firefox is slower then IE9 in them.

    ReplyDelete
  32. This article is pathetic.
    IE9 is now the fastest browser because of a massive refactoring that addressed not only the graphic rendering pipeline but also the JS engine and overall parser. Just face it.
    You can spend time trying to convince people that FF4 does this or that better. You are just part of this FUD machine you're complaining about.
    Now, just focus on your stuff and make FF4 better.
    End of line.

    ReplyDelete
  33. This article is pathetic.
    IE9 is now the fastest browser because of a massive refactoring that addressed not only the graphic rendering pipeline but also the JS engine and overall parser. Just face it.
    You can spend time trying to convince people that FF4 does this or that better. You are just part of this FUD machine you're complaining about.
    Now, just focus on your stuff and make FF4 better.
    End of line.

    ReplyDelete
  34. I'm getting 10578ms with draw on your test, 10604ms without draw. This is running your build, so something's clearly wrong? I'm not sure. The other tests are super-fast as expected compared to IE9.

    ReplyDelete
  35. Your latest builds have already "expired" on the tryserver and have been moved to the 'old' subdir.
    Also, look into the "maze solver" test on the IE site. It says it tests CSS layout performance.
    IE9 RC gets 17 seconds, Firefox 116, Chrome 12 133, Opera 11.10 (2039) embarrasingly beats all with just 9.1 seconds without any kind of hardware accel. On 30x30.
    The funny thing is, on your build, if I ctrl-tab away from it immediately, it speeds up, getting 19 seconds. You have to refresh the page to make it solve a 'new' one I think, but tabbing away makes it use much less CPU as well, for some reason. (The same thing does not appear to benefit Chrome any.)
    If Mozilla can get basic rendering and speed issues sorted out (drawing/rendering times are essentially more important for the web at large than javascript speed, below a certain threshold), and get IPC working, then I'd be relatively happy, I think.
    Firefox's "old" javascript bug with the UI makes things extremely unresponsive if any webpages start to hog for any reason, which gets worse the larger number of tabs you have open (eventually almost anything will start hogging the CPU just because you have a lot of tabs).
    Even the "lowly" IE9 is going to start drawing power users away, if FF continues to be wholly unusable (for basic performance reasons) with a large number of tabs, and where a single hog can essentially deadlock the browser.
    (Unrelated security note, there seems to be no way to simply block 'mixed content' from loading on secure pages? The warning doesn't stop it.)

    ReplyDelete
  36. Meh, I use Opera 11 anway, well away from this pissing contest and faster than FF4 and IE9 in the real world.

    ReplyDelete
  37. Instead of bragging about useless synthetic benchmarks, why Mozilla doesn't concentrate its efforts first on the basics such as proper font rendering?
    Look at the issues reported here:
    https://bugzilla.mozilla.org/show_bug.cgi?id=635490
    https://bugzilla.mozilla.org/show_bug.cgi?id=626293
    I really think Firefox jeopardize its future by ignoring such fundamental bugs.

    ReplyDelete
  38. Style Thing and keldorn: roc wrote these patches last week, after we were already frozen for RC. They're not in Firefox 4 because they're not 100% risk-free and it's very late in the endgame. So that was not an acceptable time to land the patches; they will be landing as soon as we reopen for development after shipping Firefox 4.
    James Gentile, we have bugs on some of those tests, yes. Some of those are JS performance issues. Maze solver is about half painting and half layout (on Mac; on Windows the painting may well be faster with d2d). The maze testcase is mostly a stress-test for handling lots of floats; I filed https://bugzilla.mozilla.org/show_bug.cgi?id=641340 on that.
    Zeir, it makes sense that if you don't focus that tab while running the maze it goes faster: since it's not showing we don't have to do any painting or layout on those intermediate stages that you can't see anyway. So it obviously goes much faster!

    ReplyDelete
  39. Firefox 4 is definitely cool but still it hogs a lot of memory. Whatever happened to the cool guys a Mozilla.
    I write for http://hacktivist.tumblr.com/
    It is a lot about browsers. Especially Firefox.

    ReplyDelete
  40. More on the timout clamp : it's quite easy to see on http://ie.microsoft.com/testdrive/HTML5/PowerMeter/Default.html
    Just move the mouse and see the timer drop to 3.2 (and sometimes 2.0, I don't know why nor when exactly) ...
    Funny thing is they (MS) drop the limit from 15ms in PP6 to 4ms in Final (but thay may limit to 15ms on battery, didn't test) ...

    ReplyDelete
  41. Thanks for the clarification. I tried your build, and after updating my graphics driver I did indeed see majorly improved performance, roughly on a par with IE9 in the tests you listed.
    However, although you may have disproved Microsoft's FUD about GPU usage, the fact remains that, regardless of the actual reason you ARE slower without these improvements, and since FF4 will ship without them it will be less performant than IE9, which is a real shame.
    It seems MS and Mozilla have both done a similar job of enabling GPU usage, but MS have been more aggressive about optimising other parts of their renderer to really squeeze out performance. Maybe it's a false impression, and they've just been careful about how they code their demos, but if not then it seems like Mozilla could benefit from more focus on optimisation.

    ReplyDelete
  42. Gary McAllister16 March 2011 at 14:00

    "relatively small bugs in Firefox, bugs in IE9, and bugs in the benchmarks"
    You only really elaborate on your bugs but don't go into any details about IE.
    Is this just speculation? Also, not sure if it is deliberate (or subconscious) but your language seems to portray that IE has bigger (Cockroach vs Tarantula) bugs than you.
    Anyway, keep going, I really enjoy these posts.

    ReplyDelete
  43. Jon, they've most likely just coded the demos, then optimized to them. This is pretty clear in a lot of their SVG demos, which use the same 2 or 3 things over and over again, and those are the things IE9 makes fast, while ignoring other parts of SVG.
    Nothing wrong with that, but the only way to match that in general is to optimize for exactly the edge cases their demos use. We'll end up doing that, of course, but the value of that is unclear in general.

    ReplyDelete
  44. Robert O'Callahan16 March 2011 at 20:43

    I can't go into details about IE bugs because I don't have IE source code.

    ReplyDelete
  45. Boris, just an FYI that, as requested, I've opened the following bug about the Letter-Heads demo still being very choppy even with DirectWrite/2D enabled using a Radeon 5700HD:
    https://bugzilla.mozilla.org/show_bug.cgi?id=641758
    Thanks.

    ReplyDelete
  46. Hi, I'm an average user and must say that those miliseconds don't do anything to me, but prefer Firefox for all it's addons and customization, my looks nothing like original. And this is a BIG difference comparing to IE. Also I never had a crash and every page opened I wanted. All that for free and for anyone to see (the source code).

    ReplyDelete
  47. Well, SpeedReading test doesn't work for me on Firefox 4 RC at all. The page just doesn't load. I see the blue background and that's it, the letters never come up and I can't run the test ;/
    I'm on Windows 7 64-bit.

    ReplyDelete
  48. This is really cool. I was wondering why there were differences in some of these benchmarks.
    I was wondering, will your fixes for some of these problems be included in a 'stability' release of Firefox 4, or Firefox 5?

    ReplyDelete
  49. @Robert
    You can't give us details about IE bugs because you don't have IE code. Then, how do you know there are these bugs?

    ReplyDelete
  50. Robert O'Callahan30 March 2011 at 00:35

    Because I can see IE9 behaving incorrectly. You can run the testcase yourself in IE9 and observe that it doesn't follow the spec.

    ReplyDelete