Eyes Above The Waves

Robert O'Callahan. Christian. Repatriate Kiwi. Hacker.

Wednesday 9 November 2011

The End Of Plugins

If you haven't already heard, Adobe is going to stop Flash development for mobile browsers, which given the rising importance of mobile means they're effectively giving up on Flash on the Web altogether. Now there are strong noises that Microsoft is going to stop Silverlight development. This comes on top of Windows 8 Metro mode not supporting plugins (which of course follows Apple's early lead in iOS). This is huge! It means that browser plugins are going to go away. Oracle hasn't said anything yet, but there's no way browser vendors are going to carry NPAPI support forever, or add any new NPAPI features now, just to run Java and some other relatively little-used plugins. (This is not official Mozilla policy, mind you, just my prediction.)

What does this mean to Mozilla? It means we need to stop investing in new features for better integration with NPAPI plugins. Our investment in NPAPI support should focus solely on improving the user experience for consumers of existing plugin content, but limited by the knowledge that the consumption of such content --- and production of new content --- will decline dramatically over time. There's still probably quite a few things we should do there. Eventually we'll be able to disable plugins and rip out a lot of code.

What does this mean for the Web? This is a big victory for open Web standards. Two powerful proprietary competitors have been defeated. Sooner or later we'll be able to stop taking plugins into account when we design new features (e.g. our full-screen support has to consider how plugins work in a full-screen document).



I just came here to say Hooray!
Does this mean https://wiki.mozilla.org/Plugins:AdvancedKeyHandling will no longer be invested in, even though it would fix https://bugzilla.mozilla.org/show_bug.cgi?id=78414 that is over 10 years old? By the way, your comment form doesn't warn about cookies being disabled preventing the captcha from working.
So, how will Firefox achieve hardware decoded/processed HTML5 video playback now? On my AMD E-350 based netbook flash HD video playback (hardware h264 decoding) is fine, but webm stutters etc.
Stephan Sokolow
@dbcooper: The same way Flash does, by writing code which offloads some of the work to the GPU. There's nothing magical about what Flash does to accelerate H.264 playback. The Firefox guys just haven't written the code to do it for WebM yet.
@Stephan Sokolow Well the thing is, that almost all GPU's now have dedicated H.264 support in their dedicated media decoding/processing modules. Using the API's for them is a lot simpler than writing your own Open-CL programs to be executed on the various GPU's.
I agree that this is a big step forward towards getting rid of plug-ins. However, this is also a big challenge for browser/app manufactures: wherever HTML5 offers a choice, they have to agree on the same subset. For example: providing video/audio content with flash is of course easier then in HTML5, where you are faced with several more or less exclusive alternatives (e.g. Mozilla supporting WebM/OggVorbis, Apple supporting H.264/AAC). As long as the situation there does not improve towards a consensus, I don't see the end of plug-ins...
flying sheep
how do plugins a better job than a video tag pointing to both webm and h.264 files? half the server space, ok, but that only matters on *really* video-heavy pages. also, i find “hardware-accelerated” somehow funny: everything on your pc is hardware-accelerated. let’s just say “gpu accelerated” if we mean that, ok? @blog entry: i love these news! i tried surfing without flash, and i can say that it will be really nice once youtube transcodes all vids to webm instantly: i hate it when i can watch every vid except this certain new one for which i browsed to youtube in the first place. oh, and newgrounds will be missed, too.
This is great news indeed. I just hope this means browser makers will invest more time in bringing HTML5 audio/video up to par with the current Flash performance. For example, on Android (2.3) performance is buggy and Flash is the better alternative.
Daniel Cater
Yes, this is good news. To echo some of the comments above though, work needs to be done to make WebM video playback smoother and less CPU-intensive. On my netbook, I stick with Flash on YouTube (as opposed to opting into the HTML5 trial as I do on my desktop) because it is much smoother and less CPU-intensive. Even on my desktop it's better in Flash, but bearable in HTML5 so I stick with it. I think this is due to Nvidia PureView / VDPAU support in Flash. Prioritising hardware accelerated video everywhere (and hardware accelerated layers on Linux where it is still not enabled) would be a good way to speed up the deprecation of Flash I think.
Franklin Chen
Good news indeed, the imminent end of plugins! Next: will closed app stores ever die?
What you say sounds to good to be true, and makes a lot of assumptions on how the future will actually look. I hope you're right, but I'll only believe it when Flash crashes and hangs are not a problem I worry about every day any more.
Hmm, that means we might need to move away from using NPAPI for our app. What other thing can we use to draw a large rectangle inside a Gecko app from C/C++? Is XTF still alive? (No, whatever HTML/JS based solution will not be useful, unfortunately. Legacy code.)
Yes it's good news, but not quite "effectively giving up on Flash on the Web altogether". Macromedia then Adobe struggled for a decade to get Flash on phones (2003 "We're big in Japan on Docomo", 2004 "Flash Lite for Windows Mobile!", 2005 "we're talking to carriers", etc.); that's over but Flash player is still the most popular software on PCs and for now is on many Android devices. Flash will become a niche and legacy plug-in, joining Shockwave player (still going, http://www.adobe.com/products/shockwaveplayer/ ). I hope there will be some way to sandbox or virtualize an old-school browser with plug-in support at the browser or OS level, otherwise all that old Shockwave and Flash (and VRML, Java applets, etc.) content will be lost to future archivists.
(Apparently something decided to eat my comment; here's a retry from memory.) Sounds like we need to move (our local Gecko-based app, not a website) off NPAPI, then, if it's going to go away. Too bad the majority of our window is a giant NPAPI plugin... What other techniques can we use to paint into an arbitrary rectangle in the app from C++? Canvas / JS won't work well for us (lots of C++ legacy code / users depending on it); as far as I know, attempting to use canvas directly from C++ is suicide (mostly, because we're not in libxul; secondly, because the sorts of things we want aren't the sort of things Firefox wants, and historically that's a losing proposition). I don't know of XTF is still alive - haven't heard of anybody, at all, using it. Dropping a giant HWND (or equivalent) in front of the app is no good either, that means we can't usefully drop XUL in front of it, in addition to having all sorts of horrible issues with focus. Hopefully there will be a useful replacement. I think I had a second point here last time, but can't remember what it was.
Would emscripten allow you to convert the c++ code to js in a meaningful fashion?
Mook: Good question. The things we'd most like to remove are windowed and in-process plugins --- that would let us simplify a lot --- so if you can run out-of-process and windowless, we can keep that alive longer. What do you use to draw into your HWND today? Can you paint the contents of your window into an RGBA in-memory buffer, get that using JS-ctypes and draw it to a canvas from JS?
Robert: Hah, we're awesome - we're an in-process windowed plugin, yay! ;) This is Komodo, where the text editor component - scintilla - started by emulating the Win32 RichEdit APIs; everything is done via window messages. I expect there is going to be transition pain for us whatever we do; so if there's an API that's designated as being the goal we should head towards, it should cut down on wasted effort. Skimming the code we have right now, we're doing things with HDCs like ::FillRect. Since it's got a platform abstraction layer of sorts already, we'd probably need to write a "Mozilla" version of that and use it from ctypes. Does ctypes support using typed arrays yet? I don't see us being able to pump data fast enough otherwise... (mostly because we're occasionally horribly inefficient, just not bad enough to be noticeable yet.) At least we've already started on reimplementing bits in XUL instead of letting Scintilla do its native widget thing, mostly for extensibility. P.S. Following the name listed in the comments instead of going by your IRC nick feels odd ;)
Sounds like you could create a DIB HDC and draw your window into that and then make the DIB data buffer available to JS to draw into a canvas.
Wishful thinking! Mobile != Desktop. The reason Adobe stopped mobile flash because it was consuming too much battery power and has been continuously dropped by the vendors. Although the CPU power and memory on mobile devices advance fast, battery technology is still where it was 5 years ago with fractional improvements. On the desktop, plugins are here to stay. They are established too deep in. Microsoft metro browsing is targeted at low power touch pads that would run Windows 8 as the operating system. That said I dont think NPAPI would put too much of a burden to browser vendors.
Great. So the web will look like it's 1999 all over again.
In many ways, this would be nice; however, I think it's even more clear now than it was when this article was written that plugins can't go away yet -- there are too many things people want and need to do that can't be done without some form of native extension. We're getting closer, and flash is certainly past it's day, but flash was not the only plugin.