Eyes Above The Waves

I totally agree with the decision not to use DirectShow or Quicktime. The best solution is actually what Silverlight has done: allow people to implement their own codecs. Silverlight 3 allows you to write your own video decoder in C# and include it in your app.
Unfortunately Javascript's speed improvements, while nice, are nowhere near what's needed for real-time video decoding, and Javascript may never reach that level. What I'd like to see is support for OpenCL in Javascript. Obviously this isn't a short-term solution, but in the future OpenCL could provide enough raw computing power to implement real-time video decoding. Then websites could use whatever codec they wanted and implement it themselves. OpenCL support would also fit nicely with the OpenGL support that's been proposed, and it would have lots of applications beyond codecs. All these crazy demos people have been doing with HTML5 video (chroma key/green screen, face detection, etc) could easily be massively accelerated with OpenCL.

That would be cool if it worked.

Writing your own codec in Silverlight sounds good, but nobody really wants to do it. People have been able to write video codecs with Java Applets for ages, and apart from the Theora applet I have not seen a massive call for it.
There are a few problems:
1) Writing a video codec is hard - doubly hard if you are aiming for hardware decoding.
2) If the aim is to distribute H.264 video files, then anyone how distributes a H.264 codec will need to pay license fees. This puts the burden on the web sites themselves, rather than on the client but it is still a burden.
3) Ideally, video files should be requested through the browser's HTTP code, so that things like local caching and proxy settings follow the user's settings in the browser. This is something that Safari's use of Quicktime gets wrong - Quicktime downloads the video files itself, completely bypassing the browser.
My understanding is that Silverlight does have some nice features for video, but the video tag is meant to make embedding video easy without having to develop custom Silverlight/Flash/Applets for themselves.

What about Gstreamer then?

The aim is not explicitly to allow H.264 but to allow choice and competition. I wouldn't actually expect everyone to develop their own codecs; obviously you'd have some open source projects like Theora or x264 doing the hard work of implementing codecs and web authors would just include them as libraries.
I'm not suggesting that Silverlight actually be used to implement this; I'm saying that if OpenCL was built into browsers it would enable this and more. However, it certainly would be deliciously ironic if someone wrote a library which implemented HTML 5 video + Theora in IE via Silverlight.

This doesn't help either:
https://forums2.symantec.com/t5/blogs/blogarticlepage/blog-id/vulnerabilities_exploits/article-id/199

I think, you have no choice: you should later add support for H.264 because it will be supported on mobile & portable devices playable format for it's browsers.
Everything comes to what...

Oooh lookout Robert, someone has mentioned mobile! Seems everything this last 12 months at Mozilla is driven by whatever suits Fennec. Maybe you'll have to reconsider given that H.264 is installed on a lot of phones.
Back to the topic of supporting more than theora or other codecs IE doesn't (and therefore content producers are unlikely to) support, what are your thoughts on Mozilla contributing to GStreamer?
GStreamer is LGPL and aims to provide "... a library for constructing graphs of media-handling components ..." in a cross-platform fashion. A 'relationship' between Mozilla and GStreamer similar to that between Mozilla and Xiphos would surely be a good road towards more video support in a tighter development environment (well, as tight as you can get in open source) would it not? This seems to be route taken by Songbird isn't it? Whilst Songbird appears annoyingly audio focused, they have supported video at some point when I tried one of their pre v1 builds. Surely there would be some code Songbird could contribute given it uses the same platform as Firefox?
P.S. I didn't include this response in mozilla.dev.app instead of bugzilla because this post already has it's own context.

Agreed. Firefox should only support open codecs. People complain about the lack of h.264 because it's the latest and greatest codec around these days, but it's really not necessary for web video, in my opinion. Sure it would be nice to have but it's not worth the license issues. Some people want h.264 support because mobile devices have hardware decoding for it so it could save a lot of battery power. Well, maybe it's time for hardware Theora decoding support.
The tag is a lot like the tag when the web first supported that. A lot of people still used GIFs despite the license/patent issues that it had. It'll be interesting to see which video formats will win the war, especially since the competition (Apple, Google) isn't as dedicated to openness as Mozilla regarding video.

Could someone write a plugin that adds codecs to the video tag?
I understand that someone wrote a little wrapper plugin that pretended to be the Windows Media Player plugin on linux but inserted a Moonlight object instead. (free as in beer h.264 and vc-1 decoding everywhere, courtesy of Microsoft)
As the video tag is handled internally, is this sort of thing possible.

Also, I'd like to show my support of not supporting the platform media pipelines - I agree on security reasons alone.
It would create huge new surface area of unhardened, performance tuned, code.
FYI MediaFoundation is DirectShow's successor on Windows and 3rd party DirectShow codecs are installable on Windows 7.
Number 1. is a admirable goal and I'd be getting some people working on making the encoder more accessible and attractive to content providers, as well improving it's output.
Another aspect in getting adoption is spreading the vorbis decoder around. Is Mozilla in the business plugins for other browsers? One option I can see is writing/porting/supporting a decoder for Java or Silverlight/Moonlight so that content providers could provide fallback without encoding again. Unfortunately I don't think flash allows additions to it's internal media stack.
*end of ramblings*

I'd say, Mozilla has a choice. Although of course, Mozilla's leverage might not be great enough to almost alone push Ogg audio / video. The people battling against this plan include Apple, various handset producers (probably even Nokia), Microsoft and somehow Google. The only other supporters of open video probably are Gnome / KDE / Linux developers.

"Note that the problem is bad enough that in Windows 7, Microsoft isn't even going to allow unknown third parties to install DirectShow codecs."
Unless this has changed post-RC, it's not completely true. What MS has done is to always prefer "inbox" decoders over installed ones. This is a pain because the inbox decoders, while very robust, aren't of the best quality (the ASP one postprocesses too much and blurs the picture, the AVC one is about 40% slower than the good ones unless when using hardware acceleration). However you can install new codecs just fine, and they'll be used if there isn't an internal (to W7) decoder that can handle it. Applications can workaround this by constructing the DirectShow graph themselves (MPC-HC for example does this and is not affected).
The inbox decoders can be used outside of DirectShow (they have a newer, simpler, more robust API), are digitally signed, are MS' responsability security- and stability-wise (bugs here would affect many other parts of the OS, from WMP to IE to thumbnail generation), and seem to behave very well overall. The same filters seem to handle a variety of formats (from MPEG2 to AVC).
I haven't really looked into it, but don't the WMP and Quicktime NPAPI plugins expose all the mentioned problems already?

Are you guys planning to add other uncumbered codecs in future releases, like, say, Dirac (which always seemed to me the natural open competitor of H.264)

"-", thanks for that clarification.
The codecs installed in Windows 7 do sound more attractive. The problem is that by the time Windows 7 has enough market penetration that we can rely on it (say 5-10 years from now), those codecs will probably not be very interesting. Unless Microsoft starts aggressively auto-pushing codecs to old systems, the platform codec route is never going to be appealing.
Yes, the WMP and Quicktime NPAPI plugins do expose a lot of attack surface already. This is a problem. However, they are not our responsibility and are barely used.
Diego: Dirac doesn't perform well at Web bit rates, and AFAIK no-one is saying it does, or trying to change that. Dirac would also require us to do legal analysis that has not been done. However, adding Dirac is certainly something to look at in the future.

How does this solve the security problem? I'm not a windows user, but won't Firefox already launch windows media player based on mime-type/extension/URL? If I send an AVI or WMV from a server, won't Firefox offer to open it? How would this be any different than letting the video tag delegate directly and offer the user a dialog warning about using third party codecs?
All of these excuses seem pretty weak, why not just admit that this is a Stallman-esque stubborn headed refusal to allow linkage with non-open code, and that the greater mission is to somehow, defeat H.264 as a web video codec, a mission which seems extremely naive.
The usual rants about freedom seem self contradictory. "Freedom" means being prevented from having choice in codecs. People with GPU acceleration or mobile devices should just deal with it until the future Nirvana of Theora hardware acceleration blocks arrives.
Seems to me that this is all going to force a bifurcation in HTML5 browsers between Firefox and Chrome/Safari, and this can only benefit Adobe and Microsoft. Good job.

"How would this be any different than letting the video tag delegate directly and offer the user a dialog warning about using third party codecs?"
It would be equally as annoying, a lot more work for us, and shift responsibility from Microsoft to us. No thanks.

There you go. The security issue isn't a theoretical one:
http://www.h-online.com/security/Web-pages-infect-Windows-PCs-via-new-DirectShow-hole--/news/113695

So, it won't be enough to just install my favorite codecs on my OS.
But how does it work then ? How can I install video/audio codecs on firefox ?

If there will be no broadly used codec, it's going to suck.
We need (at least) one codec supported by everyone.
To precise my question : Is going to be relevant on the web by the help of Apple letting users add a 3rd party OGG codec plugin (and thus avoiding the patent issue) ?
As a developer, I always thought I would be able to detect if a browser doesn't support my codec, and provide a link to some codec download...

Lots of your arguemnts against directshow are fairly illusionary, imho.
People do install h.264 decoders, it's not used just in windows 7, you know.
I think you can enable directshow support fairly safely, if you recommend the users to isntall CCCP, which is a well maintained (and tested) codec suite based on ffmpeg/ffdshow. It's stable and very low-risk. It's an ellegant and *free* solution to the inability to ship the needed codecs in Firefox itself.
But it seems to me you just want to push Theora... it doesn't seem you can get very far with html5 that way though.

I call bullshit. Programming and Politic don't mix and if you have to fight a war do it on your own and don't try to use your fellow supporters to do your warring

Two things are clear from the post: There is no technological or legal barrier to Firefox implementing DirectShow support in the video tag.
Second, the only reason why this developer refuses to give users access to their DirectShow codecs in Firefox is IDEOLOGICAL. Somehow ROC finds it distasteful that users are slumming around with DirectShow, and by God, he won't let his precious Firefox be an accessory to anything so profane.
I can't overstate how much this holier-than-thou attitude pisses me off. What is the point of an open source project if not to provide more freedom for the project's user? Who came up with this OSS totalitarianism, according to which it's OK to make intentionally crippled software that prevents users from doing what they want? Gee, I'm really feeling the freedom now! Please recognize that this is what you are doing, and that there are no licensing nor technical reasons why this software should remain crippled. You simply want to recruit your users into a crusade against some ubiquitous industry standard, and you don't care that you never received our consent.
I hope you re-examine your reasons for thinking that this sort of totalitarianism is wise in an OSS project. Its effect is to drive anyone who wants to watch a Youtube video in the browser to either install Flash(!) or use Chrome. You know you could prevent this by adding a perfectly legal and feasible feature to FF, and yet you proudly prefer the status quo. How could you possibly be proud of this?

You're all living a pipe dream. Chrome, Apple, and MS already have license to use H.264. They also have mobile hardware acceleration support. Why would they just toss that out the window, wait for hardware vendors to add non-existent Ogg acceleration hardware support, and revamp their infrastructures to support it, when there is no benefit to doing so above the insignificant cost of h.264 licensing? While the license fees might be daunting to a small company, they are literally nothing to the likes of these.
Firefox is dead in the water if they don't add h.264 support, and all of their current market share will go down with them. The market share that makes them a player in the game will be the very thing they lose by not adding it.
There are definitely battles to pick, and this isn't one they can ever hope to win.

Another reason is that we don't want WMV and legacy MPEGs to join the codec mess. Having market split between two is bad enough already.

Firefox without H.264 is like outlook without email. Why do you promote something what is not good, like free codecs without live encoders etc, instead of making option to install H.264 codec and play videos from the browser?
H.264 is not just the codec, it's the process of recording the digital video, editing it, encoding and transmitting and finally viewing, in your browser.
If you think you can replace this with theora... it's just silly.

"People complain about the lack of h.264 because it's the latest and greatest codec around these days, but it's really not necessary for web video, in my opinion."
Are you serious? H.264 offers the best quality/compression ratio compared to any other video format, and the less you need bandwidth to deliver good quality, the more suited the solution is for web video. Theora flat out sucks compared to what you can do with x264, and it will never reach equal quality. Not implementing H.264 support and blindly shouting for Theora is stupid and waste of everyone's time.

It sounds like the real problem is poor isolation between the user process and the plugin architecture. Sandbox those and the security problems aren't yours - plus Firefox gets less crashy overall.
That said, I used the QuickTime plug-in in Firefox (nee Moz/Fiz-illa) for nearly a decade, and it worked well, for all sorts of oddball codecs.
I appreciate the openness stand, but these things should get worked out in free competition, not by artificial barriers.

Sandboxing DirectShow plugins would require two things: 1) that you can play DirectShow video in one process and deliver the output to another process and 2) that you can run the DirectShow process in some kind of low-rights mode without breaking DirectShow. The former can probably be done, the latter is highly doubtful.

This sounds like a job for Electrolysis!
https://wiki.mozilla.org/Electrolysis
Future Phases
* Security sandboxing (involves removing the native widgets and event loop from the content processes)

i filly agree with directshow. allow people to view that type of porn picture.

All OSes which Firefox runs on have a codec framework. Patents aren't Mozilla's problem, it's the user's problem and it also depends on the country the user is in.
Just make the browser search for missing codecs like it does for missing plug-ins: make Firefox go to a page on Mozilla.org and point the user to a known good codec packs (e.g. CCCP).
Chrome plays H.264 and Ogg and Safari uses the OS's codec framework to play non-H.264 video. Their user share is going to take off because the user doesn't want to be part of this fight. By all means Firefox should have built-in Ogg support but not to the exclusion of everything else so it forces the user to change browser depending on the website they view.

Windows Media Foundation, the technology that replaced DirectShow, solves a lot of these problems. In particular, the H.264 codec is written by Microsoft and it doesn't have many of the negatives written in this article. Of course, this is Windows 7 only (and maybe Vista), but Windows 7 is rapidly replacing XP.

Don't worry, Microsoft are doing the sensible thing by providing Firefox and Chrome plugins for their H.264 codec bundled with Windows 7.
Sanity prevails! webm + H.264, both available to the tag. Now THAT is a solution.

taken from my reply here: http://forums.mozillazine.org/viewtopic.php?f=40&t=2962577&start=15 it depends on how do you read that blog post by occallahan. but here's how i read it: "we don't care about few %s of xp users". and that's it. :) to break it down further 1-since that time firefox has lost a lot of users so their talk about open formats means zilch. that failed (and that was their main excuse) 2-they essentially say nobody (or very small numbers of people) on xp ever played h264 because it's not bundled with os. that's utter nonsense. it's as if in 2009 he didn't notice flash needed to be [b]dloaded and installed.[/b] just like ffdshow. 3-there were never that many dshow h264 decoders. "rich APIs of video"? give me a break, flash video was flash video and is flash video even today. just like h264. [b]it's not interactive or anything.[/b] just one more excuse, because if you put few on top of another, it makes more sense! #-o 4-many flash versions spread malware, and no dshow codecs were malware. malware is malware and it can present itself as anything (these days it's usually flash and java "videos" that spread most malware, paradoxically), that has nothing to do with dshow decoders. 5-there we go again! like flash didn't have holes and it's still most prevalet way to play video on the web. 6-just lazy it is interesting that this decision boils down to preferences (or excuses) of one programmer. and this "Currently no browser or browser plugin vendor is using codec implementations they don't control." man....how the hell can this man be in charge of anything... #-o (i'll post this there on his blog too) and oh yeah, special greetings to ms who's just re-labeling their os' to be able to sell something new, even if every new incarnation has nothing new....heh and it's REALLY interesting to see that many video players (pot player by daum, mpc-hc, gretech gom player etc.) that bundle codecs acutally use same thing that ffdshow used, ie ffmpeg code....hehe....a history has a way of laughing at stupid ideas. (and windows media player always was and will be junk for anybody into video) so overall it's just laziness and poor predictions of what will future bring. this also serves as a reminder that video was never really truly solved on the web, even since old days of realvideo, windows media video etc. there never was one format that was good eneough for everybody, and that's a pitty. because mpeg-la should (for example) give away these things for free (they essentially are free, they won't make money on windows or linux video decoders), OR price it at some very low price so that anybody can buy it.