Thursday 18 June 2009
The Price Of Freedom
With the imminent release of Firefox 3.5 and the big step forward for unencumbered video and audio that this represents, there's been a lot of discussion about the merits of the free Ogg codecs vs the flagship encumbered codecs, especially H.264. Proponents of the encumbered codecs tend to focus on the technical advantages of patented techniques used in H.264 but not in Theora. But the real question that matters is this: at comparable bit rates, in real-world situations, do normal people perceive a significant quality advantage for H.264 over Theora? Because if they don't, theoretical technical advantages are worthless.
(It's not helpful to ask a video compression expert if they can see quality differences. Of course they can, they're trained to. That doesn't tell you what matters for the other 99.9999% of the population.)
One issue that has made a comparison difficult is that encoders are so tunable. People always say things like "oh, your H.264 would look much better if you set the J-frame quantization flux capacitor rank to 8". So Greg Maxwell had a stroke of genius and did a comparison using Youtube to do the encoding. Now, people who think the H.264 encoding is sub-par are placed in the difficult position of arguing that Youtube's engineers don't know how to encode H.264 properly.
In fact Youtube deliberately doesn't squeeze the most out of H.264 encoding. That's because in real life there are tradeoffs to consider beyond just quality and bit-rate, such as encoding cost and bit-rate smoothing. But that's fine, because these are the real-life situations we care about.
Maik Merten recently repeated Greg's experiment with different video in larger formats. In these tests, it seems pretty clear that there is no real advantage for H.264, or even that Theora is doing better.
We really need someone to do a scientific blind test with a wide pool of subjects and video clips (hello academic researchers!). But H.264 proponents have to demonstrate real-world advantages that justify surrendering software freedoms and submitting to MPEG-LA client and server license fees (not to mention the hassle of actually negotiating licenses and ensuring ongoing compliance).
Comments
If the theora videos of the same quality take twice as long, I would assume that that would be considered a big problem for mass video sites such as youtube.
I think it is a real shame that Firefox is unable to have H.264 built in, but I understand the reasoning that went into that decision. It is also a shame that browsers like Safari decided not to support Theora in addition to H.264/quicktime, but no doubt they have their reasons as well.
If a web site wants to publish video, which codec should they choose to ensure their users can actually view it? The answer is still, frustratingly, Flash video.
My main beef is with the other browsers' refusal to support Theora as a lowest-common-denominator format.
1. Hardware decoding support. This is a big win for mobile, and likely the big reason why Apple is pushing it so hard. Better battery life and better performance sans cpu upgrade.
2. Speaking with a little experience, encoding video, especially in large quantities isn't cheap... or fast. H.264 has decent penetration already, hence many organizations already spit these files out. Supporting h.264 means they already have video encoded. Just need to implement support on the frontend. It's a much lower implementation than having to build out the encoder farm to re-encode all your old video, and encode yet another flavor of video for every one you publish.
That said, ogg's license is nice and just sooo right... so I really am hesitant to jump on board the "oooh h.264 or bust" train.
Andrew: Yes, I agree we really need a common format. We want that format to be a free one (both in cost and in freedom). The way to achieve that is to get content out there in a free format that Apple's users (and eventually Microsoft's users) want to play. That's why we're doing what we're doing.
Robert: Yes, the hardware decoding issue is real. But it's not a technical issue, it's just the choices that device manufacturers make. I don't think we want device manufacturers to have control over Web formats.
Dailymotion has reencoded lots of videos. This is possible.
LOL. Flux capacitors in codecs. Hey, maybe that's the way research should go since wavelets have turned out to be a bust. :)
Ok, serious answer: Firefox should have hooks into generic multimedia frameworks (DirectShow on Windows, xine-lib on Linux, QuickTime on OSX). That way Firefox would be able to play anything you have a plugin for in that framework. The Arora browser does it like this, thanks to the Phonon framework in the Qt library.
That way you'd avoid the need to ship a h264 decoder in Firefox and I'd bet most Windows users have some sort of DirectShow h264 decoder already installed.
Dailymotion first began use of Ogg video encoding in conjunction with the One Laptop per Child Foundation’s XO laptop. Dailymotion will have 300,000 Ogg encoded videos by the third quarter of 2009, and will re-encode pre-existing site inventory in the new format.
http://www.businesswire.com/portal/site/home/permalink/?ndmViewId=news_view&newsId=20090527006237&newsLang=en
That said, it's a tough sale to convince companies to support another encoding. Especially if it's not an immediate replacement or brings in a new market (such as h.264 did for mobile). It's hard to monitize video right now (as Google has proved with YouTube). THAT imho is going to be the major challenge. Even if the technology is solid (hardware decoding and all)... how do you convince someone this is a wise business decision?
http://lwn.net/Articles/337101/
Also, it's somewhat ironic that the most widespread "hardware support" is probably the iPod and iPhone and neither support some of the more advanced (i.e. incredibly power hungry) features of H.264 and, presumably as a direct result of this, Apple's encoders don't support them either. So the real H.264 enthusiasts have to argue that both Google and Apple don't know how to encode H.264 "properly".
"Apple's encoder isn't a great example of H.264 decode complexity, as they leave out a lot of the most interesting (but slow-to-decode) features. For example, Apple's encoder doesn't support CABAC, multiple reference frames, or 8x8 blocks. It's really Baseline + B-Frames." - Ben Waggoner,
Greg Maxwell has another nice post where he details the pricing decisions of the MPEG-LA that favour the big players, and why supporting Theora makes financial sense even if you're already commited to using H.264:
http://lwn.net/Articles/337088/
One point that hasn't been mentioned much here is that H.264 can be played on a larger number of clients than Theora currently can (e.g. iPhone, Android, Mac without using Flash). Thus sites like YouTube wouldn't be able to simply switch to Theora; they'd need to encode all videos in two formats for the time being. This seems like a significant cost to pay even if the codecs are otherwise comparable.
I have also been informed by people who know more about video encoding than I do that, much like when comparing search engines, individual tests are less informative than huge batteries of tests over many types of input; supposedly Theora is competitive on some input types and less so on others (whereas H.264 is rarely _worse_). I have neither the knowledge nor the research to back this up.
In the end it will hopefully stop mattering someday, since eventually patents expire (see also GIF); and I am thankful that, in the browser I work on (Chromium), we are supporting Theora out-of-the-box so that sites that wish to choose it are free to do so.
Certainly, to people and companies whose only values are profit and power, I have little to say.
As you know, Chris DiBona said on the whatwg list that Theora was so bad that Youtube using it would literally destroy the Internet. So there's definitely some ignorance about. To be fair, the Theora encoder has been improving rapidly lately, so a lot of people's information is out of date.
Gusar: almost no Windows users have an H.264 DirectShow codec. The first version of Windows to ship with H.264 will be Windows 7. Only a very small percentage of Windows users will have downloaded one. There are other problems with using DirectShow or Quicktime; note that Chrome doesn't, for example, even though they could reuse Apple's Quicktime bindings for Webkit.
In short - rhetoric that claims that Theora is free as in cost as well as in freedom is inappropriate. Theora has its cost (or more appropriately, its financial risk) as well. That said, I wouldn't ever demonize Theora as "would literally destroy the Internet" (umm, seriously?).
As Peter Kasting said, in the end it will stop mattering; eventually patents expire. Sadly, I don't see this issue getting resolved until that happens.
This won't stop mattering when patents expire, unless you believe that H.264 is the last word in video codec technology. I don't think anyone believes that. H.265 will arrive, encrusted with many new patents. If H.264 becomes essential for the open Web platform, patents and license fees will be accepted as a normal part of the Web and H.265 will be irresistible.
That sounds like overstating the case a bit? It's very different for a vendor to say "we will ship h.264 and Theora" than to say "h.264 should be the standard, the end." Most defenses of h.264 I've seen are _reactions_ to arguments about why vendors should not even ship it at all.
"Certainly, to people and companies whose only values are profit and power, I have little to say."
At whom are you directing this? Last I checked Chrome, Safari, and IE didn't cost anything so I'm not sure exactly where the "profit" part of this comes in. And since when is it somehow conducive to a company gaining power when they ship support for both the popular encoding formats?
I'm just asking for care in the rhetoric here. This sounds a lot like demonizing actors who take actions that I think reasonable people can disagree on.
"As you know, Chris DiBona said on the whatwg list that Theora was so bad that Youtube using it would literally destroy the Internet."
I thought it was obviously that was hyperbole. I also thought it was clear (from the coders themselves) that the Thusnelda changes were not only not yet in the mainline but not proven to have "improved Theora a lot" (I do read their comments since I am a proponent of Theora and would like to see it succeed--the coders did their best to tamp expectations but there was a lot of exaggerated claims made by others afterwards anyway). I _also_ thought it was clear from my comments above that testing codec quality is difficult to do well and I personally do not feel comfortable making judgments about relative codec quality (and am suspicious of those who rapidly do).
What I am trying to say is, you seem to take it as given that there is no qualitative reason to chose h.264 over Theora, _and_ that the people at companies that choose codecs to use are more ignorant than you and me, and must be unaware of codec improvements. I doubt very much that either assumption is correct, but I have no proof.
My hope, like yours, is that by shipping Theora support, Google will either cause any patent issues to surface or else make companies like Apple comfortable enough to ship it themselves. I also hope that Theora does improve, to the point where it's so compelling in the market that everyone uses it on the merits. In the meantime, I feel a bit like I've been tarred with the "immoral actor" brush for what do not (from my perspective) seem like reasons of "profit and power".
Supporting H.264 increases the chance that it (and accompanying license issues) will become an essential part of the open Web platform. One would only ship H.264 support if one sees that as an acceptable outcome.
"At whom are you directing this?"
No-one in particular, but anyone who states that only economic considerations are relevant to a decision.
"This sounds a lot like demonizing actors"
Then I apologize, I certainly don't wish to demonize.
"I _also_ thought it was clear from my comments above that testing codec quality is difficult to do well"
Indeed. But H.264 proponents say the Web should use H.264 because it's better quality, but that only matters if it's obvious to normal people. So the important thing about Greg and Maik's examples is that H.264 is not *obviously* better. You don't have to be an expert to decide that, in fact as I said in my post, it's better not to be.
"What I am trying to say is, you seem to take it as given that there is no qualitative reason to chose h.264 over Theora, _and_ that the people at companies that choose codecs to use are more ignorant than you and me, and must be unaware of codec improvements."
I don't. It's very plausible that there are other kinds of video where H.264 has an obvious quality advantage over Theora/Thusnelda under Youtube conditions. I'm asking H.264 proponents to demonstrate that instead of just talking about how much more advanced H.264 is.
(But I also expect that due to major industry backing for H.264, and the fact that the Theora encoder used to be so bad, there's a lot of "common wisdom" that's no longer valid.)
"My hope, like yours, is that by shipping Theora support, Google will either cause any patent issues to surface or else make companies like Apple comfortable enough to ship it themselves."
Indeed, and let me repeat again how very glad I am that Google's doing that.
"I also hope that Theora does improve, to the point where it's so compelling in the market that everyone uses it on the merits."
Given the players in this game who already have H.264 licenses or are even MPEG-LA members receiving license royalties, I'm not confident that equivalent performance (which we seem to have achieved in at least a subset of use cases) will be enough to win. But we'll keep doing our best.
I don't believe that. With HD rips now a solid part of the scene, and the Anime fansubbers having used h264 for years now, h264 decoders on Windows are more spread than you claim. Though, I guess you could argue that those who download this stuff are a small part of the whole public. But if that's true, then the piracy problem certainly isn't as big as the media industry wants everyone to believe (this is a different topic though, I just thought I'd use this little bit in my argument to show that indeed many Windows people might have a h264 decoder already installed).
I wouldn't know about problems using DirectShow in Firefox, but it would be way to get h264 support into Firefox without the need to ship the decoder itself. If it's too much hassle (or downright impossible), then that's too bad.
"I'd point out that the problem here is that it's not PROVEN that Theora is patent-UNencumbered"
It goes even further - Xiph *knows* that there are patents in Theora owned by other companies (other than On2 that is), see here: http://forum.doom9.org/showthread.php?p=1296267#post1296267
BBB is, as is stated in that post, from the Software Freedom Law Center.
I don't have all the answers, I just know this:
Theora is not competitive quality-wise and h264 is already quite spread, both online (Youtube, other flash video sites) and on hardware devices (at least Apple and Nokia), so Theora gaining foothold will be extremely difficult and it might even lose this battle, despite being in Firefox and Chrome. Especially since Chrome also supports h264, and Safari supports only h264. Then there's the big elephant in the room for which who knows when it'll gain any video tag support - IE.
To finish off, a mini rant: This whole situation regarding patents just plain stinks. Why must the entire world suffer just because one country has insane patent laws?
Maybe there are a few million HD rippers, and a few million Anime fansubbers. Compared to about a billion Windows users, that's almost no-one.
"Theora is not competitive quality-wise"
I linked to examples that show at least in some cases, under Youtube-like conditions, that's not true. Please stop repeating this claim until you can show examples under the same conditions that support it.
Dark Shikari is not an x264 "advocate". He *develops* it. Is that reason to dismiss his claims? I think it's reason to listen to him even more, because he knows how h264 works and how Theora works and how other codecs work. And since x264 is an open-source, _non-commercial_ encoder, there's no reason for him to pimp x264 for, say, purposes to boost sales of his product or something like that. People use x264 because Dark Shikari's improvements have made it such a good codec. Very knowledgeable people at Doom9.org, people who test codecs all the time. I've been following that forum for years, observing how codecs develop over the years and such. That's why I make the claims I make. I've also made test for myself, ripping my DVDs. Theora's picture was less sharp and sometimes more devoid of details than even Xvid, let alone x264.
I read those links you gave... Youtube's encoder for their "HQ" videos (their "HD" videos use a different encoder - x264 that is). Compared to those "HQ" Youtube videos Theora might be competitive, but that's that. Links to other examples... unfortunately the only good recent one I know of doesn't test Theora, just several h264 encoders, Xvid and Divx, VC-1, VP7 and mpeg2. So no links from me, just a clarification of where I get my knowledge from - Doom9.org
"Compared to those "HQ" Youtube videos Theora might be competitive, but that's that."
So for the vast majority of video Youtube serves, Theora's competitive? I'll take that.
"(no, I won't do that, I've no need because I do believe that IRC conversation)"
So the burden of proving your claims is on me? Nice try.
Speed can't be a reason, x264 is SIMD-optimized into oblivion, it's plenty fast. And since they use x264 for HD videos, they already have an infrastructure in place which accommodates x264. So why they use two different h264 encoders is beyond me.
"So for the vast majority of video Youtube serves, Theora's competitive? I'll take that."
For now, maybe. But Youtube won't stay as it is forever (already more and more videos are HD, thy might switch to a different encoder even for the HQ videos) and there are other sites that serve video too. All the while Theora will stay as it is, only getting minor improvements within what it's old design allows. For reference, Theora is based on VP3 and On2 has since released newer codecs, up to VP7. Though even VP7 is not as good as h264 from what I've seen (again referring here to test made at doom9).
"So the burden of proving your claims is on me? Nice try."
I provided an excerpt of a conversation. While I don't know these people personally, I've been reading their posts on doom9 (dark shikari) or their mails on the FFMpeg mailing list ("mru" from that conversation is Måns Rullgård, a FFMpeg developer) for years. Knowing that these people have been in the world of codecs for years is enough for me to accept their words and that conversation as accurate.
And I just remembered who BBB is - Ronald Bultje. He contributes to FFMpeg and used to work for Fluendo on Gstreamer (a multimedia framework for Linux). That Fluendo is in contact with the SFLC makes sense - a company that deals with codecs and open-source software will want to have the legal stuff in order.
You're absolutely right that the burden of proof should be on H.264 to prove it's much better than Theora, since H.264 is patent encumbered. And looking at Youtube, I don't believe H.264 can win the argument.
You have my commitment to use Theora! Firefox has already proven that proprietary marketshare can be overcome, let's do it again.
Thanks for taking the time to do a real comparison instead of just talk.
Unfortunately I can't make a visual comparison myself because Quicktime (up-to-date OSX 10.5) won't play your H.264; I get an alert "Incorrect duration".
I watched the first two minutes of the Theora version, and it looks pretty good to me, except that there are obvious artifacts in the sky in some sequences. Assuming H.264 gets rid of those, I still wouldn't say the difference would be "staggering" to a normal user like me. It may well be staggering to a video compression expert like yourself, but as I keep saying, most users don't look at video the way video compression experts do.
The question of why Youtube doesn't use higher quality is still very important. It's clear that they have important considerations that override a straight bit-rate/quality drag race, even if we don't know what they are (it may be more than just encoding performance). So a comparison with some arbitrary set of x264 settings is still somewhat meaningless. I wish we knew what those considerations are so we could ensure the Thusnelda test is a fair one, but at least we do know, so far, acceptable quality for Youtube is something Thusnelda can reach.
"I'd say these "Youtube's engineers" arrived at the conclusion that is cheaper to pay for a license than it is to pay for the extra bandwidth and disk space"
We don't know what Google's licensing arrangements are. It may well be that they already had licensing agreements in place so the incremental cost of licensing H.264 for Chrome was zero.
This time I tested it on Qt 7.6.2. In my defense the original does work with ffmpeg, mplayer, VLC, as well as Flash's, Windows 7's, and even Chrome's internal decoders.
I do believe there's a significant quality difference in overall sharpness and detail, not just the specific artifacts you mention. And I'm not "a video compression expert" :)
To make myself clear, I agree there's more to a codec than peak compression performance, and I agree that Theora might very well be "good enough" for many uses. What bothers me (and it seems it bothers other people as well) are claims like "Theora looks as good as h.264" (sometimes accompanied with arguably "rigged" tests) - that's unnecessary and will only attract flames.
> It goes even further - Xiph *knows* that there are patents in Theora owned by other companies (other
> than On2 that is), see here: >http://forum.doom9.org/showthread.php?p=1296267#post1296267
> BBB is, as is stated in that post, from the Software Freedom Law Center.
I find that hard to believe, as the SFLC is representing Xiph.Org on patent concerns, and I'm pretty sure if they knew we were infringing, they'd be obligated to inform us. To be clear, to date they have not.
...For the record, 'BBB's posting style smacks of DarkShikari under yet another. He's been spreading this bull for weeks now.
Monty
Director, Xiph.Org
We were comparing directly against YouTube h264 because of Chris DiBona's statement that going to Theora and maintaining the same quality as YouTube currently offers would destroy the entire Internet because it would require so much extra bandwidth. This test shows that statement to be utterly ridiculous.
The test wasn't rigged, it was a specific rebuttal. That got lost as it spread around.
Cheers,
Monty
No, you are not. I have the same experience and I have heard of many other users with the same problem. Especially for users of older (<~1.5GHz?) machines, videos of comparable visual quality start losing frames with Ogg Theory because of the CPU usage while Flash looks nice.