Archives for : ogg

An Encoder Is Not A State Machine

I’m pleasantly surprised that Google’s removal of H.264 from Chrome in favor of WebM has been greeted with widespread skepticism. You’d think that removing popular and important functionality from a shipping product would be met with scorn, but when Google wraps itself with the “open” buzzword, they often seem to get a pass.

Ars Technica’s Google’s dropping H.264 from Chrome a step backward for openness has been much cited as a strong argument against the move. It makes the important point that video codecs extend far beyond the web, and that H.264’s deep adoption in satellite, cable, physical media, and small devices make it clearly inextricable, no matter how popular WebM might get on the web (which, thusfar, is not much). It concludes that this move makes Flash more valuable and viable as a fallback position.

And while I agree with all of this, I still find that most of the discussion has been written from the software developer’s point of view. And that’s a huge mistake, because it overlooks the people who are actually using video codecs: content producers and distributors.

And have they been clamoring for a new codec? One that is more “open”? No, no they have not. As Streaming Media columnist Jan Ozer laments in Welcome to the Two-Codec World,

I also know that whatever leverage Google uses, they still haven’t created any positive reason to distribute video in WebM format. They haven’t created any new revenue opportunities, opened any new markets or increased the size of the pie. They’ve just made it more expensive to get your share, all in the highly ethereal pursuit of “open codec technologies.” So, if you do check your wallet, sometime soon, you’ll start to see less money in it, courtesy of Google.

I’m grateful that Ozer has called out the vapidity of WebM proponents gushing about the “openness” of the VP8 codec. It reminds me of John Gruber’s jab (regarding Android) that Google was “drunk on its own keyword”. What’s most atrocious to me about VP8 is that open-source has trumped clarity, implementability, and standardization. VP8 apparently only exists as a code-base, not as a technical standard that could, at least in theory, be re-implemented by a third party. As the much-cited first in-depth technical analysis of VP8 said:

The spec consists largely of C code copy-pasted from the VP8 source code — up to and including TODOs, “optimizations”, and even C-specific hacks, such as workarounds for the undefined behavior of signed right shift on negative numbers. In many places it is simply outright opaque. Copy-pasted C code is not a spec. I may have complained about the H.264 spec being overly verbose, but at least it’s precise. The VP8 spec, by comparison, is imprecise, unclear, and overly short, leaving many portions of the format very vaguely explained. Some parts even explicitly refuse to fully explain a particular feature, pointing to highly-optimized, nigh-impossible-to-understand reference code for an explanation. There’s no way in hell anyone could write a decoder solely with this spec alone.

Remember that even Microsoft’s VC-1 was presented and ratified as an actual SMPTE standard. One can also contrast the slop of code that is VP8 with the strategic designs of MPEG with all their codecs, standardizing decoding while permitting any encoder that produces a compliant stream that plays on the reference decoder.

This matters because of something that developers have a hard time grasping: an encoder is not a state machine. Meaning that there need not, and probably should not be, a single base encoder. An obvious example of this is the various use cases for video. A DVD or Blu-Ray disc is encoded once and played thousands or millions of times. In this scenario, it is perfectly acceptable for the encode process to require expensive hardware, a long encode time, a professional encoder, and so on, since those costs are easily recouped and are only needed once. By contrast, video used in a video-conferencing style application requires fairly modest hardware, real-time encoding, and can make few if any demands of the user. Under the MPEG-LA game-plan, the market optimizes for both of these use cases. But when there is no standard other than the code, it is highly unlikely that any implementations will vary much from that code.

Developers also don’t understand that professional encoding is something of an art, that codecs and different encoding software and hardware have distinct behaviors that can be mastered and exploited. In fact, early Blu-Ray discs were often authored with MPEG-2 rather than the more advanced H.264 and VC-1 because the encoders — both the devices and the people operating them — had deeper support for and a better understanding of MPEG-2. Assuming that VP8 is equivalent to H.264 on any technical basis overlooks these human factors, the idea that people now know how to get the most out of H.264, and have little reason to achieve a similar mastery of VP8.

Also, MPEG rightly boasts that ongoing encoder improvements over time allow for users to enjoy the same quality at progressively lower bitrates. It is not likely that VP8 can do the same, so while it may be competitive (at best) with H.264, it won’t necessarily stay that way.

Furthermore, is the MPEG-LA way really so bad? Here’s a line from a review in the Discrete Cosine blog of VP8 back when On2 was still trying to sell it as a commercial product:

On2 is advertising VP8 as an alternative to the mucky patent world of the MPEG licensing association, but that process isn’t nearly as difficult to traverse as they imply, and I doubt the costs to get a license for H.264 are significantly different than the costs to license VP8.

The great benefit of ISO standards like VC-1 and H.264 is that anyone can go get a reference encoder or reference decoder, with the full source code, and hack on their own product. When it times come to ship, they just send the MPEG-LA a dollar (or whatever) for each copy and everyone is happy.

It’s hard to understand what benefits the “openness” of VP8 will ever really provide. Even if it does end up being cheaper than licensing H.264 from MPEG-LA — and even if the licensing body would have demanded royalty payments had H.264 not been challenged by VP8 — proponents overlook the fact that the production and distribution of video is always an enormously expensive endeavor. 15 years ago, I was taught that “good video starts at a thousand dollars a minute”, and we’d expect the number is at least twice that today, just for a minimal level of technical competence. Given that, the costs of H.264 are a drop in the bucket, too small to seriously affect anyone’s behavior. And if not cost, then what else does “openness” deliver? Is there value in forking VP8, to create another even less compatible codec?

In the end, maybe what bugs me is the presumption that software developers like the brain trust at Google know what’s best for everyone else. But assuming that “open source” will be valuable to video professionals is like saying that the assembly line should be great for software development because it worked for Henry Ford.

Maybe they should call the next version of Ubuntu “Compulsive Coyote”

Fanaticism consists in redoubling your effort when you have forgotten your aim.
George Santayana, Life of Reason (1905) vol. 1, Introduction

A few weeks back, I asked the Twitterverse “When did user-facing Linux become completely and officially irrelevant? Seems like a fait accompli, doesn’t it?”. With the OSS community’s reaction to the iPad, I’m now sure of it.

And by reaction, I’m of course thinking of the response by the FSF’s John Sullivan to Steve Jobs’ much-discussed Thoughts on Flash. Sullivan’s piece is a straightforward “pox on both your houses” take, damning Apple and Adobe alike for not adopting the One True Faith of Open Source Software.

I had allowed myself to hope that the OSS community would take the hugely successful launch of the iPad and the flight of developers to the iPhone OS platform, even with Apple’s cravenly self-serving App Store policies, as an opportunity to take a good, long look in the mirror. With iPad sales now already estimated at over a million, you have to wonder how soon use of this nascent platform will exceed that of desktop Linux, if it hasn’t already. What should the open source community think of this profound preference for the unfree over the free? I would think they’d see it as a very public, and very obvious repudiation of the party line, suggestive of an urgent need for some soul-searching and time to re-examine how the OSS community is engaging the world at large.

Sullivan’s piece, sadly, shows no such self-awareness. It’s the usual total detach from real people that we’ve come to expect from True Believers. Indeed, the last third of it is the all-too-familiar zealous denunciation of all proprietary software, calling on readers to adopt Linux and Ogg Theora.

Seriously, Ogg? They’re still clinging to the “everything should be in Ogg” kick? OK, let’s ask the people who should know on this one, content professionals, to see if they buy in at all. DV magazine’s website logs 0 hits in searches for Theora and for Ogg,’s 11 hits (versus 248 for H.264) are largely related to codec politics and not mentions in training articles or product reviews, and legendary retailer B & H assumes the search term theora is a misspelling of “theory”. No matter how many essays and insults the zealots spew forth, they seem unable to generate any inkling to know or care about Theora among video professionals.

In effect, the FSF’s position is to tell the world “screw you for not wanting the stuff we produce”, oblivious to the fact that the marketplace of ideas is very clearly telling them “screw you for not producing the stuff we want.”

The quote at the top of this article was often cited by Chuck Jones in describing the character of Wile E. Coyote, who was never afforded a moment of clarity to reconsider his hunting regimen, the inherent dangers therein, and the prospects for a way of life that didn’t involve falling off cliffs every 30 seconds. I fear the OSS community is so wrapped up in their own self-righteousness, they are similarly immune to equally-needed moments of humility, clarity, and perspective.

Eek! A Patent!

Mike Shaver’s blog about Mozilla forgoing H.264 support in the HTML <video> tag is getting a lot of play, predictably from the /. crowd.

It’s a shame, because it’s a childish and facile argument. Here’s the gist of it.

For Mozilla, H.264 is not currently a suitable technology choice. In many countries, it is a patented technology, meaning that it is illegal to use without paying license fees to the MPEG-LA. Without such a license, it is not legal to use or distribute software that produces or consumes H.264-encoded content.

In short, the very idea that something is patented and requires licensing is prima facie proof that it is intolerable. Going on:

These license fees affect not only browser developers and distributors, but also represent a toll booth on anyone who wishes to produce video content. And if H.264 becomes an accepted part of the standardized web, those fees are a barrier to entry for developers of new browsers, those bringing the web to new devices or platforms, and those who would build tools to help content and application development.

Yeah, can you imagine if any other technology were encumbered by patents? They’d have to pass on the costs to customers too! Imagine if television were patented, or if automobiles involved 100,000 patents… surely those products could never exist and never be affordable.

There is a case to be made against patents and intellectual property as a whole, but this blog doesn’t make it. Instead, it blithely refuses to acknowledge that we do live in a world of IP, decrying its costs as if they are out of the ordinary or unjust. Ultimately, it flees back to the intellectual dead-end of “everything should be in Ogg”, a stance so untenable that even ESR conceded it was a non-starter, four years ago.

A final irony: by refusing to support H.264, Mozilla bolsters the primary alternative for video on the web: the Flash plug-in, which is not just patent-encumbered but proprietary, available only from Adobe and only in those environments where it serves the company’s strategic ends. Shaver, to be fair, admits that proprietary plug-ins are bad too, but declines to say they’re far more bad than patent-encumbered standards. Instead, he holds out for a pollyannaish vision of completely free web video technology, naming Ogg as his moral standard-bearer, without acknowledging Ogg’s lamentable but obvious feet of clay.

Ogg: The “Intelligent Design” of digital media

Well, another go ’round with this: HTML5 won’t mandate Ogg as universally-supported codecs, and the freetards are on a tear. I was going to follow up on a JavaPosse thread about this, but I hurled enough abuse onto their list last week.

It’s abundantly clear in this blog that I don’t think Ogg is the solution that its supporters want it to be: I have a whole tag for all the posts where I dismiss Vorbis, Theora, and friends. Among these reasons:

  • I don’t think it’s technically competitive.

  • It certainly isn’t competitive in terms of expertise and mindshare, which is vitally important in media codecs: there’s a much deeper pool of shared knowledge about the MPEG codecs, which leads to chip-level support, competition among encoders, compressionists who understand the formats and how to get the most out of them, etc.

  • Its IP status remains unclear. With even MPEG-4, following a lengthy and formal patent pooling process, attacked by AT&T’s claim of a submarine patent, I have no reason to think that Ogg wouldn’t face similar claims, legitimate or not, if there was any money behind it, which there isn’t.

  • If I go to my former colleagues at CNN or in Hollywood and say “you guys should use Ogg because…”, there are no words in the English language that plausibly complete the sentence and appeal to the rational self-interest of the other party.

On this last point, I’ve got an ugly analogy: just as proponents of “Intelligent Design” are people who don’t really care about biology beyond the point at which it intrudes on their religious belief, so too do I think Ogg advocates generally don’t know much about media, but have become interested because the success of patent-encumbered formats and codecs is an affront to their open-source religion.

Ogg’s value is in its compatibility with the open source religion. It has little to offer beyond that, so it’s no surprise that it has zero traction outside of the Linux zealot community. Even ESR realized that continually shouting “everything should be in Ogg” was a losing strategy, and he said that three years ago.

I think the open source community would like to use HTML5 to force Ogg on the web community, but it’s not going to work. As others have pointed out, there’s little reason to think that IE will ever support HTML5. Even if they do, the <video> tag is not going to replace Flash or Silverlight plug-ins for video. Despite my initial enthusiasm for the <video> tag commoditizing video, I see nothing in the spec that would support DRM, and it’s hard to imagine Big Content putting their stuff on web pages without DRM anytime soon. And while you can put multiple media files in a <video> tag easily enough, having to encode/transcode to multiple formats is one reason that Big Content moved away from the Real/WMP/QuickTime switch to the relative simplicity of works-for-everyone Flash.

I’m tired of being lectured by computer people about media; it’s as ludicrous as being lectured about computers by my old boss at Headlines. Just because you use YouTube, doesn’t make you an expert, any more than my knowing how to use a username and password means I understand security (seriously, I don’t, and doubt I ever will). Kirill Grouchnikov pretty much nailed what computer people think good video is with this tweet. I’ll add this: there are probably a thousand people at Sun who understand 3D transformations in OpenGL, and maybe five who know what an Edit Decision List is. So they go with what they know.

A couple years back, I gave a JavaOne BoF in which I made a renewed call for a Java Media library which would support sample-level access and some level of editing, arguing that enabling users to take control of their own media was a manifest requirement of future media frameworks. By a show of hands, most of the developers in the audience thought it would be “enough” to just support playback of some modern codecs. JavaFX now provides exactly that. Happy now?

People who actually work in media don’t mind paying for stuff, and don’t mind not owning/sharing the IP. Video production professionals are so accustomed to standardizing on commercial products, many of them become generic nouns in industry jargon: “chyron” for character generators, “grass valley” for switchers, “teleprompters”, “betacam” tape, etc. Non-free is not a problem here. And if your argument for open-source is “you’re free to fix it if it doesn’t do what you want it to,” the person who has 48 shows a day to produce is going to rightly ask “why would I use something that doesn’t work right on day one?”

The open source community doesn’t get media. Moreover, it doesn’t get that it doesn’t get media. The Ogg codecs placate the true believers, and that’s the extent of their value.

Commoditizing embedded video: the HTML5 video tag

Surfin’ Safari notes initial support for the HTML5 <video> and <audio> tags in their latest nightly builds.

Indeed, if you have a browser that supports the video tag, then you (hopefully) can see an autoplaying video here:

There’s been a little bit of controversy over the fact that calls for inclusion of the Ogg formats have been removed in more recent versions of the spec. Section, “Video and audio codecs for video elements”, currently reads:

It would be helpful for interoperability if all browsers could support the same codecs. However, there are no known codecs that satisfy all the current players: we need a codec that is known to not require per-unit or per-distributor licensing, that is compatible with the open source development model, that is of sufficient quality as to be usable, and that is not an additional submarine patent risk for large companies. This is an ongoing issue and this section will be updated once more information is available.

That more or less matches my take on Ogg, which is that it poses an unknown patent liability risk: the /. mob insists it’s patent-free, but how the hell do they know? They don’t; they just want it to be so, because it suits their worldview. And Ogg may indeed be patent-free, but I don’t think anybody knows for sure, and even so, proving it would be expensive. To top it all off, Ogg just isn’t that popular or useful outside the warm bubble of Linux zealotry.

Still, there’s a huge need for at least one video and audio codec to be available more less everywhere, or at least for one class of devices: i.e., one codec you can expect all desktops to have, one for all phones, etc. To just dump out to “whatever QuickTime supports on the Mac, whatever Windows Media supports on Windows, etc.” ends up moving the problem, either to the web author (who has to sniff the OS from the user-agent and write the tag on the fly… to say nothing of hosting multiple encodings of every clip) or to the end-user.

It’s funny, because while the HTML5 <video> tag should displace Flash as the only practical option for web video — something that’s become screamingly obvious in the two years or so since YouTube launched — it might not, if it gets tangled up in codec hassles. The remarkable thing about Flash Video isn’t that it’s good (it’s not), but that it’s consistent and available on all Flash-enabled desktops.

That’s turned out to be a much bigger deal than the quality of competitors like H.264 and WMV, or the fact that other approaches could support many more codecs. Flash doesn’t try to support every codec under the sun, or even offer extension points for third parties to do so, but it doesn’t matter — with a known-viable video codec, content providers can just push their content with the package-deal of FLV and the Flash plug-in. Sure, the QuickTime plugin, a QuickTime for Java applet (or even a JMF applet, fercryinoutloud) could support more formats and codecs than Flash, but the typical use is not a general-purpose “play arbitrary content” application; the web-embedded player is usually meant to play the content from a single content provider, who’s perfectly happy to use a single, sub-optimal format, if the alternative is having to encode everything a dozen ways from Sunday to support the various OSes and devices.

Which makes me think that Flash’s ubiquity as a web-embedded video player won’t be threatened by HTML5, so long as there is neither a de jure nor de facto ubiquitous video codec for HTML5. Ironically, while H.264 might be the best candidate for that, Flash is already supporting it too.

Which leads me to an idea: if you were writing a browser on a platform without H.264 support from the native multimedia library, but you had Flash available, could you just pull the Flash player into service on the fly and have it play the H.264?

Things left unsaid

A couple of interesting comments from the first half day of Mobile & Embedded Developer Days:

  • Two speakers were hesitant to even use the word “android” at this Sun-hosted event, demurring with apologies like “I know that’s a dirty word around here”. Here’s the thing: both of these speakers were with outside companies. None of the Sun speakers have mentioned Android at all. All of which leads me to think the enmity towards Android among Java’s old guard is even worse than is generally known.

  • All the Sun people I’ve talked with about Java Media… and remember, I’m trying to retire from this topic… have referred over and over again to what they call “the codec problem.” Remember, I think the issue is one of capability, but we’ve been over this (the Java types generally think playback-only is “good enough”), but that’s neither here nor there. People I know at Sun, even those in no way associated with Java Media, talk about “the codec problem” or “the codec question”, implying that what Sun is wrestling with is what video (and audio?) codecs to license.

    So James Gosling gave a brief keynote and brought up a slide on the Java FX platform. One line item was a pair of key features: media and a scene graph. He noted that the scene graph is looking great (and it is; it’s already open-sourced on, but on media, he made only a very fleeting reference to “dealing with the nightmare of MPEG-LA”. Which tells us two interesting things:

    1. They’re looking at some MPEG-LA-licensed codec, probably H.264, and
    2. it’s not going well

    Is it safe to assume that the price is too dear for Sun? Maybe. Sun did pull MP3 out of the Java Media Framework years ago when the license proved unaffordable (it came back later, but now playback-only). Still, all the other companies that are serious about media — Apple, Sony, and now Adobe — have decided it was worth it.

    What about the alternatives? You have to wonder if they’re at least thinking about VC-1, especially given the new cozy relationship with Microsoft. The hoi polloi will probably demand Ogg, though I think that would be crazy.

Why not Ogg?

I was just in a chat on the #javaposse IRC channel, and I mistakenly walked into the whole Java Media topic, of which I have said far too much already (1, 2, 3, 4, 5, 6), and which I’m really not interested in discussing further, as the Java Community has given my basic argument — that Web 2.0 will call for a read/write, cross-platform, web-embeddable media library, and if Java doesn’t step up, then it will be Flash — a fair consideration, and has rejected it, saying that “all most people need is playback.”

Fine, I say, but all the consulting work I’ve taken on in the last few years has involved editing, capture, or both, so if Java Media’s big comeback is going to be playback only, then I need to do something else, either single-platform (QuickTime), or Flash.  So, see ya.

And at this point, the playback-only crowd gets into the whole “how do we support all the patent-encumbered codecs out there” snit — overlooking the fact that Flash succeeded with a small number of not-particularly-good codecs — and as software developers, they all decide that what they really need to support is Ogg Vorbis and Ogg Theora.  In fact, the feature request  to add the Ogg codecs to the long-abandoned Java Media Framework is consistently in the top 5 Java RFEs.

And it never seems to occur to anyone that while everything else Sun does with client-side/user-facing Java is a “me too” effort, that nobody else in the industry seems to have embraced Ogg in a big bet-the-company kind of way.  Despite its lack of patent encumbrances and its open-source implementation.

Let me posit that these traits are not features, they’re actually bugs.

Any company that embraced Ogg and put dollars behind it (or used it in products or services that involved serious dollars) would need to do some due diligence to prove to itself that Ogg is, indeed, not infringing on any known patents.  Given Ogg’s nature as an open-source project, this would likely be an expensive and tedious process, if even practical at all.  But failing to do such an examination would be fiscally irresponsible, as in “could get sued for not doing it” irresponsible.  Because in the end, all you’ve got is the community’s assurances that there is no patent-encumbered IP in the code base.  Are they telling the truth?  Can they even know what’s in there and where it came from?  A proper examination would demand proof, and that would be tedious and expensive.

And, even if Sun (or whoever) did such an exam, there’s no preventing some patent troll from going after Ogg anyways if it gets attached to a successful product or service.  So many of the media codecs are so similar — the various DCT-based codecs share common ancestors and many of the same ideas (page 77 of Damien Stolarz’ Mastering Internet Video has a remarkable “family tree” of video codecs) — that it’s easy to imagine a sleazy legal team convincing a jury in Marshall, Texas that those similarities are prima facie proof of infringement.

And anyone fighting such a suit over Ogg stands alone.  Sign up for VC1 and you get Microsoft in your corner, with more money than god.  Even the H.264 camp, currently implicated in a presumably-bogus suit from SBC AT&T, has deep-pocketed partners who would fight an infringement finding.  With Flash adopting H.264, and YouTube being bought by Google, it’s likely that Google ultimately has a dog in this fight.

Ogg doesn’t magically solve the codec problem.  And it’s the wrong problem to solve  anyways, since playback alone won’t be interesting by the time Sun can bring a product to market.