Begun, The Codec War Has

I just got back last night from WWDC 2017 week — I was speaking at CocoaConf Next Door — and here’s an obligatory selfie to prove it.

Selfie outside WWDC 2017

Where Is Everybody?

So let’s start off by talking about the conferences themselves. I wasn’t inside WWDC, but everyone sounds pretty happy about its contents and how it was conducted, following the big move from San Francisco to San Jose. Outside the conference, I think it was a bit of a different story. When the move was announced there were complaints about San Jose being a “nothing to see, nothing to do” town compared to world-famous San Francisco, where the most noteworthy site was the “San Jose Drainage Ditch” that became something of a running joke on the Core Intuition podcast. Ironically, the meetup on Tuesday was held next to the famous Ditch, which can be seen in the back of this photo: meetup, with San Jose Drainage Ditch in the background

We joke about this, but I think the digging on SJ had a genuine sting. While Apple generously promoted companion events CocoaConf, AltConf, and Layers, word is that these events suffered from lower-than-expected attendance. CocoaConf clearly had a much smaller attendance than their sold-out Chicago event a few months before, and AltConf tweeted photos of sessions that looked desperately underpopulated. The James Dempsey & The Breakpoints concert was also at like 65% capacity when I checked on Monday, although they did eventually reach their fundraising goal for App Camp for Girls. While other events like the live recordings of The Talk Show and Accidental Tech Podcast did sell out almost immediately, anecdotal evidence suggests that the “hang out even if you don’t have a golden ticket” crowd largely took a pass on Dub Dub week this year. And that’s a shame.

San Jose Civic entrance, hosting James Dempsey concert

View of 1st St from VTA stand Personally, I liked the locale. It was easy to get around (I was at a hotel by the airport and just took light rail to the convention center every morning), and enjoyed having a new town to stomp around. There were plenty of restaurants and shopping within a 10-minute walk, and the greenery was a nice change from San Francisco’s concrete jungle. If anything, I’d come out a day earlier next time: The Camera 3 Cinema two blocks from the convention center does Rocky Horror with a live cast the first Saturday of every month, which should consistently line up with WWDC. Maybe we could redo the lyrics of “Eddie’s Teddy” to be about Eddy Cue or something.

But make no mistake, San Jose is no Las Vegas. I heard of people going to restaurants that were supposed to be open until midnight but were instead rolling up their storefronts at 10:30. There’s also a possibly-apocryphal anecdote about a group headed up by John Gruber and Marco Arment pleading with a bar to stay open, promising they’d make it well worth the proprietor’s while to do so, but to no avail. Which is kind of weird — with a convention center that big, surely WWDC is not the only big event with thousands of attendees that would want to hit the town at night. FanimeCon had brought 25,000 otaku to the McEnery Convention Center just a week earlier (some of their banners were still up), although cosplay kids typically don’t have nearly the nightlife budget that techies do.

Anyways, from my POV, the move to SJ went well, and I have to imagine this relocation will be more or less permanent, since it’s surely far more convenient for Apple and its local employees. Hopefully, the developer community will be more receptive of the city in 2018, and will support events like AltConf, CocoaConf, and Layers. (Also, those of us participating in those conferences have a role to play in this — I bet it didn’t help that 3/4 of CocoaConf’s sessions were listed as “TBA” when the Early Bird deadline passed. How do you get your boss to expense you a trip to a conference when you can’t really show them what you’re going to learn there?)

265 Or Fight!

So, to keep things fresh for attendees, I told CocoaConf I’d do a talk on Thursday covering any interesting media bits that got publicly announced earlier in the week at WWDC.

Title slide from CocoaConf Next Door 2017 media APIs talk

Lucky for me, it was a big news year on that front, with the announcement of support for H.265, aka HEVC (high efficiency video coding). Dare we say, “finally”?

H.265 promises data rate savings of up to 50% over the ubiquitous H.264, meaning that your video can look much better with the same bandwidth, or can use half as much. That hasn’t been a super big deal to consumers yet (providers are another story, which we’ll get to), but if 4K video is going to be a thing, then we need to get the data rates down to the 10-15 Mbps range, which H.264 can’t accomplish with images that big (more details in this Streaming Media article from 2015).

Interesting, AppleInsider reported in 2014 that Apple was using HEVC for FaceTime over cellular on the iPhone 6 and 6 Plus, which lines up with the devices announced to have hardware support for HEVC. So what’s taken so long to get it into AV Foundation and Video Toolbox for the rest of us? Quite probably licensing.

So, funny story: I first met Daniel Steinberg at a Mac OS X conference O’Reilly used to run, and one year Apple attended and showed off their new H.264 codec for QuickTime, with a really pretty encode of the trailer for the Phantom of the Opera movie. Thing is, they couldn’t release the codec, because they thought the licensing terms for H.264 were too onerous, so they were holding out for a better deal, which they eventually got.

In 2017, the licensing situation is far worse. HEVC contains hundreds (probably thousands, but I couldn’t verify) of patented technologies. This has long been the case with the MPEG codecs, and the way they deal with this is with patent pools: rather than having to negotiate licenses with every company that has IP in the standard, “patent pools” like MPEG-LA offer a one-stop shop for paying all the needed licenses in one place. The problem is, some patent holders aren’t in the MPEG-LA pool. Some are in a rival pool, HEVC Advance. And then a third one recently popped up. And Technicolor SA has HEVC patents that aren’t in any of the pools, so everything still sucks. And the cherry on top is that the license demands are pricier than H.264, hitting makers of both encoders and decoders, as well as content providers. It’s pretty much a dumpster fire, and one reason that Netflix has stayed away from HEVC thusfar, according to a Streaming Media NAB article:

I asked about the future of HEVC given the imminent release of AV1, and the fact that Netflix was a founding member of the Alliance for Open Media. Ronca laughed, explaining that Netflix was still encoding in VC1 format for some older platforms, and that HEVC is important for their UHD/HDR experience, though he noted that the HEVC license uncertainty is an ongoing concern.

That quote mentions “AV1”, which brings us to the other side of the coin. H.265 isn’t the only new codec in town. In 2010, Google acquired video codec maker On2, and has used their IP and expertise to develop their own video codecs. The current offering is VP9, which is roughly the technical equal of HEVC. Unlike HEVC, VP9 is largely unencumbered by patents, with the only patented technologies being owned by Google itself, free use of which is granted by Google. Its successor, AV1, is expected later this year from the Alliance for Open Media, a group founded by Amazon, Cisco, Google, Intel, Microsoft, Mozilla, and Netflix (and yes, it would be easier to just say “everybody but Apple”).

VP9, the current Google-backed codec, is supported by basically every browser other than Safari and Internet Explorer, and is already widely used by YouTube where available. Just this week, DirectTV Now dropped support for Safari (and IE) in favor of Chrome. Conspicuous pair of browsers to drop, right?

This feels like it could be the end of 10+ years of codec peace, and a return to the bad old days of the late 90s when internet media insisted users install plugins for Windows Media, RealPlayer, QuickTime Streaming, or Flash, based on the technology choices made by the content provider. If the only way to have One True Codec today would be to standardize on VP9, well, let’s face it: there is no way in hell Apple would ever allow a core part of its business depend on a Google-controlled technology. (And hey, this is a two-way street: one reason we have the MPEG-DASH streaming container standard is that other companies weren’t comfortable with Apple calling the shots on HLS).

Apple may have had to get behind H.265 at this point, patent problems and all, or risk VP9 developing an insurmountable lead. iOS is big enough to make a codec happen, and drag Mac and (presumably) Apple TV along for the ride. If that means that content providers have to encode and host all their stuff twice — once for the Apple world and another for everyone else — oh well, too bad so sad. The video industry is quite used to problems that are solved by throwing money at them.

There’s just one thing here that bugs me, and that I think will become really ugly someday. That bit about how VP9’s (and AV1’s) only patented technologies are provided by Google. Doesn’t that sound too good to be true? How is it that H.265 needed patented technologies from all over the place, and the equally-good VP9 didn’t? Did MPEG do that crappy a job with 265, needlessly burdening their standard with patent-encumbered tech when they didn’t have to?

Or… does VP9 actually infringe on a bunch of patents? I have no idea, and it’s never been tested, but this is the thing lots of people in the business are afraid of. On its face, since the codec is open, it should be straightforward for any owner of patented tech to find infringement and speak up… unless they’re submarining their patent claims and waiting for VP9 to become a jucier target. And even if it doesn’t infringe any patents, the US patent system is so bad, it’s not a stretch to imagine an East Texas jury being convinced it does. You know how well they handle technical nuance, right? Imagine the scene in the courtroom:

This is Chewbacca. Chewbacca is the Discrete Cosine Transform. Now think about it… this makes no sense! And if it does not make sense, then you have to find for my client!

So what happens? Maybe this does turn into a legal fight at some point. That’s one good reason to side with HEVC: everyone who’s paid into that patent pool is motivated to protect their investment, so which standard you choose could ultimately depend on which side can afford the most lawyers. Historically, the MPEG standards like HEVC have enjoyed support from the traditional video industry (MPEG-2 was rarely used on the internet, but is the standard used by DVD and for the earlier digital cable and satellite television systems), so that was a good group to be in. But streaming is cutting into the old industries, and an alliance of Amazon, Google, and Netflix (among others) is not to be taken lightly. Long story short: this could get incredibly ugly. And expensive.

Or maybe not: after all, Apple has practically nothing to show for its long legal slog with Samsung.

But even if it doesn’t end up in the courtroom, competing video standards is going to suck. With 264, human compressionists have been able to tune their skills to the specific quirks of that codec, filtering video so it still looks good after compression (eg, tweaking color curves to avoid crushing the blacks). Now they’ll have to master two codecs, or maybe one or both won’t get that same level of TLC. The upside will be if there’s genuine competition, maybe we’ll see more innovation and better video, but that was already the case: MPEG’s policy of standardizing decoding but not encoding was meant to create an incentive for competition among encoders to get the best picture in the fewest bits (or the fastest encode).

Anyways, get your popcorn; I can’t imagine a full-on standards war over 4K video won’t be at least a little entertaining.

Comments (2)

  1. The low “WWDC fringe confs” attendance could well be related to a combination of SJ being less of a tourist destination than SF as well as the Trump-driven immigration problems. If you’re outside the USA and looking to spend ~$5000 to get and stay there and party for the week, it being in SJ instead and also the threat that you may not even be allowed in (and/or not be able to use any laptops etc. on the very long flight) could have been a big factor.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.