Rss

AV WWDC, part 1: Hot Dog… The AVMovie

I attended WWDC for the first time since 2011, thanks largely to the fact that working for Rev means I need to go out to the office in San Francisco every 6 weeks anyways, so why not make it that week and put my name in the ticket lottery. I probably won’t make a habit of returning to WWDC, and the short supply of tickets makes that a given anyways, but it was nice to be back just this once.

Being there for work, my first priority was making use of unique-to-attendee resources, like the one-on-one UI design reviews and the developers in the labs. The latter can be hit-or-miss based on your problem… we didn’t get any silver bullet for our graphics code, but scored a crucial answer in Core Audio. We’ve found we have to fall back to the software encoder because the hardware encoder (kAppleHardwareAudioCodecManufacturer) would cause ExtAudioFileWrite() to sometimes fail with OSStatus -66570 (kExtAudioFileError_AsyncWriteBufferOverflow). So I asked about that and was told “oh yeah, we don’t support hardware encoding anymore… the new devices don’t need it and the property is just ignored”. I Slacked this to my boss and his reaction was “would be nice if that were in the documentation!” True enough, but at least that’s one wall we can stop banging our head against.

Speaking of media, now that everyone’s had their fill of “Crusty” and the Protocol-Oriented Programming session, I’m going to post a few blogs about media-related sessions.

QuickTime’s Revenge

Warm fuzzy session for me was the inauspiciously-titled Editing Movies in AV Foundation, which I figured would be another run through the AV Foundation editing API we’ve been working with since iOS 4.

Much to my surprise, the speaker was Tim Monroe, whose appearance allayed my fears immediately. History lesson! For many years, Tim wrote the QuickTime Toolkit column in MacTech magazine, continuing a running joke of using movie titles for all his article titles (eg, Honey I Shrunk the Kids for an article on QuickTime’s image compression APIs), and epically trolling readers with an April Fool’s article claiming that QuickTime had been ported to the black-and-white iPods, in the long-obsolete SNOBOL programming language. Tim’s articles were collected into the books QuickTime Toolkit, Volumes 1 and 2, though in a case of Apple being Apple, the books couldn’t be credited to an employee and instead claim to be by “Apple”.

   

So, roll back two years and you can see me bitching about AV Foundation’s editing APIs. In AV Foundation, there’s a movie-in-memory concept called an AVComposition which can be built up from parts of other movies, either at the composition level (which will keep all audio and video tracks from source to target) or down at the individual track level. It misses a lot of the concepts of the old QuickTime APIs, such as reference movies, editing state (for undo/redo), serializability, and pasteboard access. Probably the biggest difference is that QuickTime was tightly coupled to the QuickTime file format, while AV Foundation was more agnostic.

Problem is, AV Foundation has never really supported anything other than QuickTime and its direct descendants (the various flavors of MPEG-4: .mp4, .aac, etc.), so this isn’t much of a architectural advantage. Certainly not enough to outweigh the fact that AV Foundation still can’t do things that QuickTime could do 15 years ago, like copy and pasting regions of movies, or creating reference movies.

Selecting and copying a time range… don't try this in QuickTime Player X or AV Foundation!

Selecting and copying a time range… don’t try this in QuickTime Player X or AV Foundation!

Enter Tim, and the new AVMovie. This is a new editing API, an alternative to the AVComposition stuff, and is much more strongly influenced by the old QuickTime APIs (and, in turn, by the QuickTime file format itself). This brings back some of the important concepts of the old QuickTime API, such as modifier tracks: tracks that consist only of instructions to modify other tracks at certain times. AVComposition offered some support for this with opacity ramps and affine transforms (in the hideously-named AVMutableVideoCompositionInstruction class) but QuickTime was always able to offer a much broader range of functionality, and could save those instructions in the file itself until they were needed for a final export. In AVComposition, your instructions are memory-only, so you’d need to roll your own scheme for persisting editing state. (How the hell does Final Cut Pro X do it anyways?)

And hey, copy-and-paste, amirite? The AVMovie(data:) initializer’s got your back to create a movie from pasteboard data. It looks from the headers that a data property goes the other way, but I’m not sure if that would work for getting an edited movie (or one built up of inserted segments) to the pasteboard.

And yes, I did say “headers”. The new QuickTime-inspired editing APIs aren’t in the OS X 10.11 docset yet, and they don’t exist at all on iOS.

I’m certainly happy to see this new approach to editing video, although it’s going to be hard to clarify when to use AVMovie versus AVComposition. For example, doing compositing work with affine transforms and opacity, or pulling in Core Animation graphics, may work better with the AVComopsition APIs, at least until AVMovie is built out and understood. And for the hardcore, there’s always the low-level approach of doing your own rendering, perhaps with Core Image, and writing samples with AVAssetWriter.

This also speaks to the fact that AV Foundation is now most certainly way the hell too big to make any sense of. There are about 150 classes even before we add in the new AVMovie stuff. And there are basically four distinct sub-frameworks in this container, which have little if any direct interaction:

  • Core Audio wrapper/replacement: AVAudioRecorder and the newer AVAudioEngine
  • Media capture: AVCaptureSession and friends
  • Media playback and editing: AVPlayer, AVAsset and its many subclasses and helpers, like AVComposition
  • QuickTime Reborn: AVMovie, etc.

I wonder about the internal design and architectural politics that led to this about-face on AV Foundation editing, after holding to the company line for so long on the AVComposition stuff. Was the idea of a reference movie just incompatible with the design they had, or did the old hands just hang on long enough to bring in their bits as an alternative? Did all the angst over Final Cut Pro X, or bitching from people on the outside have anything to do with it? Is it related to Randy Ubillos’ retirement? It’s a little unseemly to be second-guessing office politics like this, but given Apple’s opacity, it might be our best chance for understanding how the OS X and iOS media APIs are evolving.

Comments (2)

  1. galad

    QuickTime had a big issue: it was to dependent on the mov file format. Everything was fine, until you had to deal with a non mov file. You would have to create a in memory mov representation of the file, and for some file format without a complete index (for example mpeg-2 ts o ps, mp3, and even mkv) that would require an expensive operation in which the whole file had to be scanned; all that even if you wanted just to playback the file.
    AVFoundation fixes this, by having a file format independent api and a modern pipeline (no more “enterMainThread” or even more hideous threading functions, or performance bottleneck when exporting a h.264 video).

    AVMovie main reason to be is to expose all the quicktime file format specifics features that were lost in the AVAsset abstraction, heck, finally an API I can use to change the language of a track without having to export the whole file again 🙂

    Now all that is missing under the AVFoundation umbrella is a pubic api for third party plug ins for video codes and media importers.

  2. […] capture, the playback/editing classes, audio engine and units). And with the addition of the pseudo-QuickTime AVMovie, it feels like they’re not done yet. It seems like AV Foundation requires rolling with […]

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.