Rss

Archives for : itunes

Click Here to Fund this Blog

Don’t mind me, just testing links from the new iTunes Affiliate Program.

Interestingly, it seems you can’t create affil links to Apple apps (Mac or iOS) anymore. But the format is so simple (append at= and your affil ID to the usual iTunes URL), that it’s easy to do by hand. Like this: Final Cut Pro X. Question is, does it count?

Practical upshot of these programs: I get a 7% cut of anything you buy on iTunes for 24 hours after you follow one of these links. Doesn’t change your price; it’s just a payment from Apple to me for sending them traffic. And it’s never very much money (average $20 a month), but infinitely more than I made with a brief Google AdSense experiment (never got to $1, and the Apple and Amazon affil links are way less intrustive than banners), and it’s also a little more than I make off my old apps in the App Store.

EDIT: one huge improvement I forgot to mention is that the URLs no longer begin with click.linksynergy.com… which scared off some users who moused over them and rightly worried where they’d be taken. Since the URLs are now all itunes.apple.com with just an additional key-value pair in the parameters, users should feel more comfortable clicking them. They’re probably a lot more Twitter-friendly in this format too.

Seriously, would it be so hard to ship a new Mac Pro?

Can’t be a WWDC prediction because Apple would never expend precious keynote time on it, but why oh why can’t we have a new Mac Pro? Seriously, it hasn’t been updated in nearly two years, and if sales are poor, that’s probably a reflection of selling 2010 technology at 2012 prices.

I don’t see either the Mini or the iMac as a suitable replacement for the Mac Pro. For me, there’s a specific issue of drive technology. I put an SSD in my Pro as the boot drive, which has given me much needed relief from the 10-minute disk-thrashing festival that is Lion startup (#lionsucks), and provides a substantial (if not extraordinary) improvement for large Xcode builds. But I have traditional platters in the other bays to handle enormous amounts of media: my 1,000-album iTunes library, all my DVD rips, Final Cut and Soundtrack projects, etc.

SSDs are a terrible choice for big media files: the files are large, seldom accessed, and are read sequentially, meaning they gain nothing from the fast-access traits of flash memory, and certainly don’t justify the price. It’s gruesome to think of the idea of someone sync’ing their iPad to a MacBook Air and burning up a big chunk of the laptop’s storage with backups of the apps… a problem I didn’t think about until my kids’ iPads exhausted the puny 60 GB internal drive in the Mac Mini the family shares.

On the Mini, I have a 1 TB external drive that houses iTunes libraries (and now hosts the various “Mobile Applications” directories, which insist on living at ~/Music/Mobile Applications, but can be moved to another volume with a Unix simlink), my Subversion repository, old websites, etc. It’s not bad, but it’s a little kludgy to have external drives sprawling all over the desk.

Well, in a no-Pro future, we’re all going to have to make that choice: either use a traditional platter in the Mini or iMac and suffer the slowness, or boot on an SSD and then waste half its capacity or connect a bunch of external drives to hold all our media and iOS backups. Alex Lindsay keeps saying on MacBreak Weekly that Apple sees Thunderbolt as obviating most of the need for Mac Pros — Thunderbolt’s bandwidth supporting external storage, multiple displays, video input, etc. — but a sprawl of daisy-chained devices all over the place seems like a return to the bad old 80’s (check out what happens when you connect all the available peripherals to a TI-99). Furthermore, third-party Thunderbolt adoption has been disappointing, and it’s hardly unfair to say it’s just FireWire all over again.

Perhaps the other story is that Apple expects iCloud to serve our long-term storage needs for things like iOS device backups and media storage. I admit I haven’t given iCloud much of a chance — I activated the first beta during WWDC 2011 and soon had three copies of all my calendar events and contacts, so I’ve been slow to trust it with my data again. So, maybe this is their long-term answer.

But in the here and now, when I’m at my desktop, I want 3.5″ drive bays and monstrous CPU and GPU capacity. Screw the MacBook Pro — the iPad has all but obviated laptops for me — I want the pro Pro back.

Instant-start music with HTTP Live Streaming?

Up today on Mac Rumors is the article Apple’s Music Streaming Service to Eliminate Buffering Lag With Locally-Stored Snippets?, which details a patent application for a streaming music locker service that makes music instantly playable, with no connection-time delay, by cacheing the first few seconds of the audio locally, and playing that while the network connection is negotiated and the buffer fills. From their summary:

In essence, local storage requirements can be significantly reduced by keeping only a few seconds’ worth of each audio or video file on the device itself. That content could begin playing immediately upon selection by the user as the remainder of the requested file is streamed in the background and seamlessly appended onto the locally-stored portion. The solution seeks to eliminate lag that can be generated by a streaming-only solution as the device builds an initial buffer of content to ensure smooth playback.

There’s a potential hassle here in perfectly timing the handoff from local file-based playback to getting the audio from the network stream (though of course a local buffer like an Audio Queue would help).

But it gets even easier. Assume your stream protocol is HTTP Live Streaming. Then you’re not opening and holding a socket connection, you’re just pulling down a series of 10-second segments of audio, and your cached audio is just the first n segments. You also don’t have to negotiate performance right off the bat, and instead just let HLS adapt during its refreshes later in the playback. And you don’t have to expect to hold the connection consistently, it just has to be available often enough for the .m3u8 refreshes and fetches of later segments to work.

Was not really paying attention to this music locker stuff before, but this idea sounds pretty clever.

Choose These Adventures

I’m planning to spend part of the holidays doing some serious reading on the iPad, but not with iBooks, Kindle, Stanza or the like. Let me explain.

Visual Novels are a electronic narrative format from Japan that, for reasons I’ll get into in a bit, are appearing in greater frequency on iOS devices. Visual novels are a first-person narrative with the reader experiencing the story first-hand, and which the text of a story atop digital artwork showing the settings and active characters (who may or may not also have their spoken lines delivered as digital audio). In some cases, the reader makes a limited number of choices, branching the story in a style similar to the “Choose Your Own Adventure” series. In the Japanese taxonomy, introduction of any interactive elements makes a story an “adventure game”, but in the West, the term “visual novel” is preferred for these cases with a small number of interactions, to contrast with the tradition Western adventure games that require constant interaction, puzzles, inventory, etc.

The form has never really caught on in the West, initially for lack of access. Unlike game genres like racing and fighting that can be fairly well enjoyed by importers who can’t read the small amount of Japanese text in the menus, visual novels are all about the text, and therefore aren’t import-friendly and instead present a massive localization challenge. Since every possible branch through the story will have different text, the complete text of a visual novel is potentially much longer than a typical book: this thread shows some typical visual novels with a word count exceeding that of The Lord of the Rings, with longer works being several times that. Of course, this means that paying by the word for localization becomes enormously expensive, prohibitively so for a niche genre that few in the West even know about.

Still, a few companies have tried to make a go of it in Western markets. A company called Hirameki International localized several visual novels that were released as standard DVDs, playable/readable on any DVD player. It didn’t pan out and the company ceased releases in 2008. A few other companies translate adults-only erotic titles (“eroge”, a contraction of “erotic game”) for PC; these include Peach Princess and JAST-USA. They at least seem to be doing well enough to continue to put out new titles each year and buy advertising in anime magazines and on websites.

However, the emergence of iOS may open up of new doors for visual novels in the West, as it represents the convergence of several factors vital to the genre’s success. First and foremost is access to an audience: tens of millions of Americans, Europeans, and Australasians own iOS devices, and can easily find new titles through the App Store, more easily than they could through small companies with limited retail distribution (Hirameki) or effectively no retail at all (the eroge companies).

I went looking for visual novels in the App Store and realized I could much more easily find them by searching iTunes’ web pages, rather than using the search field in iTunes or the App Store on my iPad. Use your favorite search engine with a term like site:itunes.apple.com “visual novel” and you seem to be able to perform deeper and more sophisticated searches, and get more interesting “related” links, than is possible with just the search bar in Apple’s apps.

(As an aside, related links in a search for visual novels is what led me to Electro Master, a loving (if not deeply replayable) tribute by a Japanese indie developer to the era of 8-bit gaming, complete with a mimicry of the Namco startup sequence.)

Visual novels are also helped by the nature of iOS devices themselves. In my mind, sitting down at a desk and spinning up the computer is something of a turnoff as a prerequisite for reading a story. The iPad, of course, is exceptionally well-suited to casual reading around the house, while the iPhone and iPod are always-available devices, tucked away in a pocket or purse and easily pulled out for a brief diversion. Japan’s developers embraced mobile devices long ago and readily adapt their work for mobile, and often design it for mobile devices in the first place. Add to this the popularity of iOS in Japan, and you’ve got a fertile environment for this format to grow.

But having discussed the format and the business around visual novels at length, it’s well worth asking whether any of these stories are actually any good, and any different from other kinds of fiction and gaming. Like any genre, there’s plenty of junk mixed in with the good stuff, but the good stuff is truly remarkable. A recent Kotaku article, How Erotic Games Learned To Cry discusses nakige, a sub-genre that developed from eroge and uses the player’s romantic and sexual interaction with the characters to intensify feelings of sadness and melancholy (the term is a clever portmanteau, turning nukige, “games to masturbate to”, into nakige, “games to cry to”).

What’s remarkable about this is that so many of these games are male-oriented stories that involve matters of the heart… a combination that is almost non-existent in the US. The American genre of “romance” is almost exclusively targeted to women, and there simply isn’t a male equivalent. In a Twitter exchange with a Japanese visual novel author, I mentioned this phenomenon, and from what he knew about American media, the closest thing he could think of to male-oriented romantic stories in American media is the “American Pie” movies. And it’s remarkable because he’s kind of right!

The ways in which Japanese visual novels and adventure games involve the reader/player are strikingly different than their Western equivalents. Consider the adventure game: with the Western version, in the tradition of, say, LucasArts and Telltale games, the appeal and the interaction is almost entirely intellectual. You solve puzzles to move the game along, and the left brain is often further tickled with smart humor, a crackling wit best exemplified by games like Sam & Max or Monkey Island. In the Japanese games, much of the interaction is emotional: instead of puzzling over how to open a door, you’re tasked with choosing the right girl, not breaking someone’s heart, enduring a personal tragedy, and so on. Personally, this is why I’ve long gravitated towards the Japanese styles of storytelling in anime, manga, and games: the mixture of charm, bittersweetness, and melancholy is moving in ways that Western games and genre works (sci-fi, fantasy) rarely even attempt (the relaunched Doctor Who being a much-appreciated exception). Put simply, if you’re not in tears when the end credits song of Final Fantasy X rolls, you’re doing it wrong.

I think visual novels are also helped by the fact that they can be created by fairly small teams. Once you start putting a lot of people and money into something creative, it gets harder and harder to have a point or say anything controversial: the process naturally erases the distinctive voice of any of its creators. Visual novels require fairly straightforward programming skills (and there are toolkits that reduce it to very simple scripting), along with writing and drawing. A multi-talented individual could create a visual novel, though it’s more common to have small teams with assigned roles: writer, artist, etc. Still, for the development cost of a single blockbuster title, you could have many visual novels, each with its own unique voice and point of view. The low development costs are also likely to be a asset in the App Store era of ever collapsing prices, where $1.99 is too dear for some shoppers.

So where to begin with iOS visual novels? The trick right now is finding competent localizations. A few games have apparently been localized into English with machine translation, and the results are predictably hideous. Gift was a fairly successful visual novel in its Japanese release, but it won’t pick up many Western fans when its characters talk like this:

(Update: I can’t find “Gift(EN)Lite” in the App Store anymore… perhaps it was pulled?)

Much better are the titles where creators have cared enough to work on a genuine English translation. My favorite iOS visual novel so far is Kisaragi (Eng) Lite, a free sample from the paid game Kisaragi no Hogyoku. The story involves a high school with some strange traditions — such as the compulsory pairings of senior and junior classmates as couples — which relate to a mysterious artifact (the “kisaragi” of the title). It’s hard to tell from the first chapter if the story is headed towards fantasy, horror, romance, some combination thereof, or something else entirely. I’m mostly in at this point just because of the competent translation. It’s surprisingly nuanced, such as introducing a foreign character who stands out by her slightly-too-direct speaking style (what, she’s using dictionary form instead of -masu?)

In an interesting note, a crowd-sourced translation project of Kisaragi is underway, with the developers’ blessings. I’m hopeful this will eventually result in a release of the full game to the App Store.

I’m also reading the lite (free) version Kira☆Kira(eng), which has yet to win me over, but is well-reviewed and has a far more competent translation that its App Store description would suggest. Note, by the way, this is completely unrelated to Kira*Kira, which uses an asterisk in its name rather than the Unicode white star character.

Ripples is also worth mentioning. This short (10 minute) story is probably too insubstantial to deserve its change-of-heart ending — usually in visual novels you typically have to earn your love against the backdrop of a critical illness/injury or a centuries-old curse — but it’s noteworthy as it seems to be a Western creation in the style of the Japanese novels, using the same development tools as the big ones. It’s certainly a good start.

I’ll wrap up with a couple links to other visual novels I’ve found for iOS, and a few sites with more news and information about the format.

It’s a start. As the iOS platform grows, we’ve got a good chance of seeing the top-tier visual novels: Key’s Air, Clannad, and Kanon, âge’s Kimi ga Nozomu Eien (aka, Rumbling Hearts), and Type-Moon’s Fate/Stay Night. Researching this blog, I discovered that the best-seller HIGURASHI When They Cry has been ported to iOS with an English translation, so it’s not unreasonable to hope that other top-tier visual novels may make it all the way to Western iPhones, iPods, and iPads.

Now if you’ll excuse me, I’ve got some reading to do.

Beauty and the Box

Yesterday, I took my 5-year-old daughter, Quinn, to the Beauty and the Beast sing-a-long event. Quick summary: would have worked better with more people (we only had about 20, and most were shy), but helped to be in front of some theatre girls who knew the songs by heart and were into it. Still, one of my favorite movies, one I’ve surely seen 20 or 30 times. But let’s get back to digital media…

The event was meant to promote Tuesday’s re-release of Beauty and the Beast on home video, this time in its first HD edition. I’ve already owned B&tB on VHS and DVD (the 2003 edition cleverly contained the “work in progress” film circuit version, the original version, and the IMAX re-release that added an unneeded song). So I found myself wondering if I would be buying this release. Probably not, since I don’t own a Blu-Ray player and now that we’re many years into the Blu-Ray era, I don’t see that changing anytime soon. We don’t do a lot of movie watching anymore, as most of what we watch is DVR’ed off the DirecTV, and I didn’t fall for the “PlayStation 3 as Blu-Ray trojan horse” due to the PS3’s absurd unaffordability. And I don’t feel like we’ve missed it.

Then I thought: “wait, Blu-Ray isn’t the only form of HD.” There’s also on-demand from DirecTV, and what about iTunes? A little search there shows that yes indeed, the B&tB platinum edition will also be available on iTunes: $14.99 for SD, $19.99 for HD.

Of course, these Disney classics are usually only available for a short time before they “go back in the vault”, to enhance demand for the next re-release. So if I felt I did need to grab an HD version before it went away, which would I get?

Thinking about it, I think I’m more likely to buy an AppleTV — or at least rig a Mac Mini to a TV — before I get a Blu-Ray player. As it is, I could play the HD .m4p on a bunch of the devices I currently own (computers, iPhone, iPad), and the only thing that’s missing is connectivity to the TV. In fact, various video out cables allow for iOS devices to serve as a sort of “poor man’s” first-gen AppleTV, depending on your available connections, how many videos you’ve loaded on your iPod, and your tolerance for SD. A Blu-Ray disc would be locked to the TV the player is connected to, and wouldn’t be rippable for the iDevices (though this particular bundle may come with a digital copy… haven’t checked).

Still, I’m surprised to find that I’ve blundered into exactly what Steve Jobs purportedly told a customer in one of those alleged off-the-cuff e-mails: Blu-Ray is coming up short, and will eventually be replaced by digital downloads, just as CD successors were beaten by downloads (anybody spun up an SACD lately?).

BTW, Apple’s resolute anti-Blu stance is made all the more interesting by the fact that Apple is a board member of the Blu-Ray Disc Association.

Another note about the AppleTV: teardowns and other spelunking reevel that the new device runs iOS and has 8GB of storage, which would be suitable for apps, should Apple ever choose to deliver a third-party SDK. Clearly the UI would be different — perhaps it exists not as an “AppKit” or “UIKit” but rather a “TVKit” atop Foundation and the rest of the usual Apple stack — but there would be all sorts of interesting opportunities.

One of the most obvious would be for all the existing iOS streaming media apps to connect to the TV. This includes the sports apps — everyone knows about the MLB app, but look further and you’ll find apps for major events like the PGA Championship and Ryder Cup also have their own apps with live video available via in-app purchase, DirecTV’s “NFL Sunday Ticket” streams to phones, etc. There are also specialized video apps for all manner of niches. For example, as an anime fan, I use Crunchyroll’s streaming app, and might someday sign up for Anime Network Mobile. I imagine every other little video fetish has its own streaming app, or soon will.

(By the way, none of these apps can use the standard def video out cables like Apple’s iPod or Videos apps can. When you connect the composite video cable and do [UIScreen screens], you only see one screen, so these streaming apps can’t access the video out and put their player UI over there. rdar://8063058 )

By Apple fiat, essentially all of these apps need to use HTTP Live Streaming, and an AppleTV that permitted third-party apps would presumably drive even more content providers to this standard. I had previously wondered aloud about doing an HTTP Live Streaming book, but if we get an AppleTV SDK, it would make perfect sense for the HLS material to become one or two chapters of a book on AppleTV programming, along the lines of “if you’re programming for this platform, you’re almost certainly going to be streaming video to it, so here’s how the client side works, and here’s how to set up your server.”

From iPhone Media Library to PCM Samples in Dozens of Confounding, Potentially Lossy Steps

iPhone SDK 3.0 provided limited access to the iPod Music Library on the device, allowing third party apps to search for songs (and podcasts and audiobooks, but not video), inspect the metadata, and play items, either independently or in concert with the built-in media player application. But it didn’t provide any form of write-access — you couldn’t add items or playlists, or alter metadata, from a third-party app. And it didn’t allow for third-party apps to do anything with the songs except play them… you couldn’t access the files, convert them to another format, run any kind of analysis on the samples, and so on.

So a lot of us were surprised by the WWDC keynote when iMovie for iPhone 4 was shown importing a song from the iPod library for use in a user-made video. We were even more surprised by the subsequent claim that everything in iMovie for iPhone 4 was possible with public APIs. Frankly, I was ready to call bullshit on it because of the iPod Library issue, but was intrigued by the possibility that maybe you could get at the iPod songs in iOS 4. A tweet from @ibeatmaker confirmed that it was possible, and after some clarification, I found what I needed.

About this time, a thread started on coreaudio-api about whether Core Audio could access iPod songs, so that’s what I set out to prove one way or another. So, my goal was to determine whether or not you could get raw PCM samples from songs in the device’s music library.

The quick answer is: yes. The interesting answer is: it’s a bitch, using three different frameworks, coding idioms that are all over the map, a lot of file-copying and possibly some expensive conversions.

It’s Just One Property; It Can’t Be That Hard

The big secret of how to get to the Music Library isn’t much of a secret. As you might expect, it’s in the MediaLibrary.framework that you use to interact with the library. Each song/podcast/audiobook is a MPMediaItem, and has a number of interesting properties, most of which are user-managed metadata. In iOS 4, there’s a sparkling new addition to the the list of “General Media Item Property Keys”: MPMediaItemPropertyAssetURL. Here’s the docs:

A URL pointing to the media item, from which an AVAsset object (or other URL-based AV Foundation object) can be created, with any options as desired. Value is an NSURL object.

The URL has the custom scheme of ipod-library. For example, a URL might look like this:

ipod-library://item/item.m4a?id=12345

OK, so we’re off and running. All we need to do is to pick an MPMediaItem, get this property as an NSURL, and we win.

Or not. There’s an important caveat:

Usage of the URL outside of the AV Foundation framework is not supported.

OK, so that’s probably going to suck. But let’s get started anyways. I wrote a throwaway app to experiment with all this stuff, adding to it piece by piece as stuff started working. I’m posting it here for anyone who wants to reuse my code… all my classes are marked as public domain, so copy-and-paste as you see fit.

MediaLibraryExportThrowaway1.zip

Note that this code must be run on an iOS 4 device and cannot be run in the Simulator, which doesn’t support the Media Library APIs.

The app just starts with a “Choose Song” button. When you tap it, it brings up an MPMediaPickerController as a modal view to make you choose a song. When you do so, the -mediaPicker:didPickMediaItems: delegate method gets called. At this point, you could get the first MPMediaItem and get its MPMediaItemPropertyAssetURL media item property. I’d hoped that I could just call this directly from Core Audio, so I wrote a function to test if a URL can be opened by CA:



BOOL coreAudioCanOpenURL (NSURL* url) {
	OSStatus openErr = noErr;
	AudioFileID audioFile = NULL;
	openErr = AudioFileOpenURL((CFURLRef) url,
		 kAudioFileReadPermission ,
		 0,
		 &audioFile);
	if (audioFile) {
		AudioFileClose (audioFile);
	}
	return openErr ? NO : YES;
}

Getting a NO back from this function more or less confirms the caveat from the docs: the URL is only for use with the AV Foundation framework.

AV for Vendetta

OK, so plan B: we open it with AV Foundation and see what that gives us.

AV Foundation — setting aside the simple player and recorder classes from 3.0 — is a strange and ferocious beast of a framework. It borrows from QuickTime and QTKit (the capture classes have an almost one-to-one correspondence with their QTKit equivalents), but builds on some new metaphors and concepts that will take the community a while to digest. For editing, it has a concept of a composition, which is made up of tracks, which you can create from assets. This is somewhat analogous to QuickTime’s model that “movies have tracks, which have media”, except that AVFoundation’s compositions are themselves assets. Actually, reading too much QuickTime into AV Foundation is a good way to get in trouble and get disappointed; QuickTime’s most useful functions, like AddMediaSample() and GetMediaNextInterestingTime() are antithetical to AV Foundation’s restrictive design (more on that in a later blog) and therefore don’t exist.

Back to the task at hand. The only thing we can do with the media library URL is to open it in AVFoundation and hope we can do something interesting with it. The way to do this is with an AVURLAsset.


NSURL *assetURL = [song valueForProperty:MPMediaItemPropertyAssetURL];
AVURLAsset *songAsset = [AVURLAsset URLAssetWithURL:assetURL options:nil];

If this were QuickTime, we’d have an object that we could inspect the samples of. But in AV Foundation, the only sample-level access afforded is a capture-time opportunity to get called back with video frames. There’s apparently no way to get to video frames in a file-based asset (except for a thumbnail-generating method that operates on one-second granularity), and no means of directly accessing audio samples at all.

What we can do is to export this URL to a file in our app’s documents directory, hopefully in a format that Core Audio can open. AV Foundation’s AVAssetExportSession has a class method exportPresetsCompatibleWithAsset: that reveals what kinds of formats we can export to. Since we’re going to burn the time and CPU of doing an export, it would be nice to be able to convert the compressed song into PCM in some kind of useful container like a .caf, or at least an .aif. But here’s what we actually get as options:

compatible presets for songAsset: (
 AVAssetExportPresetLowQuality,
 AVAssetExportPresetHighestQuality,
 AVAssetExportPreset640x480,
 AVAssetExportPresetMediumQuality,
 AVAssetExportPresetAppleM4A
 )

So, no… there’s no “output to CAF”. In fact, we can’t even use AVAssetExportPresetPassthrough to preserve the encoding from the music library: we either have to convert to an AAC (in an .m4a container), or to a QuickTime movie (represented by all the presets ending in “Quality”, as well as the “640×480”).

This Deal is Getting Worse All the Time!

So, we have to export to AAC. That’s not entirely bad, since Core Audio should be able to read AAC in an .m4a container just fine. But it sucks in that it will be a lossy conversion from the source, which could be MP3, Apple Lossless, or some other encoding.

In my GUI, an “export” button appears when you pick a song, and the export is kicked off in the event-handler handleExportTapped. Here’s the UI in mid-export:

MediaLibraryExportThrowaway1 UI in mid-export

To do the export, we create an AVExportSession and provide it with an outputFileType and outputIURL.


AVAssetExportSession *exporter = [[AVAssetExportSession alloc]
		initWithAsset: songAsset
		presetName: AVAssetExportPresetAppleM4A];
NSLog (@"created exporter. supportedFileTypes: %@", exporter.supportedFileTypes);
exporter.outputFileType = @"com.apple.m4a-audio";
NSString *exportFile = [myDocumentsDirectory()
		stringByAppendingPathComponent: @"exported.m4a"];
myDeleteFile(exportFile);
[exportURL release];
exportURL = [[NSURL fileURLWithPath:exportFile] retain];
exporter.outputURL = exportURL;	

A few notes here. The docs say that if you set the outputURL without setting outputFileType that the exporter will make a guess based on the file extension. In my experience, the exporter prefers to just throw an exception and die, so set the damn type already. You can get a list of possible values from the class method exporter.supportedFileTypes. The only supported value for the AAC export is com.apple.m4a-audio. Also note the call to a myDeleteFile() function; the export will fail if the target file already exists.

Aside: I did experiment with exporting as a QuickTime movie rather than an .m4a; the code is in the download, commented out. Practical upshot is that it sucks: if your song isn’t AAC, then it gets converted to mono AAC at 44.1 KHz. It’s also worth noting that AV Foundation doesn’t give you any means of setting export parameters (bit depths, sample rates, etc.) other than using the presets. If you’re used to the power of frameworks like Core Audio or the old QuickTime, this is a bitter, bitter pill to swallow.

Block Head

The code gets really interesting when you kick off the export. You would probably expect the export, a long-lasting operation, to be nice and asynchronous. And it is. You might also expect to register a delegate to get asynchronous callbacks as the export progresses. Not so fast, Bucky. As a new framework, AV Foundation adopts Apple’s latest technologies, and that includes blocks. When you export, you provide a completion handler, a block whose no-arg function is called when necessary by the exporter.

Here’s what mine looks like.


// do the export
[exporter exportAsynchronouslyWithCompletionHandler:^{
	int exportStatus = exporter.status;
	switch (exportStatus) {
		case AVAssetExportSessionStatusFailed: {
			// log error to text view
			NSError *exportError = exporter.error;
			NSLog (@"AVAssetExportSessionStatusFailed: %@",
				exportError);
			errorView.text = exportError ?
				[exportError description] : @"Unknown failure";
			errorView.hidden = NO;
			break;
		}
		case AVAssetExportSessionStatusCompleted: {
			NSLog (@"AVAssetExportSessionStatusCompleted");
			fileNameLabel.text =
				[exporter.outputURL lastPathComponent];
			// set up AVPlayer
			[self setUpAVPlayerForURL: exporter.outputURL];
			[self enablePCMConversionIfCoreAudioCanOpenURL:
				exporter.outputURL];
			break;
		}
		case AVAssetExportSessionStatusUnknown: {
			NSLog (@"AVAssetExportSessionStatusUnknown"); break;}
		case AVAssetExportSessionStatusExporting: {
			NSLog (@"AVAssetExportSessionStatusExporting"); break;}
		case AVAssetExportSessionStatusCancelled: {
			NSLog (@"AVAssetExportSessionStatusCancelled"); break;}
		case AVAssetExportSessionStatusWaiting: {
			NSLog (@"AVAssetExportSessionStatusWaiting"); break;}
		default: { NSLog (@"didn't get export status"); break;}
	}
}];

This kicks off the export, passing in a block with code to handle all the possible callbacks. The completion handler function doesn’t have to take any arguments (nor do we have to set up a “user info” object for the exporter to pass to the function), since the block allows anything in the local scope to be called from the block. That means the exporter and its state don’t need to be passed in as parameters, because the exporter is a local variable that can be accessed from the block and its state inspected via method calls.

The two messages I handle in my block are AVAssetExportSessionStatusFailed, which dumps the error to a previously-invisible text view, and AVAssetExportSessionStatusCompleted, which sets up an AVPlayer to play the exported audio, which we’ll get to later.

After starting the export, my code runs an NSTimer to fill a UIProgressView. Since the exporter has a progress property that returns a float, it’s pretty straightforward… check the code if you haven’t already done this a bunch of times. Files that were already AAC export almost immediately, while MP3s and Apple Lossless (ALAC) took a minute or more to export. Files in the old .m4p format, from back when the iTunes Store put DRM on all the songs, fail with an error, as seen below.

The Invasion of Time

Kind of as a lark, I added a little GUI to let you play the exported file. AVPlayer was the obvious choice for this, since it should be able to play whatever kind of file you export (.m4a, .mov, whatever).

This brings up the whole issue of how to deal with the representation of time in AV Foundation, which turns out to be great for everyone who ever used the old C QuickTime API (or possibly QuickTime for Java), and all kinds of hell for everyone else.

AV Foundation uses Core Media’s CMTime struct for representing time. In turn, CMTime uses QuickTime’s brilliant but tricky concept of time scales. The idea, in a nutshell, is that your units of measurement for any particular piece of media are variable: pick one that suits the media’s own timing needs. For example, CD audio is 44.1 KHz, so it makes sense to measure time in 1/44100 second intervals. In a CMTime, you’d set the timescale to 44100, and then a given value would represent some number of these units: a single sample would have a value of 1 and would represent 1/44100 of a second, exactly as desired.

I find it’s easier to think of Core Media (and QuickTime) timescales as representing “nths of a second”. One of the clever things you can do is to choose a timescale that suits a lot of different kinds of media. In QuickTime, the default timescale is 600, as this is a common multiple of many important frame-rates: 24 fps for film, 25 fps for PAL (European) TV, 30 fps for NTSC (North America and Japan) TV, etc. Any number of frames in these systems can be evenly and exactly represented with a combination of value and timescale.

Where it gets tricky is when you need to work with values measured in different timescales. This comes up in AV Foundation, as your player may use a different timescale than the items it’s playing. It’s pretty easy to write out the current time label:


CMTime currentTime = player.currentTime;
UInt64 currentTimeSec = currentTime.value / currentTime.timescale;
UInt32 minutes = currentTimeSec / 60;
UInt32 seconds = currentTimeSec % 60;
playbackTimeLabel.text = [NSString stringWithFormat:
		@"%02d:%02d", minutes, seconds];

But it’s hard to update the slider position, since the AVPlayer and the AVPlayerItem it’s playing can (and do) use different time scales. Enjoy the math.


if (player && !userIsScrubbing) {
	CMTime endTime = CMTimeConvertScale (player.currentItem.asset.duration,
		currentTime.timescale,
		kCMTimeRoundingMethod_RoundHalfAwayFromZero);
	if (endTime.value != 0) {
		double slideTime = (double) currentTime.value /
				(double) endTime.value;
		playbackSlider.value = slideTime;
	}
}

Basically, the key here is that I need to get the duration of the item being played, but to express that in the time scale of the player, so I can do math on them. That gets done with the CMTimeConvertScale() call. Looks simple here, but if you don’t know that you might need to do a timescale-conversion, your math will be screwy for all sorts of reasons that do not make sense.

Oh, you can drag the slider too, which means doing the same math in reverse.


-(IBAction) handleSliderValueChanged {
	CMTime seekTime = player.currentItem.asset.duration;
	seekTime.value = seekTime.value * playbackSlider.value;
	seekTime = CMTimeConvertScale (seekTime, player.currentTime.timescale,
			kCMTimeRoundingMethod_RoundHalfAwayFromZero);
	[player seekToTime:seekTime];
}

One other fun thing about all this that I just remembered from looking through my code. The time label and slider updates are called from an NSTimer. I set up the AVPlayer in the completion handler block that’s called by the exporter. This call seems not to be on the main thread, as my update timer didn’t work until I forced its creation over to the main thread with performSelectorOnMainThread:withObject:waitUntilDone:. Good times.

Final Steps

Granted, all this AVPlayer stuff is a distraction. The original goal was to get from iPod Music Library to decompressed PCM samples. We used an AVAssetExportSession to produce an .m4a file in our app’s Documents directory, something that Core Audio should be able to open. The remaining conversion is a straightforward use of CA’s Extended Audio File Services: we open an ExtAudioFileRef on the input .m4a, set a “client format” property representing the PCM format we want it to convert to, read data into a buffer, and write that data back out to a plain AudioFileID. It’s C, so the code is long, but hopefully not too hard on the eyes:


-(IBAction) handleConvertToPCMTapped {
	NSLog (@"handleConvertToPCMTapped");
	
	// open an ExtAudioFile
	NSLog (@"opening %@", exportURL);
	ExtAudioFileRef inputFile;
	CheckResult (ExtAudioFileOpenURL((CFURLRef)exportURL, &inputFile),
				 "ExtAudioFileOpenURL failed");
	
	// prepare to convert to a plain ol' PCM format
	AudioStreamBasicDescription myPCMFormat;
	myPCMFormat.mSampleRate = 44100; // todo: or use source rate?
	myPCMFormat.mFormatID = kAudioFormatLinearPCM ;
	myPCMFormat.mFormatFlags =  kAudioFormatFlagsCanonical;	
	myPCMFormat.mChannelsPerFrame = 2;
	myPCMFormat.mFramesPerPacket = 1;
	myPCMFormat.mBitsPerChannel = 16;
	myPCMFormat.mBytesPerPacket = 4;
	myPCMFormat.mBytesPerFrame = 4;
	
	CheckResult (ExtAudioFileSetProperty(inputFile,
			kExtAudioFileProperty_ClientDataFormat,
			sizeof (myPCMFormat), &myPCMFormat),
		  "ExtAudioFileSetProperty failed");

	// allocate a big buffer. size can be arbitrary for ExtAudioFile.
	// you have 64 KB to spare, right?
	UInt32 outputBufferSize = 0x10000;
	void* ioBuf = malloc (outputBufferSize);
	UInt32 sizePerPacket = myPCMFormat.mBytesPerPacket;	
	UInt32 packetsPerBuffer = outputBufferSize / sizePerPacket;
	
	// set up output file
	NSString *outputPath = [myDocumentsDirectory() 
			stringByAppendingPathComponent:@"export-pcm.caf"];
	NSURL *outputURL = [NSURL fileURLWithPath:outputPath];
	NSLog (@"creating output file %@", outputURL);
	AudioFileID outputFile;
	CheckResult(AudioFileCreateWithURL((CFURLRef)outputURL,
		   kAudioFileCAFType,
		   &myPCMFormat, 
		   kAudioFileFlags_EraseFile, 
		   &outputFile),
		  "AudioFileCreateWithURL failed");
	
	// start convertin'
	UInt32 outputFilePacketPosition = 0; //in bytes
	
	while (true) {
		// wrap the destination buffer in an AudioBufferList
		AudioBufferList convertedData;
		convertedData.mNumberBuffers = 1;
		convertedData.mBuffers[0].mNumberChannels = myPCMFormat.mChannelsPerFrame;
		convertedData.mBuffers[0].mDataByteSize = outputBufferSize;
		convertedData.mBuffers[0].mData = ioBuf;

		UInt32 frameCount = packetsPerBuffer;

		// read from the extaudiofile
		CheckResult (ExtAudioFileRead(inputFile,
			  &frameCount,
			  &convertedData),
			 "Couldn't read from input file");
		
		if (frameCount == 0) {
			printf ("done reading from file");
			break;
		}
		
		// write the converted data to the output file
		CheckResult (AudioFileWritePackets(outputFile,
			   false,
			   frameCount,
			   NULL,
			   outputFilePacketPosition / myPCMFormat.mBytesPerPacket, 
			   &frameCount,
			   convertedData.mBuffers[0].mData),
			 "Couldn't write packets to file");
		
		NSLog (@"Converted %ld bytes", outputFilePacketPosition);

		// advance the output file write location
		outputFilePacketPosition +=
			(frameCount * myPCMFormat.mBytesPerPacket);
	}
	
	// clean up
	ExtAudioFileDispose(inputFile);
	AudioFileClose(outputFile);

	// GUI update omitted
}

Note that this uses a CheckResult() convenience function that Kevin Avila wrote for our upcoming Core Audio book… it just looks to see if the return value is noErr and tries to convert it to a readable four-char-code if it seems amenable. It’s in the example file too.

Is It Soup Yet?

Does all this work? Rather than inspecting the AudioStreamBasicDescription of the resulting file, let’s do something more concrete. With Xcode’s “Organizer”, you can access your app’s sandbox on the device. So we can just drag the Application Data to the Desktop.

In the resulting folder, open the Documents folder to find export-pcm.caf. Drag it to QuickTime Player to verify that you do, indeed, have PCM data:

So there you have it. In several hundred lines of code, we’re able to get a song from the iPod Music Library, export it into our app’s Documents directory, and convert it to PCM. With the raw samples, you could now draw an audio waveform view (something you’d think would be essential for video editors who want to match video to beats in the music, but Apple seems dead-set against letting us do do with AV Foundation or QTKit), you could perform analysis or effects on the audio, you could bring it into a Core Audio AUGraph and mix it with other sources… all sorts of possibilities open up.

Clearly, it could be a lot easier. It’s a ton of code, and two file exports (library to .m4a, and .m4a to .caf), when some apps might be perfectly happy to read from the source URL itself and never write to the filesystem… if only they could. Having spent the morning writing this blog, I may well spend the afternoon filing feature requests on bugreport.apple.com. I’ll update this blog with OpenRadar numbers for the following requests:

  • Allow Core Audio to open URLs provided by MediaLibrary’s MPMediaItemPropertyAssetURL
  • AV Foundation should allow passthrough export of Media Library items
  • AV Foundation export needs finer-grained control than just presets
  • Provide sample-level access for AVAsset

Still, while I’m bitching and whining, it is remarkable that iOS 4 opens up non-DRM’ed items in the iPod library for export. I never thought that would happen. Furthermore, the breadth and depth of the iOS media APIs remain astonishing. Sometimes terrifying, perhaps, but compared to the facile and trite media APIs that the other guys and girls get, we’re light-years ahead on iOS.

Have fun with this stuff!

Update: This got easier in iOS 4.1. Please forget everything you’ve read here and go read From iPod Library to PCM Samples in Far Fewer Steps Than Were Previously Necessary instead.

Stalk my stream

Hey, when did this happen? I have iChat set to “Current iTunes Song”, and while playing a Shoutcast stream (CBC Radio 3) with embedded song metadata, the iTunes song display kept up with the changes in the stream:
ichat-streaming-itunes

Speaking of streaming, I’m still deep in the streaming audio chapter, the last one for the book. Recording (PCM and iPhone-supported encoded formats, like Apple Lossless, IMA4, and iLBC) and playback are largely working in example code, converting with ExtAudioFiles isn’t as happy. In the interests of immediacy, I’ll probably start the chapter today and include everything but conversion, then get it fixed in the edit/rewrite cycle.

After that, tech review, tons of errata and off to print!

Link: Harmonix announces Beatles game

Details from Gamasutra. With all the iconic design associated with the Beatles — the Yellow Submarine movie, the Sgt. Pepper concept, the B&W of “A Hard Day’s Night” or the Bond-spoof “Help!”– this could be as fascinating visually as much as anything else. Maybe “Rock Band” meets “Kingdom Hearts”, where the different Beatles eras and concepts serve as the various lands?

Since Harmonix is made up of musicians, it’s safe to assume it won’t suck like the Sgt. Pepper movie. Worst case, it’s Across the Universe, which sits unwatched in my DVR…

BTW, with rumor mongers saying for years that a deal for Beatles on iTunes was close, who would have thought that we’d see them in a game before being on iTunes or one of its rivals?

When content filters fail

iTunes tries, and fails, to censor an obscene band name:

iTunes tries, and fails, to censor an obscene band name

WWDC: Whenever We’re Done Compressing

Finally! WWDC 2008 session videos and slides are available to attendees.

I’m up at the folks’ place, on their wireless broadband from across the lake ISP, and I managed to get about 40 sets of slides and 4 videos overnight before the site went down for “maintenance”.

When I get back to GRR and 3 Mbps, I may just have iTunes go and get everything. That might finally use up the untouched 60 GB on my iPod Classic that I haven’t filled yet.