Authoring instead of / in addition to coding

A commonality I’ve recently noticed is that some of the Mac frameworks encourage a combination of coding and authoring, in that for some set of functionality, access is possible both with user-level authoring (via visual tools, scripting, etc.) as well as with full-on coding. And this may be something that developers overlook.

It’s the old “if the only tool you have is a hammer, everything looks like a nail” analogy. Let me give you a practical example. Lots of people post to the various QuickTime lists, seeking to do some sort of “overlay”, taking a video and putting a static image (or, less commonly, another video) on top of it. When this comes up on the QTJ list, the poster almost always assumes they need to hack into the rendering pipeline, either stacking views atop one another with a JLayeredPane or, even worse, doing some sort of nasty callback-driven repainting hack.

Almost nobody realizes that they can do it with about 20 lines of XML:

<smil xmlns:qt="">
      <root-layout id="root" width="320" height="240"
      <region id="main" width="320" height="240"
              z-index="1" fit="meet" />
      <region id="logo-reg" width="286" height="94"
              z-index="2" left="17" top="146"/>

      <img src="sf-logo-2.png" 
            region="logo-reg" dur="15"/>
      <video src="" region="root" dur="15" />

When opened in any QuickTime application, the result looks like this:

SMIL example 1

The above is a simple SMIL movie, which defines areas for the movie and overlay and sets the compositing mode to punch out a transparency. The <par> tags indicate the elements are to be presented in parallel, which would allow you to add an audio file if you were so inclined. A corresponding <seq> tag lets you build sequences with script. And you can composite video too, using a fit attribute on the region to scale the overlaid movie:

SMIL example 2

One advantage of this approach is that all the rendering takes place in QuickTime, so you don’t sacrifice performance by having to copy your pixels over to Java2D/Swing.

But of course the big advantage is obviously the fact that it’s easy. It’s well within the range of the newbie QuickTime or Java programmer, as opposed to the pipeline-rendering hacks implied above. In this sense, it’s preferable to another option I’ve posted to the lists before: creating a one-sample video track with your overlay image. Creating the SMIL file does imply actually writing to disk somewhere, but I suspect that if you must do this on the fly and can’t write to disk, you could probably write the XML file in memory, wrap it with a QTHandle and then create the movie with Movie.fromHandle(). Bonus for experts: this might work faster / more reliably by creating a SMIL-specific MovieImporter and using its fromHandle(). Details…

See, that’s one of the things with a lot of the Mac technologies: there are multiple entry points. With QuickTime, there’s a bunch of functionality that you get by authoring, which you can do with GUIs (QuickTime Player Pro), scripting languages (AppleScript, JavaScript in a browser), or even markup, as seen here. These same things, and of course a lot more, can be achieved with code, but sometimes it makes sense to put down the compiler and just author your functionality. And QuickTime isn’t the only example. Don’t we also see this pattern in the following:

  • AppleScript – as opposed to low-level approaches to inter-application communication and automation
  • WebKit – HTML could be an option for large blocks of styled text, particularly if it needs to change at runtime
  • Quartz Composer

I’m working on a article or screencast proposal on the latter, and the “take” I’m using is that instead of writing your own fancy rendering code (yay, OpenGL), you could just use Quartz Composer to build a .qtz file with a few malleable properties, and then bind those to the values in your program that will change. Imagine a fancy, skinnable MP3 player: if you want the artist and title to be displayed with some crazy combination of gradients, reflections, 3D effects, etc., then instead of coding it, you (or better yet, a QC-savvy graphic designer) could just build those effects on dummy text with the Quartz Composer GUI, then expose those inputs as bindable values, which you’d then wire up in Interface Builder or directly in your code.

Java developers never think this way, perhaps because there are almost no examples of this kind of layered-technology approach in the Java world. About the only examples I can think of are Drools and Fitnesse, both of which are about coding your essential logic in something other than Java, usually because it’s meant for someone other than Java programmers to do. But why shouldn’t we want to see more of this kind of approach? Indeed, on the Java side, we now have all these scripting languages running on the JDK, and have a JavaScript interpreter build into JDK 6. If there’s some part of your system that’s particularly volatile or meant to be hacked on by many people, why not just define that as being written in JavaScript? Early on at Pathfire, we had to handle different business logic for different customers — today, I might well create an API for customer-specific JavaScripts to talk to, with the added bonus that there are millions of techies who might not know (or want to know) Java, but who can bang out JavaScript easily enough.

Speaking of which, Quartz Composer also lets you put JavaScript inside a composition. Authoring begets scripting, and if applied correctly, allows us developers to work on the hard parts they pay us the big bucks for.

Comment (1)

  1. […] who try to do video overlays by trying to hack QuickTime’s render pipeline rather than just authoring a multi-layer movie like an end-user […]

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.