The Delicate Dance Between AR and Special Effects

The Interplay Between Augmented Reality
and Special Effects
 
Humble's Chris Grayson, Director of Digital, on the interwoven
nature of AR and VFX as seen in commercials.


By Chris Grayson

Humble's Chris Grayson says production companies with VFX expertise are well suited to bring AR to brands.

If you're wondering what Augmented Reality is and what it can do to help drive the creative quality and effectiveness of your commercials and web content, you're probably not alone.  There are a lot of interesting uses of AR that's cropping up in ads these days, and just as much confusion, it seems.  We're here to help.
 
I've read many definitions of Augmented Reality. "Special Effects, Rendered in Real Time, Over Real Life" is a succinct definition of AR that I wrote to encapsulate the broadest range of AR executions, irrespective of application. The definition focuses on production / execution, rather than how it's used.  It's also a definition that puts the effects industry's contextual relevance to AR front and center.
 
It's worth pointing out that AR is not really a technology unto itself, but rather a collection of technologies that overlay some form of virtual content over your perception of the real world.  This is usually done in the form of graphics (they can be either text or objects) inserted into one's field of view, most often using the camera view of a computer or mobile device. Think of the HUD (heads up display) used by pilots in a fighter jet, or the simulated HUD showing strength, ammunition and other stats in first-person-shooter video games.
 
Right now, the most common form of AR are the graphical overlays in live televised sporting events, such as the 1st down line in pro football games or the advertisements that pop behind home-plate in baseball - all applications from the broadcast industry.
 
Some colleagues were surprised when I left the digital agency world (I'd worked at the digital arms of Euro and Ogilvy, and consulted at Grey's G2 interactive group) to join Humble - a pre-to-post production and special effects boutique - instead of say, joining a mobile AR startup or other tech company. But the decision was very deliberate.  I believe that companies that understand narrative and filmmaking, and can combine that knowledge with digital production skills and visual effects expertise, are well positioned to help advertisers and agencies tap into AR's unique capabilities to enhance messages.
 
So where did this technique come from? The term Augmented Reality was coined in 1991 by an engineer at Boeing named Tom Caudell as part of a project he proposed to cut back on their reliance on paper blueprints - instead putting the assembly-line plans on head-mounted, heads-up displays. But this was not the beginning of AR. Augmented Reality existed as a concept without a name for many years.

"Terminator Vision," from "Terminator 2," was an early form of AR.

Consider that the top sci-fi/action film of the year in 1991 was "Terminator 2." "Terminator Vision" is the most common pop-culture reference to explain AR to the uninitiated. That's the on-screen graphics and data that would appear when the audience was seeing things from the Terminator's POV (see image at right). And the original "Terminator" was released in 1984.
 
But that's not the whole story. The symbiotic relationship between sci-fi, special effects and science are in a constant, self referencing feedback loop. Augmented Reality as a modern communication theory goes back at least to the early papers of ARPA (Advanced Research Projects Agency, the Pentagon's new technology think tank) leaders like J.C.R. Licklider's, "Man-Computer Symbiosis" (1961), Douglas Engelbart's "Augmenting Human Intellect" (1962) and Ivan Sutherland's "The Ultimate Display" (1965).
 
While these early scientific papers and programs may have built the philosophic foundation of Augmented Reality, it has been the special effects industry that's done the most to both conceptualize the interface and to spread these ideas throughout pop culture.
 

MK 12 used AR techniques in the opening sequence of "Stranger Than Fiction," directed by Marc Forster.

Besides "Terminator," one of my favorite examples is the graphic treatment from the intro to "Stranger Than Fiction," the 2006 feature directed by Marc Forster. The film is a comedy about a novelist suffering writer's block who encounters, in real life, the character of her currently blocked and half-finished book. In the opening sequence the writer, played by Emma Thompson, describes how this compulsive everyman (he's Harold Crick, played by Ferrell) keeps count of every mundane detail in his life such the number of brushstrokes in his morning tooth brushing. As she describes it in voiceover, these details are illustrated on screen with a profusion of clever, 3D type elements.
 
The look appears to have inspired the product call-outs and features seen in a 2010 spot for the GMC Terrain SUV. Titled "All Standard," it was created by Leo Burnett Detroit and directed by Mark Glaser of SWAY. (He's now with Big Block.) Another example of AR in commercials was the Super Bowl spot from earlier this year titled "Wonderful World," produced by Radium/ReelFX, which used AR graphics to illustrate the news and feature content you could get on your iPad or other mobile device from The Daily, the new News Corp. digital publication.

AR helps call out product features in this GMC spot titled "All Standard."

As the technology matures, the underlying AR platform could even become commoditized, great content will not. Back when rendering realistic 3D computer-generated graphics required multimillion-dollar super computers, such as in the era of "TRON," owning the platform was the business choke-point where margins could be made. Today there are a kaleidoscope of 3D development platforms on the market, most of which can be run, to some extent, on any teenager's sufficiently souped-up desktop machine. The choke-point now revolved around talent - those who can conceive of the best ideas and execute them with the available tools hold the keys.
 
One of the shortcomings in the AR space is that content is currently treated as secondary to the technology. With tech driving the train, most current executions lack depth, creativity or a captivating narrative.  Creative production shops like Humble, on the flip side, have the proper collection of talent to develop rich creative content and the special effects skill-set to produce them. As AR matures, and its use and implementation becomes less about gimmick and more about quality, the effects industry is in a unique position to capitalize on this opportunity.

In "Wonderful World" for The Daily, AR demonstrates the digital publication's ability to add context to articles.

But we in the effects world are also spoiled, and we've spoiled the audience. Over the years the techniques and technology have advanced to the point where the line between real and virtual within video production has become indistinguishable.
 
We have, as a result, established a high bar of expectation that consumers have when it comes to content in the AR space. When a user launches a mobile AR app, their expectation is not set by Layar (a technology platform that superimposes AR content over images captured on your mobile device), it's set by "Avatar." Users expect the content and production value to be on par with what they are familiar with from television and movies - the world of professional special effects.
 
The reason we ourselves are spoiled is because we have the tools at our disposal to create content at this level of quality. Tools that allow content to integrate this seamlessly into the real world simply don't exist yet for AR content. In the world of mobile AR, this is especially true of registration and occlusion. (Registration refers to the accurate alignment of digitally created content into the real world, and occlusion describes the effect where, when one object goes behind another, its hidden portions are not visible.)
 
I'm convinced that a solution will come out of the effects world which will ultimately to solve this technical challenge, largely because the issues of registration and occlusion have already been solved for in the post-production effects industry, and partially because those in the effects world will insist on producing AR content that's up to our own industry's standards.
 
Chris Grayson is Director of Digital at Humble. He's also an active event organizer, co-organizing an annual event in Santa Clara called ARE (Augmented Reality Event), and this year is founder and organizer of the inaugural TEDxSiliconAlley. He also writes the blog GigantiCo that frequently covers AR, creative and emerging media technology. You can check out his Twitter feed here.

Published 23 September, 2011