3D Mapping the World, Cheaply
Computer graphics these days are expensive; only two of the twenty-five most expensive movies ever came out before the mid-nineties, even adjusting for inflation. While computer power is getting ever-cheaper, the artists aren’t. Scripts are only getting wilder, and audiences demand more and more spectacular films with every passing summer. And as these graphics are already prohibitive for movies, they’re way outside the potential budget of any academic or non-profit projects. Is there any way out of this bind?
One field of computer graphics promises to take advantage of computing power and turn it towards making artists’ jobs easier. Called image-based modeling and rendering, the essential insight of this movement is pretty simple to understand: if we can take a 3D model in the computer and turn it into a 2D image for our movie-screen, can we figure out a way to do it the other way around? Can we turn a 2D image into a 3D model, automatically?
One early effort in the field was The Campanile Movie, made by Paul Debevec using techniques from his UC Berkeley Ph.D. thesis. Using a rough 3D model of the campus campanile tower and some images they took of it, they were able to project the color, texture, and finer qualities from the photos onto the model. It’s a similar process to how you can use an overhead projector to blow up an image so you can copy it in larger-scale on a wall. Debevec baked these projected details onto the rough model, giving him a far better model that he could then have the computer view from any angle he wanted. The technique was so successful - see the movie for examples - that it was later commercialized for use in an obscure art-film called The Matrix.
That kind of image-based modeling was useful, but only if you built the 3D model first and then painstakingly reconstructed the angles and distances of the reference photos you took of the real-life object. Finer variations used 3D scanners to quickly computerize models and sculptures built by artists, which is what Weta Workshop used for some effects in the Lord of the Rings films. The technology even was useful outside of films, with Google using large 3D scanners to quickly model cities (and a Radiohead music video). As with any technologies, it had its limits; John Gaeta tried using the technique on Keanu Reeves’ face, but with disappointing results.
But 3D scanners might not be necessary for the newest wave of innovation. Microsoft’s Photosynth technology (appearing about 2:40 into this impressive video) gives a very rough idea of what the shape of a building is like, using certain targeted commonalities to understand where each photo of is in relation to the next. The object you’re viewing looks like a collection of dots, but from most viewpoints photos will fade in to show you what someone’s photo looked like from that angle. This is useful for organizing photos, but can’t give us anything more than a rough, stuttering fly-around.
Now that we know how to locate where images were taken from and what they’re taken of, the obvious next step is to try and reconstruct a serious 3D model of the object; that’s where this project comes in. The University of Washington’s GRAIL lab - the same group that produced the precursor to Photosynth - has come up with a way to extract finer 3D model information from hundreds of thousands of images, producing a 3D cloud that looks roughly like the real thing! With just a flickr search, a collection of careful algorithms, and a day or so of cluster computing time, they can build an entire city without any human intervention.
It’s just a matter of time before computer scientists figure out how to create a 3D model good enough that you can start directly mapping the image data onto its surface, automatically creating an image as good as the The Campanile Movie out of a bunch of photos you found on the internet. And there are still limitations that stand out as Hard Problems to Overcome: how to deal with radically different lighting at different times of day and night, how to deal with moving objects, and how to make this feasible for sets that don’t have thousands of images to work with. But this newest step still looks incredibly promising, especially if it finally justifies the endless numbers of photo albums on Facebook.
