Performance capture
Performance capture differs from standard motion capture due to the interactive nature of the performance, capturing the body, the hands and facial expression all at the same time, as opposed to capturing data for reference motion and editing the motions together later. Since animation is defined as the optical illusion of motion, and performance capture and similar techniques are based on the live action movements of the actors, it has been suggested that this form of animation is called live motion. It is becoming widely used for movies, sports analysis and games. In effect it is a digital replacement for the furry or latex rubber costume, allowing the actor to give the performance without wearing a heavy latex suit, as demonstrated in the latest Pirates movie [1] where the actor was filmed using a combination of high resolution sources and their face replaced by digital masks, on the actual set rather than later on a special Mocap stage.
Besides being used to create photorealistic non-human characters in live action movies, it can also be used to make photorealistic human characters in CGI movies. Some may claim it would have been easier to do this as live action, but that is not necessary the case. For instance, it makes it possible to see how a character will age over the decades in a realistic way. Or if all or most of the movie contains surroundings which is not found in the real world, like alien cities in sci-fi or impossible forests in fantasy movies, everything around the actors would these days usually be made on computers anyway. The actors would then either have to act in front of a blue screen or using performance capture. If performance capture is chosen, the director is free to film in any angle that is desireble and in ways impossible with a physical camera, and don't having to worry about the light or other things on the set that is importent if filmed with a conventional camera.
Whether the audience is the director, or a live television performance, it is an ensemble Performance, rather than a list of motions that will be edited together later. Performance capture allows the director or audience to see the end character interacting with the environment to fix eye contact, timing, reaction and other issues that can effect performances with other characters that may or may not be filmed at the same time.
The actor usually interacts with models of the objects in the scene. The recorded performance data can be used to animate different actors. In The Polar Express Tom Hanks played five roles - an 8-year old boy, his father, the train's conductor, a hobo, and Santa Claus.
Using advanced hardware and software, entire scenes can be animated in real time to allow characters to be viewed in digital sets. This unique capability allows Performance capture to create real time animatics. It is expected that the success of animatics in Lucas Films "Star Wars" and other projects will eventually lead to an entire industry segment where all movies will be previsualized before they are greenlighted.
Performance-driven animation technology is an advanced imaging solution for animation and visual effects in the feature film, video games, interactive toy and communication industries. In some setups, the actor performs to the camera on set, with no special suits, markers, motion capture cameras or complex stages, with a tradeoff of minutes or days to see the results.
For Active Appearance Model, Principal components analysis or Eigen Tracking to work, Cameras need to see the desired movement approximately every 30 degrees or the actor must perform to the camera or the software will not be able to see enough information to process the image. Automatic software tries to track the facial features or psuedo markers, or hand tweaking provides the desired data, but is time consuming. Other combinations use traditional motion capture technology for body, and image capture and processing for the face. Hands and fingers tend to be a problem for all technologies as they are easily occluded by the other hand, and clothing and rotate through many angles that are difficult to track. Even at 5 megapixels, from 5 meters away to see the entire body with a 30 degree field of view lens, a single pixel is 2 mm limiting the resolution and accuracy.
Using Active marker systems, the characters can be animated in real time so the director can see the CG character and get a performance that is interactive, rather than waiting weeks to see the data, hense the term 'performance capture' The real time character driving is accomplished by modulating the active output of the LED to differentiate each marker, allowing several markers to be on at the same time, while still providing the higher resolution of 3,600 x 3,600 or 12 megapixel resolution while capturing at 120 (128 markers or four persons) to 480 (32 markers or single person) frames per second. Active markers allow real time tracking at a lower price without the problems of marker swapping or heavily filtered data. This higher accuracy and resolution requires more processing than older passive technologies, but the additional processing is done at the camera to improve resolution via a subpixel or centroid processing, providing both high resolution and high speed. These motion capture systems are about 1/3 the cost of passive systems. The key advantage besides higher resolution and data quality is low latency for Performance capture. Additionally the active markers reduce data cleanup times by a factor of ten over older technologies.
Software solutions often apply to every style of animation from basic 2d through 3d CG, to photo-realistic visual effects, including re-animation of real actors. The animated character can be anything from a photorealistic human to a cartoon creature. Examples of performance-driven animation technology include the facial animation for Rockstar’s “Table Tennis” game and the “Grand Theft Auto” series beginning with “San Andreas.”
Although performance capture has been used in some earlier films and computer games, The Polar Express was the first movie made solely with the process. This film was directed by Robert Zemeckis, who had a long history of technical innovations in filmmaking (historical composites in Forrest Gump and the combination of animation and live action in Who Framed Roger Rabbit) and became a self-professed fan of performance capture (he produced the 2006 thriller Monster House made using the same technique) because of the creative freedom it gives the director. Zemeckis is currently using performance capture in an adaptation of Beowulf scheduled for 2007 release.
Often cited as "performance capture" is the Lord of the Rings character Gollum. There is some dispute on this subject, as an the cleanup and adjustment of the data took over 100 artist years and the final result was more of what Peter Jackson wanted, than what the performers originally did to create Gollum as well as many of the background characters. There have been many publicity stills showing Andy Serkis in a motion capture suit, with dots on his face; however, many scenes were keyframe animated at Weta Digital using Serkis's performance as a reference. Artists would animate on top of film plates of Serkis, using the human eye instead of the computer to capture the subtleties of his performance in an effective but time-consuming process. This method is often referred to as "rotomation," and is a CGI form of the traditional animation technique called rotoscoping. Serkis did this again to play the title character in King Kong (2005). Another example is the character of Sonny in I, Robot (film), played by Alan Tudyk.