Buzz in Hollywood and various filmmaking groups around the world is about performance capture, real time previz, virtual film production technology and more recently in India it became a talk of the tinsel town. No doubt it is one of the new mantra for modern film making or new media globally. Performance capture technology is much matured nowadays and many are embracing it in their productions. Will using this technology make one part of a new breed of filmmakers? Is this mantra for you to become a future proof filmmaker? Many still have a dilemma from the unknown to experiment it. Thus, I thought to let you know the mantra before exploiting this modern tool effectively. You can easily find it on the World Wide Web, but where to start from is the question. This piece may guide you to with the right choice.
Performance capture (motion capture) as we call it now has been around for many decades in the entertainment industry. As we all know necessity is the mother of all inventions, motion capture was invented essentially for the field of medicine to study patients’ illness. Motion capture slowly spread its roots into military, space science, entertainment and so on after that. In the entertainment industry we used it to quench our creative needs and attempt in achieving the unimaginable in competitive media. It was generally called motion capture technology and we gave a little glimse of what can be attained with ‘performance capture’ while producing ‘The Polar Express’. This was the first time we managed to capture the full performance of an actor on a single stage by combining body, finger and facial movements captured together. Thus, according to my hypothesis, since we record complete performance of the actor and not just simple movements or motion, the right terminology for this modern tool should be ‘performance capture’ as against ‘motion capture’ which most filmmakers prefer to call it. So where does India stands on this path of ‘performance capture’ technology in movies?
India is not new to ‘motions capture’ technology. In fact it has been used long back in the Indian entertainment industry varying from animation movies to feature film VFX. India is also credited with the production of the world’s first full motion captured movie (Sinbad: Beyond the Veil of Mists) way back in the year 2000 and subsequently more features followed using the same technology.
India’s first live action movie which utilised motion capture was ‘Jeans’ in 1998 for some VFX portions. ‘Kochadaiiyaan’ (2014) was credited with India’s first full performance captured movie. Around the world motion capture has been used long back in movies since the time of ‘Total Recall’ in the year 1990. The pioneer in full motion capture photorealistic features was ‘Final Fantasy: The Spirits Within’ released in the year 2001 and world’s first full performance capture movie was ‘The Polar Express’ in the year 2004.
In spite of having little history with the motion capture in India, we experiment with this technology almost a decade late in every milestone achieved internationally. But we clearly see this timeline may come down in the future of filmmaking. Although technically it was difficult during that time, we never stop experimenting with it until we find a perfect blend between creativity and technology. Hollywood successfully created memorable performance captured characters like Gollum, Kingkong, Davy Jones, Hulk, Azog(The Hobbit), Navi (Avatar), Caesar( Rise of the Planet of the Apes). Gollum won the ‘Best Virtual Performance’ award in 2003 MTV Movie Awards and was placed thirteenth onEmpire’s ‘100 Greatest Movie Characters of all Time’. Many even debated if these characters were worthy of an Oscar Award for the best acting category. Now we have reached the age of digital technology where any computer generated (CG) characters can be reproduced at a photo realistic level which can be indistinguishable from any live images.
Motion capture has been glorified in the field of entertainment, but it becomes a regular method for many media productions. Video game companies do hours and hours of performance capture to complete one AAA game title. In a common parlance, motion capture has been imagined as people wearing special bodysuits with many white dot markers on their body in a specially designed studio room, but this is true for only certain type of motion capture technology, the one that is commonly used in entertainment is passive/active marker based motion capturing technology. What many of us might have seen is a reflective marker ball which is stickered on top of a tight suit made of lycra. These markers are captured by a special infrared camera that record these markers on a world space and creates 3D data.
Like any other product, this also has many technology focuses on how data has to be captured using various scientific approaches. Some of them are marker based (passive and active), marker less, inertia technology, magnetic, mechanical, depth sensor etc. In a way Xbox Kinect motion sensor camera is a motion capture device at the consumer end.
Over the years we found, what is used for body capturing technology may not be sufficient for facial capturing since it is one of the complex move in the body using muscles. Facial expression also contributes in connecting audience emotionally to the character. With the past experience from experts, one of the reasons why CG characters terribly fall into uncanny valley phenomenon was lifeless expressions. So the new technology replaced in the industry recently and successfully proven in ‘Avatar’ called markerless (some use colour ink dots) facial capturing system using a head mounted camera (HMC). It is possible to capture every emotion of the face faithfully and reproduce them digitally.
So the question of why these performances need to be captured using technology arises over traditional approach? Performance capture leverages the new medium of acting as well as storytelling for creative thinkers. Unlike olden times where actors had to wear hours of makeup on their face which didn’t move like regular skin and made it a very cumbersome process. The limitations were many for olden day makeups, example it is limited to the height, form and body mass etc. Moreover emotional performance is limited in the facial part since heavy load of prosthetic make up cover up the actor’s facial performance and thus the connection with the audience is sacrificed. Now actors can perform any character they desire without the boundaries of their physical nature. Emotional connection is well possible in this technology since every micro level of emotions are captured and interpreted in the digitally created character. This in a way boosts the capabilities of actors and enhances their acting skills as well.
Filmmakers’ creative power is unleashed with this technology by dreaming beyond the limitations. Directors of the modern day are thrown with many challenges in combining several things. In Hollywood most modern and renowned filmmakers choose performance capture or virtual film production technology nowadays. They don’t like to be detached from the heavy multi stage process of the CG world. Earlier directors would use the performance capture and stage the CG character to be added on the live plate in the edit later. Modern day filmmaking is more complex and involves many CG shots. Directors would like to see the result on set in real time so they can better visualise their composition. With the help of modern motion capture technology they are able to see digital characters and directly interact with live actors on the set. It seamlessly integrates into regular live action shoots and that is where every creative filmmaker’s dream comes true where they have a control on CG in real time.
When it comes to usage of technology, most people have reluctance in knowing it. My advice for those creative makers is to forget about technology and concentrate on the creative side of it and technology will be taken care by the expert teams, who specialise in those areas. All you have to do is to take your creative vision to the right experts. A suitable story, which is well crafted for the optimum usage of this very technology, will enhance the final result. This should be a medium of choice and it should be well justified for the story and not to be used for the sake of it. If it is used with the right intention and desire to create a visual spectacle, then it works beautifully. Otherwise it will encounter tremendous loss in the usage of technology, cost and time included.
Sometimes multiple motion capture technology is required to be used for same production or even for one single shot. So planning becomes crucial when one decides to use it in the feature. It is advisable to use it in the multiple shots rather than for single shot. Since the cost of production would be justified for the longer usage of motion capture. If one uses it for hours of performance capture the production cost will witness a huge spike as against a regular live action shoot. But at the end it should be justified why the technology was used in giving a different visual style to the film.
Performance capture/ motion capture can be used for many purposes. Here are some possibilities that one can use it for. A full length 3D movie can be made with performance capture technology and it can be a cartoon animation style like ‘Happy Feet’ or photorealistic style like ‘Avatar’ or mix of both like ‘The adventures of Tintin’. It can be used for live action movies in variety of methods. Motion capture can be used to capture the complex action of actors and applied to a digital double. This is usually where shots are complex or dangerous to achieve in real life, in a movie like ‘Matrix’, ‘King Kong’, ‘Gravity’ etc. But the thing to note is these are merely used for digital replacement of actors and it doesn’t involve any performance capture (mainly facial animation).
It can be also be used as a main casting actor as a performance capture character. As a protagonist or antagonist which has significance in the movie for acting. Usually it will be of different look other than human form. Hollywood nowadays creates number of memorable successful characters in this form. Usually it is not advisable to recreate performance for live human actors as it is in digital human form. This has various reasons and mainly due to ‘Uncanny valley’ effect which was a theory for android scientist and also applies to CG characters. This means that more and more of realism that one is trying to create in human characters, it pushes further away from the realistic feel or eerie feel towards that character. It is a psychological feel that we create hatred towards it or trying to find a fault in the character.
Andy Serkis who pioneered in performance capture acting playing ‘Caesar’ in Rise of the Planet of The Apes also feels the same and during his interview with ‘Popular Mechanics’ magazine said: “…But where it falls down, I think, is when you’re creating a lifelike human being. At the moment it seems to be kind of a pointless exercise, making an actor look like themselves in a very realistic way. There’s always a disconnect…” But technology growth is promising and we expect in few years it could be possible. Until then it is advisable to keep away from it unless some strong reason to do it.
Motion capture can be used for facial performance capture to retarget the performance of the live actor to different look of digital face like in the movie ‘The Curious Case of Benjamin Button’, ‘Tron: Legacy’ etc. But each has its complexities in the way or method of approach and it completely differs from each project.
Other areas where we can use motion capture are in the pre-visualisation stage. The complexity of visual effects is more in the recent movies making it is difficult for the filmmakers to clearly visualise the end result before the final out. With the motion capture technology we can previz the shots using a human to represent a 15 foot tall creature character and a virtual camera to fix angles and so on.
All this are combined together and form a new breed of filmmaking called virtual film production. It started earlier with the digital backlot concept and now it is taken a main stage of most visual effects films by combining motion capture technology to it. Here we can use primarily all CG elements in a real time against the live composition. It can be CG character to sets and live actors can interact with much ease. All this is possible with the various motion capture technology for representing various needs. Virtual film production is a real boon for directors whose visions are unlimited.
After deciding to use motion capture in the pipeline it is important to find the experts in the field of motion capture/ performance capture. Since experience does matter in bringing quality data of your capture elements or characters. There could be a situation where your captured data will be unusable because of less quality or noisy data. Most people think once we capture the data, all things are set perfectly, while it is not true. There needs to be a lot of preparations before and after the capture to get a quality data that can be used in 3D production. The quality differences can be seen in various projects from gaming to movies. Example the quality of gaming motion capture is different than in movies etc. It is also important to check the data after every shot and make sure it has come out the way you desired. The expert team usually does all this as a mandatory checkpoint and keeps notifying the creative team on the progress. Nowadays most studios offer real time visualisation of your 3D character and virtual camera on the set for the directors to have a better control on the shot. But it is limited to the technology that how much data that we can push to realtime, so it is better to discuss with the studio experts on your requirements.
As a creative decision it is important to choose a right actor for the right character. It will show in the end result of motion capture and you cannot do much with it to correct that in animation editing. Example you can’t use a dance specialist for some martial art action shots and action specialist for some performance acting. If the creative crew (director and actors) is new to the performance capture pipeline, it is better to get some rehearsal to understand how you can exploit the technology to a maximum. This will help the team on reducing time during the actual shoot. There is always a chance of fixing some animations during the animation edit using the animators. But it is important to get the maximum result from the performance capture during the shoot. Some points to be noted are the assuming all the occlusion of the characters costume, size of the character, interaction props, distance between characters, casting all main characters in the same shot are all very important aspects of avoiding later fixes that eat invisible cost. So there is a tremendous amount of pre-planning needed for filmmakers along with the technical team. One thing that is needed in this pipeline is patience from the filmmakers. Since what is usually ridiculously easier in regular live action may take longer time in this process. But what is amazing here is the director can play God, since the environment is digital, he has an absolute control over the look of the shot.
India is one of the largest film producers in the world and increasing acceptance of our films in many countries and international award shows mean that our boundaries are growing. At this level we are forced to deliver quality content to the world which has to be on par with the international standards. With the performance capture we can break the boundaries of race and region. Performances previously restricted only to physicality and look will disappear. With the right story it infact makes things possible which were not in a traditional method and thus by making it a cost effective mode of production. Indian audience is now opening up for a new genre of films and this means the supply for this demand has to be equally met.
Digital age of filmmaking lends their hand in every stage of productions by replacing old methods. Like any other new technology equipments, the prices may come down gradually as the number of users increase. Performance capture has already come off the studio volume and I hope it will be more flexible to use it in the future. More users of this technology means, we can see many genres made in this format including drama, comedy etc. We can witness a marriage between live action films to animation and to gaming as well. The thin line between these mediums will disappear in the future and there would be a day when a bunch of filmmakers assemble in a sophisticated room and choose their digital characters, performance, sets and camera angles to direct their own virtual world of storytelling for the audience. As long as the new technology keeps evolving, one day we may not be able to tell the difference between ‘real’ and ‘unreal’. Who can embrace it earlier has the power to withstand the future. I leave the new mantra to the hands of filmmakers who would like to join the race to the future.
(These are purely personal views of Centaurdigital founder Lino Stephen and AnimationXpress.com does not subscribe to these views)
About the author
Lino Stephen is a founder of Centaurdigital, which specialises in various aspects of performance capture including real time facial performance capture, digital humans creation etc. He has many credits including a technical director for Kochadaiiyaan, The Robot etc. and provides solution for movies, games and new media productions.
