Pankaj Khandpur : Visual Effects
Tony introduced Pankaj Khandpur, the Creative Director at TATA Elxsi’s Visual Computing Labs saying has been around in this industry since its early days, when industry took hold in India. He has been a guide for many, he has shaped many careers.
Pankaj spoke in his inimitable style – narrating anecdotes and making the audience laugh while understanding the more serious aspects of seamlessly integrating special effects into live action.
“Dhimant and Vaibhav are extremely creative people and I am glad to share the stage with them. I wish I had half their talent. The interesting thing about today’s forum is the agenda has been set up very nicely – we are talking about claymation and classical 2D animation, and I am somewhere in between. What interests me the most is the marriage of Animation/CGI and Live Action. To me, I always prefer seeing everything in its totality and in the real world. It is my personal passion.
We started working on TZP almost a year before it released, maybe a little bit more than that. Aamir, with whom I have worked with since the Lagaan days, called me, gave me a copy of the script, we discussed it and decided the various areas where we would be involved.This is not all of the work we have done, just some of the interesting bits and more relevant to the TASI forum. TZP is a very genuine film, true to the script, unlike most bollywood films.”
He then proceeded to show the raw footage, the shaded model and the final composite for each of the shots:
1) PROLOGUE SCENE
This scene comes at the beginning of the film. Nightmare effects, comes before the titles. The idea was that Amol Gupte who wrote the screenplay and Aamir who directed it, wanted to get the audience into the mood that, this is not your ordinary standard formula Bollywood film, this is something different. We don’t want to tell you what it is, but we want to get you to sit up in your seats and say “What’s going on?!” and I think it achieved its goal.”
“The idea was to keep doing this till you got irritated. It looks like a simple thing but there’s a reason I am showing this. The last frame had more than 250 layers. Remember, this is film resolution – you are talking 2K and above. The reason for keeping the layers separate was that we wanted to control every single layer, every single aspect.The animation was done in Maya and each animation was done on a separate layer – so we could control each and every aspect of it – when to bring it in, when to take it away, when to colour it etc. Every render takes up a lot of time. I’ll share an anecdote with you. During Lagaan, when we worked with Aamir, – the scene where villagers are waiting for rain, they are looking up and finally, one day clouds gather. But it doesn’t rain. We created the clouds in particles, some 8-10 years ago and those renders used to take an incredible amount of time per frame. Everytime we did a render of the clouds, Aamir and Ashutosh would come in and say it looks very nice but what if we moved this cloud a bit here or there. So every change took forever. Eventually I said, let’s render each cloud on a separate layer so only the one that is changed needs to be re-rendered. Separate layers to aid swift changes. And it worked! It is just the methodology. Make it simple for yourself. The objective is that the artist has to creatively evolve himself in whatever he is doing and stop getting carried away with how to execute.”
2) FALLING PAINT DROP SCENE
“In this scene, Ishan is observing, fascinated, a paint drop that falls all the way down and onto his cheek, and then Ishan wipes it off. I was present at the shoot. Aamir, being the meticulous person that he is, insisted that they shoot a real paint drop because otherwise it will look fake, and audiences will be able to make out that it is CG. I said – Don’t put a real paint drop there. First of all, how are you going to drop it in sync with your camera? You can’t control the paint drop! and I am going to have to go through a lot of trouble trying to paint it out. How can you sync a paint drop with the camera? The camera can move only at a certain speed. This seems like a simple shot but there is a lot to it.
The second problem is that the camera is a physical object. It is mounted on the ‘jimmy-jim’ and can move only at ‘x’ speed. So we asked the cameraman to do this as quickly as possible because I knew what was going to happen. He tried it, but it is a giant object, some 16 feet or so, it has weights on one side and it can’t move as fast as you would like it to. So we said, let the kid just mime it and we will add the drop later. We shot it like that, edited it and chose the shot we were going to use. We thought – it’s no big deal, in 3 hours it will be done! But we were wrong.
The problem was that the paint drop would fall at its own speed and the camera is moving 20 times slower than that. The problem was two-fold. If you slow the paint drop to match the camera speed, it will look unreal. The second problem is tracking the paint drop – we were using a jimmy jim to shoot and with its long arm, it is such a precarious piece of camera holder that when you are bringing the camera down, it is constantly swaying and shaking because of the wind and the downward mechanical motion. Our paint drop has to be tracked to the shot. While its falling it is not so much of a problem but the moment it hits his cheek, it has to stay there on his cheek at exactly the same place. It can’t go out of registration. So it turned out to be much more complex than what we assumed because of 2 things – tracking and getting the speed right.
How do you make a paint drop fall at 96 frames per second in a 96 frames shot with a shaky jimmy jim, and how do you make it real is the key to this whole thing. We just had to keep doing it again and again till it looked right. We cheated a bit. We introduced a paint drop later in the sequence rather than from the beginning because otherwise it would have been obvious that it was done in CG.
This is the biggest challenge when doing VFX. I don’t mean to take away from animation – we are doing Roadside Romeo which is a pure animated film. It is a different genre altogether, but when you are doing a live action film, your CGI has to be believable. You should get no awards for effects. Because if you get an award, it means that somebody knows what you have done in the film. and 90% of our work is hidden. We are the unsung technicians (would have liked to say heroes!!) If you are doing creature animation or something like Transformers that is great stuff. It is in your face and it’s cool. But many of the feature films that we work on are meant to be completely credible. If I destroy the credibility for even a single frame, I have lost the audience for good.
In Rang De Basanti all the MIG aircrafts and the airports, everything was CGI. But we were told do not talk about it because it will destroy the believability.”
3) FORMATION OF PAINTING
“This scene comes towards the end of the film. You see Ishan sitting alone and contemplating and you see his imagination transforming and becoming this beautiful child’s version of Sameer Mondal’s painting. Research shows that Dyslexic children typically do not react to alphabets and text in the structured form as most of us non-dyslexics do but they seem to have a great affinity and sensitivity to colour. They can express themselves far better in colour than we simple folks can. I sometimes wish I could do so too.
The point was that this is where we communicate the child’s imagination and how these colours transform into a painting of some sort. It is an abstract thought. The interesting execution exercise here was Rohini – one of our talented Flame artists actually created many steps of this painting. We got the original Sameer Mondal painting and she actually repainted it digitally in about 6 to 8 stages, as though Sameer was actually painting it. Taking colours from the canvas, making sure that they are consistent and true, thinking – this could be a stage in between or that could be a stage in between and so on and so forth.
Then we also used Maya tools. We got some renders done using Maya Fluids to give a liquid kind of effect. We first thought of using smear brushes. The original thought was to run the painting backwards and smear all the colour out of it. That was the most obvious way to do it. We thought, when you play it backwards it will look like the colours have smeared into place and formed the painting. But it actually looked like you are running it backwards. We didn’t want that feel. We wanted it to appear as though it was forming. Things had to look dynamic, things had to change in it. Hence, this route was better. Rohini created the 7-8 stages and we transitioned those from one to the other. We actually handpainted some of the frames in between, used meshes to deform them into the shapes that she wanted and used Maya Fluids to see through them, and refractives to get the feel of water.
We don’t show any work to our directors untill we are happy with it and we are convinced that it is working unless it is a completely different concept, where we involve the director right from the beginning from the concept stage so that they are part of the process. Once we get cracking with it, we do not encourage directors to hang around with us, we don’t want them looking over our shoulders. When we are satisfied, we call them and present it. This painting took just two tries before it was approved, which is a big deal when you are working for Aamir Khan. Specially, because art is a very subjective thing. It is not about creating photoreality where there is only one way possible.”
4) DAYDREAM
“This is a very cute sequence and very quick. The boy needs to get ready for school, his mother is trying to get him dressed, get him to eat his breakfast but Ishan is in his own world, busy daydreaming. So, we just did a quick survey amongst all of us and asked what do little boys daydream about? I came up with the engine – as a kid I wanted to be an engine driver – almost all little boys want to grow up and become an engine driver.
This is a very straightforward shot, all of us picked an element each, Aamir was clear he wanted a bird at the beginning. For a split second the audience must do a double-take, thinking it is a real bird. A second later, they can realise it is fake and it is animated. Then we had an argument, of course, we always have arguments, that the kid should be specifically looking at something so that when we place the objects in front of his eyes, it looks like his eyes are following the objects. Aamir’s point was that it is a daydream, he is not looking at it in real life, he is looking into infinity. These elements are supposed to represent his fantasy. There was a cutout – a stand-in we used so we could block the little boy correctly, which was removed at the time of the final composite. Bird, train, dinosaur – each element had to be different. It is a memorable shot in the film. To me, it is representative of the film – of what the film is about. We got this right the first time itself.”
5) BHEJA KUM SONG
“It was an interesting song, we wanted to treat it differently and so Aamir called upon one of the most imaginative ad film makers in the business – Ram Madhwani, who we had also worked with in the past. Aamir wanted him to direct the entire song and left him absolutely alone after they had their preliminary discussion, of which we were also a part. What was the song supposed to communicate? It is about the teachers ganging up against this boy and it is really from the boy’s perspective. This child, who is actually extremely intelligent and very talented, imaginative, is being treated like he is stupid. So this is the terror the child experiences from the world around him, it is from his perspective.
We brainstormed on what would a child feel when he is threatened, when he is tortured, when he is hassled? We all took a call in the beginning that we have to show the alphabets actually going crazy. Till this point in the film, we haven’t really seen the child’s point of view, except for the opening sequence – ‘Fail, Fail, Fail, Fail’. Apart from that, we don’t know as yet what the child feels. The layperson in the audience doesn’t know about Dyslexia, or has a different opinion about what Dyslexia means, and so far we have built a case where all the teachers say this child is stupid, the parents are saying he is stupid or crazy. Now, for a change, let us explain to the audience what this child is going through.
We thought we must start with the alphabets because a Dyslexic child basically reads wrong. There is nothing wrong with the child, it is just that he is unable to read structured text properly. APPLE can become LAPPE or APEPL. He must get the feeling of being attacked. First with letters then numbers… it goes on piling on till the point that the little boy is overwhelmed. This was the graph of the song.
The text book was filmed with a little bit of text. We thought we would mix it with CG text. As an afterthought, we felt that all text should be CG so we had to take out the text that was filmed. We then regenerated all the text in Maya. We tried zillion combinations of text – big text small text, mixed. Simple, straight-forward animation but very effective in conveying the message. If you look at the graph of the song, you realize that there are lots of little things in the song that convey the terror experienced by this child, to a point where he actually scratches his nails on the blackboard. Text flies out of his books, teachers put big red crosses in his books. We also did a spot poll in between asking what scares young kids the most – creepy crawlies. Most of us even today, as grown-ups hate the thought of cockroaches crawling over us and spiders and snakes. So we decided to end the song with the snake coming out of the blackboard. In the original shot there is an in-camera zoom on the very last beat of the song where the snake appears. It gives a nice punch to end the song with. It was just 18 frames so we could get away with anything.
Remember the golden rule – the longer the shot is, the more chances are that your going to get caught. The trick is to make it very short, you can get away with things.
Another tip – try and embed all your CGI in live action. Because, if there is some live action, chances are that the audience will be fooled into believing that the whole thing is live action. Unless, of course, it is something like this shot – cobra coming out the blackboard – something which can’t happen in reality.
In Jodha Akbar, we created CGI soldiers but we knew we could never pull them off unless we use them on live backgrounds and we mix them with live actors. Like in magic there is sleight of hand, in VFX it is sleight of eyes – how to fool the eyes.”
6) SPIDER ALPHABETS
“Mandar (sitting in the audience) did a lot of the concept art with me. We said let’s put the ultimate horror for this child. But by themselves the creepy-crawlies are meaningless within the context of the film. We had decided to do creepy-crawlies, spiders, caterpillars, lizards. make a whole army of creatures. But how do we link them to the concept of the film. We decided to make their bodies with letters of the English alphabet and numbers. Aamir of course made it clear that we must use Hindi letters too because the audience is not necessarily English speaking. So it was decided that we would do 26 insects with English letters, 33 with Hindi and 10 with numbers. We now had our army of creatures. But when we actually got to creating them, they didn’t look like letters and numbers when you put them on the back of a caterpillar or the body of a spider.”
Pankaj then showed the concept drawings. “If you notice these concept drawings, not all of them are very clear, which meant that the entire concept would be lost. People won’t understand why these creatures are coming there. We needed to choose letters and numbers – Hindi or English, which would be clear. The other thing was that we should not do multiple creatures. It is always a swarm of the same type of creatures. This is something we had overlooked initially in our excitement to create various creatures.
The result was strange but interesting – it looked like an animated horror film. We took a call – which of the insects looks most creepy-crawly and arrived upon the spider at the concept stage itself. Our modellers then created the spider with Hindi letter ‘ka’ in Maya. It looked nice and frightening. Then we hit our second bottleneck. Only the ‘ka’ was working. Nothing else was as obvious as we wanted it to be. So we had to weed out a whole lot of letters till we got the ones that work.
Rope Trick!
Ram shot the raw footage in Panchgani – We used very fine nylon threads – known in filmi lingo as “Tangoos”. It doesn’t show and is easy to paint it out. The bag had to open in the shot and creatures had to crawl out. So the Tangoos was tied to the rim of the bag and someone from outside the frame had to gently tug at it to make the bag open. If we were a Hollywood studio, we would have meticulously created the Bag in CG using a mesh and created the effect and charged a million dollars for it! But I believe in not wasting time and effort. The Tangoos trick creates the effect accurately and quickly. The leaves around were moved by prodding them with a stick which we later painted out. We finished the shot in about 3 minutes – in just one take.
We mixed some physical effects with our CGI and achieved satisfactory results. Then we timed the movement of the spiders accordingly. We had been working on this for a long time – close to 2 months, Ram was thrilled because he wanted it to look as close to reality as possible. I wanted it to be subtle. But Aamir felt that one couldn’t tell that it is a letter of the alphabet, atleast not the first time round, perhaps after looking at it 10 times maybe! We had a big argument – with Ram, me and the entire animation team on one side and Aamir on the other. We called people in who had no clue about it and screened the entire song for them. Then we asked them “Did you realize the spiders were made of alphabets?” and not one of them got it!! So we ate humble pie, put our heads down in shame and said to Aamir, “You were right, we were wrong, as usual!” It will look CGI – but then it is an unreal scenario to begin with anyway. The body was made a little brighter and more yellow – and it became very clear
Pankaj showed the Concept Art, Wire Mesh, Shaded Model, Raw Shot, Original Spider Composite and the Final Composite for the sequence.
Peer Pressure!
“I must say one thing – most of the work that we do is actually for the appreciation of our peers and not for the audiences. You do it because your peers will understand and appreciate what you have done. And they will know how difficult or easy it was. If you are in this part of the business that I look after – which is Visual Effects, you won’t get too many awards, because the guy who is judging you has no idea what it took to do it. He has no idea of how difficult or simple because he thinks it is real. And you have done your job.”
Next, Pankaj screened some of the sequences from the BHEJA KUM SONG.
“When the spider is crushed, we had to have some life juices coming out. Ram didn’t want CG, he wanted real juices. He said he would give some references. We finally decided we would get some real cockroaches and kill them and shoot that. I hope there are no animal rights activists or people from PETA here in the audience! Some poor cockroaches were banged to death. By the way, none of us went for the shoot! VFX supervisors don’t go for cockroach killing shoots. This is a production job and they should handle it. When we tried animating this, guess what? Ram was right! It did not look real, it looked fake. We ended up using the real goo that was shot by them for reference. As a result, it looks real and yucky. Remember it HAS to look yucky – that is the objective of the song.”
Flushed Away!
He then showed the next bit of the song, where Ishaan is trying to flush the insects down the potty! The thought that we had was – if you ever try flushing a creepy-crawly insect down the toilet, it doesn’t get flushed away. It climbs right back up. They somehow manage to float and survive. So we put that thought in – very few people got the idea. But we just wanted to put this in there – lets just flush down the spiders in the potty. Everything that we put in there is deliberate – nothing by chance. A lot of thought went into each and every element of the song. A lot of planning was done. Each thing has to press its own button in the viewers mind.”
Next came the questions:
Q. Any Tip or guidance for aspiring animators/VFX artists?
A. Only tip I can think of is Passion! You need to enjoy what you are doing or else don’t do it. Animation and VFX to me is one universe. These are not separate – as institutes and some other people put it. An artist is the same. But it is not for the faint-hearted, so if you don’t have the passion, you are looking for safety, my advise is join a bank.
Follow your instinct. When you do something, if you feel first off that it is not right, then it is not right. Don’t attempt to sell it to yourself and to your client. Don’t keep seeing it because the moment you see it more than twice, you begin to believe that you have done a good job. I have been in this business for 28 years and I have been an artist for 20 of those 28 years. I know that sometimes you are working on something and it kills you- you work on it day and night, skipping sleep, not going home, trying to finish whatever it is. Finally, when you are done, you pat yourself and say what a job! Actually, it is probably not very good. But you have done it yourself, so how are you going to judge it and say this a great job or that it actually needs to be relooked at and reworked? You can’t – because you have invested too much time, sweat, blood and tears into it. So develop the instinct to look at your work coldly. If it doesn’t work the first time, no matter how many times you look at it, it won’t work. Go back, figure out what’s wrong. That is where experience comes in. Fix it. And then it might work.
Q. If you see the original shots and the composited shots, there seems to be deliberate camera movement. Was it so?
A. Actually that’s not true. There are shots where the camera moves, we try and track it. Because if you are attaching animation, CGI, anything that is outside the world of live action, you are tracking something that is rock solid and steady onto a piece of film which is moving, even if your camera is static, (this is something which the students here may not know) the film that is running through the camera is moving. Even though there is this locked camera, the film is actually beating back and forth, 25 times a second, if it is not a pin registered camera. Now what happens if I place an object in the static shot, it appears to be moving. Actually it is not the bottle that is moving but the film but it indicates that something has been done to the shot. So we need to lock it. Then we track it. If there is enough detail in the shot we can track it and lock it and actually use it as moving footage. Because, if the grain of the film doesn’t move, you can tell that there is some effect going on. The grains of film move imperceptibly, so if it doesn’t move, you know its a freeze which has been used. In desperation, when things don’t work, we resort to a freeze, add grain back, make the grain move on a freeze, so it looks like a moving shot, and then put it together. Generally, if the camera moves very quickly we can track it, but if it moves very slowly, we’ve had it! Because all the tracking errors get slowed down so much that the tracking software can’t catch them easily as it doesn’t know whether it is a pixel shift or there is indeed a camera move.
Q. When the paint drop is falling, you haven’t made the geometry of Ishaan’s face. Wouldn’t it have been easier for you to keep the paint drop on his cheek, when the camera is moving?
A. No, because I would then need to erase the paint drop from all the frames until the drop actually hits his face. We did shoot a few frames with paint on his face, after the main shot was complete, but for reference only. Remember, it is running at 96 frames – which means for every second of movement I have 96 frames to work on. So my artists have to work on 350-400 frames – while the camera is going down towards his face, they have to paint it out. The choice was between making my artists go through 400 frames of painting out the drop frame by frame or do using CG and pull it off quickly. I made my choice for which my artists were very thankful :)
Also, remember when you are doing this kind of rotoscopy, there is a lot of effort required to make sure that you don’t make a mistake. Especially when you are dealing with skin. Any change in skin texture will show. What am I doing when I am painting the drop out? I am actually painting skin back on! There is no such thing as painting it out. We get a lot of film footage with these major stunts and stuff where the wires are all over the place, going over the actors face. They think it is just a matter of pushing a button and getting rid of the wires. That is nonsense. One of the most difficult tasks to do is wire removal. Because there is no such thing as wire removal. You are putting back a bit of the background, a bit of the character. It is additive. I have to paint back whatever was behind the camera for every frame. It is a LOT of work! We try and control it as much as possible – by asking them not wear patterned shirts, avoid the wires crossing face, if it crosses, then try to do a retake etc.
Keep it as simple as you can. The simpler you keep it in the shoot, the simpler it will be in post-production. Particularly in Indian films, where you have 2 months and say 800 shots and you are working day and night, to try and deliver, there will be a few mistakes. One shot goes wrong and people will catch it on the big screen.
In the case of Ishaan, we did not need a mesh. What we did was we put a sphere – because his face is round. We just had to put one curved mesh. We tried all this but it was not necessary. Remember, simple is better. So finally, in keyframes, it worked. We splattered it, did some squash and stretch. At that size, that distance and that speed we could get away with it. It worked quite well.
Q. (Tony) In all the visual effects work that you showed us, how much of it was actually boarded before or was it just discussed and executed?
A. The ‘Bheja Kum’ song was story-boarded because it was a full sequence. The rest was pretty straight-forward. We start story-boarding when you have more than 5 or 6 shots in that scene or sequence, then you need to story-board it. You need to understand what is the geometry, what is the magnification you are working with but if it is just 2-3 shots or it is one shot like Ishaan’s daydream with the train going around, there is no need for boarding.
Q. In a film like this, how much of a difference does the visual effects budget make?
A. Let’s assume the number is X. What happens is – this is one of the biggest problems – you are always quoting for the work that you think you will need to do. You get the script. You break it up into the number of shots that you think you will need to do You run it past the director. You chat. You brainstorm. And you believe this is what needs to be done. Based on that you figure out how much it is going to cost.
Unfortunately, this industry follows a system of seconds. One second of what? One second of wire removal is going to be different from one second of claymation which is going to be different from one second of hand drawn animation which again is different from building a transformer set. So what does 1 second mean? It is a meaningless unit. But most producer understand only these simple things like seconds. The actual costing is the effort that goes into it. How many man hours, man weeks, man days are you going to spend doing this piece of work? So you arrive at a cost and you kind of distribute it over the work so at least the producer has the simple unit and you have the complex unit at the back end and they match.
Then you start the shoot. Depending on the kind of film, complexity, budget and the actors’ dates, anything from 3 months to 5 years later, the film is complete. And you are seeing the edit for the first time. Suddenly, almost 100% of the time, the requirements have drastically increased. There will be the original list of what needs to be done plus many more new requirements would have cropped up. This happens with almost every project that we take up.
There is a huge discrepancy in what you budget and what you actually deliver. The trick is to explain and work towards a model which says, this is based on the budget and the estimate, and when I see the final edit, I will give you a more consolidated a more real cost. Most people don’t buy it. I’ll tell you why. Film production is like a party. By the time the shooting is over, the money is over too. Unnecessary expenses and wasteful habits means there’s no money left for post-production. The director/producer wants you to work with the old quote that you gave. But there is so much more work now, so many extra fix ups and we don’t necessarily like doing those fix ups. But we believe that once we are in a project, we do whatever it takes. Whether it is cleaning up a blemish or tyre track or anything else. It’s hardly creative work but it’s difficult to do and time-consuming.
In Hollywood, it is very clear. They understand that you have given a bid today and this bid is liable to change after the shoot is done. In India, first they will take a bid, negotiate with you, pay you only 50 percent, try to negotiate for the other 50 percent and eventually give you 200 percent of the work. That is typically what happens. But things are changing a bit. Some of the professional film makers follow this model.
What we do is true visual effects – what we call editorial visual effects, which means stuff is slipped into the edit which you are not supposed to know about. I am not talking monsters, creatures or transformers. I am not talking the particle guy in Spiderman. That is obviously an effect so you will even get awards for that. But when I am doing editorial effects or narrative effects where things have to be hidden into the story and you are not supposed to know, it is imperative that you don’t know. It will destroy your experience of watching the film if you do. Let me tell you, for most Bollywood films today we do at least an hour of effects, and some of them are love stories. And people ask, Love story? 40 minutes? What did you do in the film? and you reply – oh, nothing, just some stuff here and there.
Karan Johar’s rain, snow, water is always CG. Because it has to be just right. So you do it in CG where you can control it. I want my rain to move left to right on this note of music. Try telling God that! But that’s the director’s sensibility. He wants it – he believes that works for him. But now, what am I going to do? Tell an award company that I made the rain move like this? Why does he need to know it?
In Rang De Basanti, they are all sitting on the fort and looking down at the Indian Air Force Base and you have suspended your beielf for that moment, you think it real, right? I actually went to a service officers dinner where I overheard an argument where they were saying “Yeh Ambala airport hai, ki Srinagar airport hoga?” But the truth is there was no airport! Which means I did my job well. Which also means no award!
The audience rewarded Pankaj with a big round of applause!
Making of TZP – Part 1 : Dhimant Vyas – Clay Animation for Title Sequence
Making of TZP – Part 3 : Vaibhav Kumaresh – 2D Animation for 3×9 Sequence