Steve in Half Bad as our VFX team starts to rip him apart.
I often describe what we do in visual effects as recreating the most primitive version of reality that we can get away with. Thinking that you can recreate reality fully is insane. Don’t even try. That’s not the game we’re playing in VFX. We’re creating the illusion of reality. That’s a whole different ball game.
We study reality closely, learn the rules, and learn how to break them. That’s actually one of the foundational skills of VFX, but it’s difficult. It takes a while to learn and it’s time-consuming. So, we’re also always on the lookout for any way to shortcut it.
One of the main shortcuts is by stealing from reality any way that we can. Any technique that allows us to rip something from the real film set and beam it up into our computers can save us hundreds of hours of work.
So without further ado, here are some of my favorite opportunities for stealing from reality. Some obvious, some less so, but all are able to be executed even at a beginner level of VFX.
Motion capture and body tracking motion capture is one of the most fun and exciting ways to map reality into the digital sphere, and it’s more accessible than ever.
There are free or cheap options with video-based AI-generated MoCap or for more freedom and flexibility, you can get a Rococo Moca suit for $1995.
MoCap is pretty plug and play, but getting your results into the shot isn’t. There’s the extra step of retargeting and cleanup
If MoCap is the sexy athletic boyfriend of capturing motion, then body tracking is the boring but reliable husband that is always going to stick around. It basically involves taking a digital version of your character and animating them by hand to line up with an actor that you filmed. It’s pretty boring, but it doesn’t require much animation skill because you’re just tracing (Rotoscoping) with a 3D model, and it’s done all the time in professional VFX.
Ripping faces and now freezing them, our team went all out on Half Bad!
The example image at the start of the newsletter and the one above are two examples of me and my team stealing from reality for the Netflix Show, Half bad. We took digital versions of the actors’ heads. We tracked those to follow the filmed actors and then we manipulated them to get the final result, whether it’s covering them with ice or ripping their faces apart.
And how to get the digital versions of the actors heads?
Well, that’s something else that we can steal from reality using 3D scanning. We can very easily scan actors on set and recreate them into our ‘digital fake’ of the real world.
There are a few ways of doing this:
1 - Photogrammetry: This creates scans based on photos, and it’s the most popular technique for 3D scanning. You just take many photos of the object from as many angles as you can and boot it through the software. There are great free options for doing this like Meshroom, although my go-to is RealityCapture for pretty amazing results.
2 – Lidar (light + radar = lidar): It’s basically scanning using a laser, and the technology is built into the latest iPhones and iPads. So if you have one of those, lidar is a great option.
Photogrammetry is normally my go-to approach for scanning, but lidar is fast and simple. You just run the scanner over the surface that you’re scanning and you’re done, perfect when time is tight on set.
3 - NeRFs (Neural Radiance Fields): How NeRFs work is well outside what my brain is capable of understanding. It’s leveraging neural networks to create 3D scenes, and like photogrammetry, it builds environments based on data captured from multiple camera angles. Unlike photogrammetry, its basis is machine learning, and this means that it can calculate viewpoints not visible during the actual training process. And it gives more flexibility for making changes to the environment in post-production.
I generally find machine learning-based processes too inflexible for professional visual effects and have yet to fit NeRFs into my visual effects company’s workflow. But it’s something that I’m going to explore more in the future.
If you want to explore them yourself, there’s a great article here and the Corridor Crew did a fun video on them here.
There are plenty more ways that we can steal from reality. So let me know if you found this article useful, and maybe I’ll revisit the theme soon.
Until then, have a great weekend.
