As I continue to work through what is hype and what is useful when it comes to using AI in VFX, I decided that the ultimate test would be to actually use it in a live project.
As I continue to work through what is hype and what is useful when it comes to using AI in VFX, I decided that the ultimate test would be to actually use it in a live project.
I really don’t think it’s ready for use in professional work, at least not with big studio clients, or in any obvious way. Even if they don’t stipulate no AI in the contract, they come with quality control departments that meticulously check the final shots, and I don’t think an AI shot would get past those QC guys - they’re too low resolution and warpy.
However, myself and the Shoot First team are currently putting the finishing touches on our latest short film, and I spotted the perfect chance to try it out. We had a montage sequence of the protagonist drifting into a haze of drink and medication, but we didn’t have enough coverage - enough filmed footage - to get a really good feeling of progression.
I found AI footage to be really useful in two ways here:
- Generating generic stock footage type shots (pills falling through space, for example). My first port of call was to go to actual stock footage, which is going to be much higher quality, but AI footage gave me the flexibility to get the shots that I wanted. The usual AI problems of the footage being weird and warpy were mostly sidestepped because I’m using it overlaid over professionally shot footage.
- Using AI to push existing VFX further. Some AI video generators (Luma’s Dream Machine, for example) allow you to input an image as a prompt. I’ve been using images that I’ve already done VFX work on, which gives me some control over the final output.
In this example, it was always my intention to have the character flattened by a creature on his chest, and that’s all done with traditional VFX. Getting him to merge with the creature and crawl across the floor screaming? That’s AI.
My final verdict is that I’m really grateful that AI was able to help me out on this project. I had an edit that was almost identical in terms of timing, but I just needed more progression on the shots - I think that the final impact would’ve been much less without it.
As always with AI, there are caveats though:
- Unsettling imagery: AI imagery has a tendency to gravitate towards unsettling visuals. That worked great for me since I was going for horror. It might be more limited in other genres. When Toys R Us decided to create a whimsical magical advert using Sora, most people just found it disturbing (by the way, that advert is pitched as AI-generated, but there’s a lot of VFX work in it).
- Quality discrepancies: There’s a large jump in the amount and crispness of the details between the filmed footage and the AI-generated footage. I tried to close that gap by using AI upscaling to sharpen the footage and then degraded the quality using lots of bloom, motion blur, and lens flare (I’ll cover this next week). Again, the nature of the sequence I was creating meant that I got away with it - Toys R Us wouldn’t have been able to do it!
- Limited directorial control: This is only half-directed, at best. I wanted the man and creature to merge, and this result is the most powerful performance that it gave me, but I can’t really say it’s me directing it. I just chose it over the other options (like when the man and creature merged into a weird mediaeval knight and wobbled off the screen together). I’m grateful for what I got, but there’s every chance that it would’ve given me nothing, and I’m unwilling to take credit for the result.
I’ve put the whole montage on the Discord, if you’d like to take a look and discuss it (it needs the music, really!) , head over there!
And of course, stay tuned for the full film and all the behind the scenes/tutorials that we’ve been making from doing it.
Until next time, happy creating!