Gaussian
Splatting
Gaussian splatting is one of those wild, overly hyped, misunderstood corners of tech that influencers tell us will change film FOREVER, and then doesn't. That doesn't mean it won't, and it doesn't mean it isn't really powerful.
It's essentially a way of 3D scanning, but it's the best technology we currently have for digitally recreating a person, a prop or an environment as though you were actually there.
The other forms of scanning, photogrammetry and LiDAR, both of which are more common in film right now, give you polygonal meshes. Polygons are great for a lot of things, nearly all the rendered VFX you've ever seen uses them as a basis. But they have limits. They're pretty terrible at fine detail that breaks the silhouette, like hair and fur, and they need re-rendering to capture lighting and materials accurately.
Gaussian splats aren't polygons at all, but little blobs of colour information. They store the colour of the object at that point, and also how it reacts to the light it's under at different angles: reflective, transparent, matte.
That makes them perfect for taking a 3D digital snapshot of a complex set or character, exactly as it was on the day of the shoot.
The Idea
Drag to orbit. Slide from the sparse points a camera solve gives you, to the trained splats that fill in the surface.
One Splat
A single 3D Gaussian is a soft ellipsoid. There's not that much to it, on the surface. It has colour, opacity, scale and rotation. But part of its magic is that it stores view dependent colour, and that's the secret sauce that gives it the power to recreate a full film set. The more splats, the more complexity you can describe.
What gets stored
Per splat, as plain numbers. Nothing neural.
Scale it up
Actual Real Light, Baby!
Here's the thing with splatting that makes it great: colour is view dependent. You have local colour, what colour the object actually is, and that's different to the colour that reaches your eyes. That colour depends on the reflection, and the reflection changes based on the angle you're viewing the object from.
Think about shining the reflection from your watch into someone's eyes. Aim it at that person and they see blinding light. Everyone else just sees a watch.
If you had a splat of that watch and moved the digital camera to where the blinded person was standing, you'd see blinding light too. No other kind of scanning captures that.
Since a splat stores the colour from each direction around a sphere, this effect in Gaussian splatting is called spherical harmonics.
The Pipeline
Photos in, splats out. The bit in the middle is where the magic happens.
Capture
200 to 1000 stills or frames covering every angle. As crisp and detailed as you can get them. It's impossible to get good results without good photo input.
Solve
Map out a 3D structure from the photos. There are tricks to refine this stage, but it's mostly down to how good your input was.
Seed
The sparse point cloud becomes the first, crude splats.
Train
Render, compare, nudge, split, delete. Repeat 30k times. This training stage is what makes splats seem so magical. Other scanning techniques are just raw data, with no training at all.
Play
Sort by depth, blend front to back, show the viewer the splats they'd expect from that angle. Speedy, with none of the slow rendering that polygon scans need.
Mesh vs Splat
TLDR: meshes aren't as good for detail, splats are terrible if you need to relight them. Meshes give you much bigger files, but they're easier to manipulate later, since splats usually have to be converted to meshes before you can edit them.
| Strength | Mesh | Splat |
|---|
Shooting The Scan
Splats are made from photos, or from stills pulled out of video, though be careful with that. You'll need a fast, global shutter to get images anywhere near photo quality. Anything that can't be seen in the photos won't be recreated accurately in the splat.
So hit every angle you can, exactly like you would for photogrammetry. For a person or a prop that means going round them three times, at three different heights. The problem with people is that they move, which wrecks the scan. Either go full Darth Vader and trap them in carbonite, take your photos fast, maybe with more than one person shooting at once, or hire a studio with a rig carrying 50+ DSLRs that fires from every angle at once.
What ruins a scan
Floaters
Blobs left hanging in empty air where two angles disagreed about what was there. Just as offensive as a floater left in a toilet, and to be removed with as much prejudice.
Anything that moves
People, traffic, water and leaves land in a different place in every frame, so they train to mush.
Angles you never shot
Push the camera outside the volume you walked and the image smears. Shoot wider and more angles than you think you need. If you're not crawling about like a soldier behind enemy lines, you're not splatting right.