After Effects - Rotoscoping
- L K
- Apr 22, 2022
- 4 min read
Adobe released an improved rotobrush tool (roto 2.0) with their 2020 edition of After Effects, it works much the same as the old rotobrush tool except it utilises the GPU for faster processing and is enhanced by AI to produce a much cleaner result.
The basic workflow of rotoscoping in After Effects is to use the rotobrush tool on a subject in a layer, refine and propagate across all the target frames, freeze the propagation, and then final adjustment of the matte.
I've marked in red the rotobrush tool which is used to add the rotobrush effect and to refine it across each frame. The yellow box is used to freeze the propagation and lock in the boundaries of the matte. And the blue box is the effects panel where adjustments to the matte can be made.
When painting the first rotobrush stroke, choose a frame with the subject fully in view and paint a line roughly through the centre as above.
AfterEffects will try to find the edges of the subject once you paint a line. You can see in the example above the magenta edges it's found, but also that it's missed out a lot of the body and added in some unwanted background. Simply paint in the missing parts with further strokes where needed, and remove unwanted areas with strokes while holding the Alt key.
It's recommended to use as few strokes as possible.
The rotobrush size can also be adjusted in the "Brushes" panel if needed.
Hitting the 'spacebar' will start the propagation throughout the rest of the timeline. I highly recommend using the "high" quality setting instead of "standard" in the effects panel first though. I've found much more consistent results using this.
You can wait for the entire sequence to propagate, but you'll likely see areas where the tracking slips and little adjustments are needed.
'Spacebar' will pause on the current frame, 'PgUp/PgDn' move the timeline forward/backwards by a single frame. Paint in/out your refinements and After Effects will try to propagate this new information onto the following frames.
The image to the right shows how the tracking has slipped on the foot and needs another brush stroke.
Refining the Matte
Adjusting the options in the 'Roto Brush Matte' menu are the main sliders for controlling how the edge looks.
The greyed out 'Refine Edge Matte' is an additional rotobrush tool that works well for areas without a hard edge like frizzy hair - Further explanation on this tool will be noted later in this post.
'Feather' is how soft the boundary edge is. This applies with the edge as the centre line with the feathering applied to both sides.
'Contrast' is how sharp the edge is. A higher contrast is a tighter line, lower is a softer feathered line.
'Shift Edge' offsets the edge line, can use both a positive and negative value.
'Reduce Chatter' tries to decrease any noise on the edge between frames. This is a dangerous setting to use and can often lead to obvious ghosting if the target subject is moving.
Enabling 'Motion Blur' adds a feathered edge to any parts that After Effects finds moving quickly across frames. This is very useful for moving subjects and the amount of blur can be further dialled in using the additional options in the menu.
Decontaminate Edge Colours attempts to background colour from any feathered edges.
This option is mostly used to try to fix halos from the subject or if colours are bleeding from a blurred edge.
Using it on frames with little to no dithering on edges means that there is no background colour that it can remove as it assumes that the subject is solid in that location.
Again there are some adjustment sliders if it's not performing as needed.
'Refine Edge Matte'
None of the greenscreen footage I shot for this project would benefit from this so I've used Nicolas Cage's infamous hair from Con Air as an example.
The left example is a quick rotoscope just using the normal rotobrush tool. It's immediately apparent the the hard edges that work well for clothes and other edges are just not suitable for hair or semi-transparency. Trying to add a feathered edge with low contrast would not suit all the different hair conditions, and would not work on the sharp clothing edges.
The example on the right is the same rotobrush as above, but with the 'Refine Edge Tool' applied to the semi transparent areas of hair.
This was just done as a demonstration and clearly still has a lot of issues that could be cleared up with more time spent adjusting, but the different is huge.
The only different adjustment with this tool is 'Smooth' which reduces the sharpness of the edge curves
Spill
As a result of filming on a greenscreen, the subject in my recorded footage has a green tint/bloom due to global illumination. This is known as spill. There are a few effects within After Effects to deal with it. One is the 'Decontaminate Edge Colours' in the rotobrush mentioned above. This is a more generalised filter than works by looking at the edge of the matte itself.
For footage contaminated by a singular colour - like a greenscreen - is to target that colour specifically. There are spill suppressor effects where a colour can be designated for removal.
Comments