Fine-tuning the matte
Roto Brush does a pretty good job, but there may be stray bits of background in the matte, or foreground areas that weren’t included. You’ll clean those up by refining the edge.
Adjusting the Roto Brush & Refine Edge effect
When you use the Roto Brush tool, After Effects applies the Roto Brush & Refine Edge effect to the layer. You can modify the effect using settings in the Effect Controls panel. You’ll use those settings to further refine the edge of the matte.
Press the spacebar to play the clip in the Layer panel; press the spacebar again to end the preview when you’ve seen the whole clip.
As you preview the clip, you may notice that the segmentation boundary area jumps around quite a bit. You’ll use the Reduce Chatter setting to make it smoother.
In the Effect Controls panel, increase the Feather amount to 10, and increase Reduce Chatter to 20%.
The Reduce Chatter value determines how much influence the current frame has when performing a weighted average across adjacent frames. Depending on how tight your matte was, you may need to increase Reduce Chatter to 50%.
Preview the clip again. Notice how much smoother the edge of the matte has become.
Using the Refine Edge tool
The boy’s coat and face have hard edges, but his cap is fuzzy, and the Roto Brush tool didn’t pick up the nuanced edge. The Refine Edge tool lets you include fine details such as wisps of hair in designated areas of the segmentation boundary.
Though it might be tempting to use the Refine Edge tool immediately after creating the base frame, it’s best to wait until you’ve refined the segmentation boundary across the entire clip. Because of the way After Effects propagates the segmentation boundary, using the Refine Edge tool too early results in a matte that is difficult to use.
Go to the base frame, which is the first frame of the clip, and then zoom in so that you can see the edges of the cap clearly. Enlarge the Layer panel if necessary, and use the Hand tool to move the layer so you can see the entire cap.
Select the Refine Edge tool (), hidden beneath the Roto Brush tool in the Tools panel.
The cap is relatively soft, so a small brush size will work well. For a fuzzier object, you might have better results with a much larger brush. The brush needs to overlap the stray edges that emerge from the object.
Change the brush size to 10 pixels.
When you use the Refine Edge tool, draw strokes along the edges of the matte.
In the Layer panel, move the Refine Edge tool over the edge of the cap, straddling the segmentation boundary and including the fuzzy variations. You can use multiple strokes to move the tool around the entire cap.
When you release the mouse, After Effects switches to the Refine Edge X-ray view so that you can see how the Refine Edge tool changes the matte, capturing the detail in the edges.
Move the current-time indicator across the time ruler to view the edge. Where more of the hat is visible onscreen, extend the Refine Edge boundary to include the top areas of the hat. Just as with the Roto Brush tool, press Alt or Option to use the tool to remove areas from the matte if necessary.
Zoom out to see the entire scene, resize the Layer panel if you enlarged it, and then choose File > Save to save your work.