8/18/2020

FFmpeg Retro Camera Filter Experiment

This post is related to a question I recently asked on reddit

As an experiment, I've been trying to recreate some of the effects from this AfterEffects tutorial only using FFmpeg and ImageMagick. In particular, I wanted to see how you might recreate the film matte that applies the rounded square around the video, and then recreate the moving/shaking video effect.

Here's what I've got so far:

Let's take a quick look at what I've done so far to create the film matte, overlay it onto my video, and then to try and recreate the video shaking/jittering effect.

1. Creating a Rounded Square with ImageMagick:

My first step was to create a rounded square that can be used as the center of the "film matte" that I want to create. Creating a shape like this is actually pretty easy to do using ImageMagick. Here's an example creating a green, 1000x1000 square png:

convert -size 1000x1000 xc:transparent -fill green -stroke green -draw "roundrectangle 0,0 1000,1000, 100,100" square.png

Here's the resulting image this command will create:

You can read more about ImageMagick's draw commands here.

2. Creating the Film Matte

Next, to creat my film matte, I used FFmpeg to generate a black video and then added the green square on top as an overlay. Here's my FFmpeg command for this:

ffmpeg -f lavfi -i color=s=1920x1080 -i square.png -filter_complex "[1][2]overlay=(main_w-overlay_w)/2:(main_h-overlay_h)/2" -t 5 overlay.mp4

And here are the results we get from this step:

3. Overlaying the film matte onto my video with chromakey

Now that we have our film matte, we can try overlaying it onto an actual video. FFmpeg has a chromakey filter that we can use to "cut out" the green portion of the overlay (the inner square), revealing our video underneath. Here's an example using FFmpeg's chromakey filter to do this:

ffmpeg -i input.mp4 -stream_loop -1 -i overlay.mp4 -filter_complex " [1]chromakey=0x008000:blend=0:similarity=0.15[2]; [2][0]scale2ref[2][1];[1][2]overlay " -t 5 test.mp4

In this example, I'm setting chromakey=0x008000 because we want to target the green in our overlay, which has a hex value of #008000. I'm also setting the similarity parameter to 0.15 - this filter is used to determine to what degree we want to target colors similar to our target color. 0.01 will match only the exact key color, while 1.0 matches everything. Read more about this filter here.

Here's the result we get from this command:

4. Shaking/Jittering Video Effect

Next, I attempted to recreate the shaking video effect by moving the video up and down randomly +/-100 pixels with a break every half second:

ffmpeg -f lavfi -i color=s=1920x1080 -i test.mp4 -filter_complex " [1][2]scale2ref,overlay=y='if(lt(mod(t,1), 0.5), random(1)*100+random(1)*(-100))' " -t 5 test2.mp4

In this example I'm using mod with the video's time (t) to move the video up and down randomly in half second intervals. Here's a closer look at the code for this:

overlay=y='if(lt(mod(t,1), 0.5), random(1)*100+random(1)*(-100))'

Basically this is saying if mod(current video time, 1) is less than 0.5, set the y position of the video randomly between 100 and -100.

And here are the results from this step:

What I'm still trying to figure out

- How would you handle the shaking jittering video movement? Currently, my approach has been to use mod with the current time as a way to adjust the y-axis position in half-second intervals, which has gotten some interesting results but doesn't look as good as the results in the original After Effects video.

- How would you get the wrap-around effect (see the original After Effects tutorial), where the top of the video comes back in at the bottom when it goes off-screen and vice versa?