E_NO_SUCH_BLOG

Programming, philosophy, pedaling.


Random Mosaics with ffmpeg and Ruby

Mar 5, 2017

Tags: programming, art

This post is a bit of a spiritual successor to one I did about glitch art with RMagick back in 2015.

One of my favorite aesthetics is that of video mosaics. There’s something about firehoses of information and pattern emergence that really appeals to me, and video mosaics embody both of those properties.

I had already written a script to produce NxM mosaics using ffmpeg, but it had two obvious shortcomings:

To illustrate these shortcomings, here’s a 10x10 mosaic rendered with filter_complex:

As you can see, the shorter sources simply freeze when they end, drawing your eye to the remaining ones. This looks really bad, and defeats the entire point of the mosaic.

The Solution

To solve this, I came up with a pipeline:

  1. Fetch GIFs from Giphy’s API.
  2. Normalize the GIFs into the one resolution and color palette.
  3. Take the normalized GIFs and put them into random groups such that each group is approximately N seconds long.
  4. Merge each group into a single GIF.
  5. Mosaic the groups together to produce the appearance of changing sources.

The first step was pretty simple, since Giphy provides a free beta key for their API. The end result was giphy-fetch, which is invoked like this:

$ giphy-fetch --verbose --directory cat_gifs "cats"

yielding 100 GIFs under the cat_gifs directory:

Querying Giphy for 'cats'...
Saving cat_gifs/GvD9nPHWj86VG.gif...
Saving cat_gifs/5r5J4JD9miis.gif...
Saving cat_gifs/l0HlGRDhPTqVEvhCw.gif...
Saving cat_gifs/UotLuplZSzKRa.gif...
[repeat 100 times]

Of course, since each GIF has a different resolution and color palette, they need to be normalized prior to combination. ImageMagick’s convert kinda worked for this but was incredibly slow, so I went with gifsicle instead.

This was accomplished with another quick script (in bash this time), normalize-gifs:

$ DIMS=500x500 normalize-gifs cat_gifs/*.gif

yielding a directory with the same resolution (500x500) containing the normalized GIFs:

normalized GIFs

Grouping the GIFs into random collections with the same total duration was slightly trickier, but made possible by exiftool’s -Duration flag:

$ exiftool -quiet -Duration normalized_gifs/*.gif | awk '{ print $3 }' | sed '/^$/d' | sed '$d'

yielding:

3.78
5.07
3.60
4.05
0.40
4.20
7.07
1.68
[...]

These then need to be mapped to their original sources, and used to build groups whose durations are each N seconds (plus or minus 1 second, since we don’t have perfectly sized GIFs to pad everything out). Once grouped, gifsicle can be used again to merge each individual group into one big GIF:

$ gifsicle -w --colors 256 ${group} > groups/group${n}.gif

This was all Ruby-ified and wrapped in a script, gif-length-groups, which gets invoked like this:

# build 100 groups, each 60 seconds long, with no GIF longer than 5s
$ gif-length-groups -n 100 -l 60 -i 5 500x500/*.gif

yielding the groups directory:

GIF groups

each of which looks something like this:

Now we’re finally ready to use ffmpeg-mosaic:

# if you rendered less than NxM groups, you can add -r to shuf to repeat them
$ ffmpeg-mosaic 1920x1080 10x10 mosaic.mp4 $(shuf -e -n 100 300x300/*.gif)

yielding the final product:

Of course, we can do less dense layouts as well:

These all flow smoothly from one GIF to another, bypassing the limitations of filter_complex.

Next Steps

My original plan was to use these mosaics as a screensaver via XScreenSaver + mpv, although I quickly realized that XScreenSaver doesn’t have an easy way to orient one of the outputs to fit my vertical monitor. Still, it looks pretty awesome on my horizontal monitors:

mosaic screensaver

A friend suggesting playing the mosaic on an old CRT television, which I think would also look pretty awesome (especially with some static or interference introduced). I’ll post an update if I can get my hands on the right equipment for that.

Summary

You can find the scripts used in this post on my site, on the snippets page. You can also check out my vidme channel, where I’ll post more mosaics as I continue to experiment. Feel free to download the generated mosaics from there - they provide direct links!

As it turns out, generating lots of unique large GIFs takes up a decent amount of space, enough to make it (currently impractical for me to distribute the normalized GIFs and GIF groups that I rendered.

If you’d really like to use the data that I used (instead of just pulling from Giphy or another source), send me an email and we can figure out the best way to do a point-to-point transfer. Otherwise, check out “psychedelic”, “hacker”, “cyberpunk”, and “vaporwave” via the Giphy API - most of the GIFs I used came from those queries.

Happy hacking!

- William