The assumption some people have is that I got a whole bunch of people together to perform in sync, but that is absolutely not the case. Instead, every single critter, and many of the visual props you see, are on their own separate chromakeyed layer, composited using DaVinci Resolve, each one acted by me personally.

I had learned a lot of lessons from the video I produced for Furality Somna; for that one I just recorded a simple looping scene with a fixed camera, and played every single character in it, and again composited everything in DaVinci. I ended up making it all up as I went along, which was fun but also added about 3x as much work as it normally would have taken, and this year’s video being much more ambitious I knew I needed to do way more planning for it to come out well.

(The music in last year’s video, incidentally, is a lo-fi version of Material Change.)

Here is a video showing some of the stuff that I’ll be describing in further detail:

Music production

I wrote and recorded the song up-front, and used that to plan the visuals as I figured out which musical genres to use. Some of it I had planned out ahead of time but some of it I figured out what made the most sense while I was recording the music. As usual I produced the music in Logic Pro, and I made heavy use of the folder stack functionality to keep my project organized. Each genre was recorded on its own separate set of tracks contained within a folder stack.

I did end up making some changes to the music during production. The jazz segment in particular changed a few times, both to improve the lyrics and the vocal delivery.

I went with a tempo of 128BPM, because the video is one minute long and there were exactly 128 beats in it. Sometimes you don’t have to overthink it.

Locations

After getting the music arranged, I wrote up a simple shooting script to plan out the scenes, and did a lot of location scouting to find places that would work well. My main criteria were:

• It needs to have decent performance (so I could record at a steady framerate)
• It needs to be possible to turn off post-processing (so that it doesn’t interfere with VRChat’s stream camera green screen)
• It needs to be somewhere that’s recognizable to folks who are in my orbit

The locations I ended up choosing were:

• Community Open Mic (pop)
• Piano Lounge Freude (jazz)
• Trans Academy (choral)

The first two are venues I’ve performed at many times, and Trans Academy is an organization I volunteer for so it’s special to me.

Originally I was going to record the first segment in Transitions Club, where I’ve also performed many times, but unfortunately there’s no way to turn off the camera postprocessing there, which I only realized after doing a test recording and discovering that it affects the green screen.

Avatars

All of the avatars I used were my own; in particular, I made several versions of my standard critter base with different instrument setups, and also a specific drummer version of the Critter Junction resident base.

The guitar, electric bass, keyboard, and drum sticks were my own models. I also modified the following meshes (all of which are free to use in this way):

• Instruments: Violin / Piano / Double Bass by Tom Zimmermann (for the violin, viola, and double bass)
• Cello by VM-Models
• Drum Set by zaurian868
• Violin bow by CharlotteMeehan

All of these were extensively modified for my purposes (particularly to make them VRChat-compatible).

I also created a simple green screen avatar for part of the compositing workflow.

Recording layers

VRChat has a camera dolly system which allows you to set up and replay a camera path, but it doesn’t provide any real built-in synchronization mechanism. To keep things in sync, I did the following:

1. Created a basic lead-in click track (I used Audacity for this, but any music software will do)
2. Split the music into sections for filming
3. Downloaded sendosc to trigger the dolly camera
4. Wrote a basic Powershell script to synchronize everything up

$click = New-Object System.Media.SoundPlayer 'click.wav'
$player = New-Object System.Media.SoundPlayer $args[0]

.\sendosc.exe localhost 9000 /dolly/Play b false

echo "Playing click"
$click.Play()

Start-Sleep -Milliseconds 5000

echo "starting dollycam"
.\sendosc.exe localhost 9000 /dolly/Play b true

Start-Sleep -Milliseconds 1000

echo "Playing song"
$player.Play()

Start-Sleep -Milliseconds 16000

In order to film a pass, I would then invoke the script with a command like:

powershell -ExecutionPolicy Bypass -File play.ps1 ".\1 pop.wav"

I kept a single powershell instance open in the foreground so I could just press up and enter every time I wanted to film a take, which was quite often.

Anyway. This script will play the click track for five seconds, then starts the camera dolly path, then one second later starts the music playing, so that I could then perform my choreography in sync with the audio track.

I did several takes of each layer, and also kept an eye out for obvious rendering hitches during it, since those often cause the camera path to fall out of sync and cause other visual artifacts in the composite. Unfortunately, VRChat’s dolly system is a bit limited when it comes to synchronization.

As far as synchronizing things in the timeline go, I found that there were 21 frames of latency between the display capture and the audio output capture from my headphones, and I used the click track to line the clips up in the composite.

Dolly management

I made heavy use of DollyManager to manage my camera paths, which was super useful whenever I had to do reshoots (which was often). Unfortunately, Camera Dolly does not store a bunch of settings when you export so I still had to make notes of a lot of things. And also it does store a bunch of settings that you don’t actually want it to store, such as the green screen color. And the in-game editor is kind of a nightmare to actually use, especially for keyframes that are out of reach.

Basically, the camera dolly as it stands right now is kinda half-baked. Hopefully VRChat improves it over time.

Dealing with in-world props

Most in-world props do not appear on the green screen, which is why I created the green screen avatar to mask those out. For those render passes I disabled the camera green screen, and stood behind the prop while recording another pass. This usually required a lot of extra work to make a clean plate in DaVinci, but it was worth it.

The rock/pop scene

This scene I ended up redoing multiple times while I figured out the workflow. Fortunately it was the simplest of the composited scenes, so it was well worth the time spent experimenting so I could make the later scenes go more smoothly.

There are “only” seven layers in this composite:

• Background
• Drummer
• Drum kit (world prop)
• Backup vocalist
• Bass player
• Lead critter
• Microphone (world prop)

I recorded this in three separate shots each with their own camera path; I ended up only using two of the shots.

During the first shot with the sweeping camera pan I had to do multiple cuts within the layers to change their ordering, which you can kind of see in the timeline:

I added a bit of blur to the background layer, mostly to hide some annoying aliasing artifacts from the render since I had antialiasing disabled in VRChat to get the maximum possible framerate. Lens blur hides many sins.

The studio scene

For the scene where I’m in the studio I just found a world where there was a chair I could sit on, disabled fullbody tracking (so that the chair would work right), and lined myself up over a photo of my actual messy studio. Pretty straightforward. In the composite I added some extra thick cartoon outline and some film grain to make things match a bit better. I also spent way too much time figuring out how to post-process a shadow onto the seat cushion, which mostly got covered up by the speech bubble. Oh well, it was a good learning experience.

Incidentally, I am not singing along with the lyrics there, and instead am saying “Why isn’t this working?!” which is a common utterance from me whenever Logic is misbehaving.

The pixel art scene

This was another super-complicated composite. DaVinci doesn’t really have any good way of doing pixel-precise positioning or scaling, so what I ended up doing here was to draw every layer at 320x180 with things lined up how I wanted them in the frame, then scaled those layers up 6x with nearest-neighbor filtering (using ImageMagick) and then did the composite in DaVinci.

The note lanes for the rhythm game in the background were just drawn as sprite layers which I moved in sync to the music, and I did a Fusion composition to extract the few rows of pixels where those notes hit the bottom edge of the controller sprites and then applied a ridiculous glow with a directional blur to them.

The bouncing coin is something I quickly modeled and rendered in Blender as a 6-frame sequence. I looped that sequence in DaVinci and then gave it a simple motion path, synchronized with the beat.

Each of the performers was multiple layers, typically a base sprite, a sprite for the instrument, and any additional sprites for hand positions (which I moved in 6-pixel increments using “step” interpolation) and so on. The falling sprites were done separately, of course.

When the entire stage falls apart, I just applied some simple motion paths to them and went with what felt right, rather than anything physically accurate (but it was fun to make the trellis bounce off the floor, a decision I came up with on the spot).

The main composite is around 20 layers, and many of those layers are also compound clips with 3-4 layers of their own. There’s a lot going on. Watch it a bunch of times!

The cartoon critter head

That was all just traditional hand animation! I made a layer for the head, the blinking eyes, and a bunch of mouth shapes, and then I did the usual thing one does. Resolve, incidentally, is not particularly great for animating this way, but it worked well enough. For a longer animation I’d definitely want to use something else though.

The best thing that came out of this was when I was drawing the base layer and doing a symmetry check:

The big grid of games and videos and such

The Undertale-style critter sprite was a modified version of the one I drew for the Undertale memories microsite that went up for the 10th anniversary of Undertale’s release.

The footage all came from things I’ve worked on. Most of it was games I composed for:

• Ass Without a Trace 2 by HoggTied (NSFW)
• Psycho Burst and Airship Chef by Yamatsu (both NSFW)
• Snack Stacker by Mitha
• Pawmune by SundewBry
• Treestoration by rtwfroody
• Currying Flavors by chinniedraws
• Magnolia’s Early Morning Walk by Definitely a Fox (NSFW-ish)
• The Balance of Sappho by The Firecrackers
• Hallowed Grounds by Paxton
• The Herbal Spice Problem by Glass Dragon Studios
• Admiralo Island Witches Club by cloverfirefly
• Brickris Tetout by Bamboy
• Elevate by Kamil Bazydlo
• Haggle Waggle: Duck & Doge by z0lly
• Spooky Sushi by Spectronaut
• Refactor, Colorful Critter¹, and CATcher by fluffy (that’s me!)

But I also included two of the sillier videos from my random crap channel:

• jukka wins it all
• critter dance (a test of my customized Winterpaw rodent avatar)

To actually animate the grid I just made two compound clips, each with its own 5x3 grid of cropped videos, and then scrolled each one across the screen. For most of the videos (which were 16:9) I could size and position them consistently by zooming them to 0.3333 and giving them a left/right crop of 384 (to make their effective resolution 1152×1080, the horizontal resolution being 1920×3/5, trust me this all makes sense mathematically). Horizontal positioning was in multiples of 384 (1920/5), and vertical was in 360 (1080/3). There were a couple of non-16:9 videos which I also had to deal with separately, but it was basically the same process, just with different numbers.

The jazz club scene

There wasn’t anything particularly different from this vs. the rock/pop scene, but in this case I recorded the background and microphone plates with postprocessing enabled, since Piano Lounge Freude’s lighting is all done in the postprocessing step. For the microphone I was able to clean up the postprocessing glow well enough, mostly using the “despill” function built-in to 3D Keyer.

Oddly enough, the stage mic in Piano Lounge Freude does appear on the VRC stream camera’s green screen, somehow. I assume there’s some specific render tag set on the object. As a result it appeared in my avatar video layer. However, it was still good to have the properly-lit microphone to composite on top to improve the look. To account for the postprocessing-based lighting I used DaVinci’s color grading to make the avatars look appropriately-lit in the composite.

On the background I was originally thinking of using a green screen and then tracking it to put some of my other random videos in the space, but I decided that would have been too much work and also too visually-busy, so instead I made a quick-and-dirty plaid background video to run in a loop and then composited in the one shot from last year’s video more traditionally.

The choral scene

The choral scene was a nightmare to record and composite; in my Resolve project this clip eventually got the name “the shot from hell.” The final composite had around 35 layers! These are due to the following:

• There are 16 critters (11 in the choir, 4 in the string quartet, and the conductor)
• Because of the sweeping camera motion I needed to change the composite order of many of the critters in places and it was easier to do that in additional layers
• Because the critters are partially obscured by the fountain I needed to make a greenscreened traveling matte of it
• And of course the website that appears in the background is another layer

If you step through that scene frame-by-frame you will possibly notice some things sliding around a little weirdly, because VRChat’s framerate is also not in sync with OBS’s. I had OBS recording at 60 FPS, and VRChat generally renders at 75 FPS on my system, so it was the luck of the draw as to whether the frame timing would perfectly line up. Fortunately it’s Good Enough. In future projects I will definitely try running my headset at 90 FPS to see if that helps any.

There were also some movements that fell out of sync due to render hitches, and for those I mostly fixed them in post by adding additional motion pathing to the layers. I didn’t get them all. There’s also a visual quirk in how the cellist’s tail hangs through the floor but remains visible. Ideally I’d have added a floor collider to the avatar to prevent this, but perfect is the enemy of done, and you probably didn’t notice those issues until you read this text.

Some chromakey tips

Most of the chromakeying was just using DaVinci’s basic “3D Keyer” chromakey. In some spots I had to mask some things in separately from how I masked them out, because of color clashes between avatar parts and the key color. (This was especially problematic for my main critter’s colorcycling-rainbow material, as well as my collar which is green in most forms.)

In a couple of spots I did find it useful to use “magic mask” as a pre-filter for 3D Keyer, but most of the time it was much easier to just do combinations of 3D Keyer and use its built-in garbage matte.

Also, while you can change the green screen color in the VRChat camera, you have to remember to set it up before you set up a camera path, or edit the saved path JSON. Fortunately, VRChat’s green screen is a single solid color with precise values so you can generally get away with having a green object on a green background as long as you set the key mode to “tight,” although some of the mask refinements will still require extra work.

That halftone wipe transition

DaVinci doesn’t have a built-in halftone wipe or any sort of built-in halftone functionality at all, so I had to roll it myself. I approached it from a graphics programming perspective. You can think of a halftone as being similar to an ordered pattern dither, where the threshold pattern is given by:

\[ u = k(x\cos\theta - y\sin\theta) \\ v = k(x\sin\theta + y\cos\theta) \\ t = (\sin u + \sin v + 2)/4 \]

So, what I did for this was to generate a halftone screen function (I used a procedural texture in Affinity Photo for this but I could have probably figured out a way to do it all in DaVinci) and then merged it with a gradient with an additive blend, then applied a threshold filter to that, then used the resulting image as the blend mask to overlay the incoming clip over the outgoing clip. Animating the gradient’s start and end points to sweep across the frame generated the wipe transition. Easy.

(Someday I will release this as a template for others to use, because it’s a really cool effect!)

Some basic statistics

• Amount of raw footage recorded: 5 hours, 36 minutes and 25 seconds
• Amount of time spent working on the music: around 14 hours, apparently (it sure didn’t feel like that long though)
• Amount of time spent editing video: I have no idea because it turns out Time Sink wasn’t running on my editing computer this whole time, oops, but I’d estimate probably 40-60 hours?

So this video took around 70-80 hours of work altogether. If someone wanted to hire me to make something like this it would probably cost around $5000 at a highly-discounted rate, or $8000 if I actually valued my time, which I clearly do not. Hopefully I get at least $5000 in music sales out of this. (Mirlo and Bandcamp are probably the best choices for that.)

1. Technically Colorful Critter doesn’t have music, but it does have (minimal) sound design, and it’s one of my favorite little game-toy things so I don’t regret including it here. ↩

Context

Once a year, Song Fight! people get together to do a show, Song Fight! Live, where we basically get together and do Song Fight! as an in-person live event. The original format from the first three years was that we’d have basically an in-person reprise of prior title challenges (with different musicians taking turns to play their songs for the titles), followed by what was essentially an open mic for songs people wanted to perform. In keeping with the challenge of Song Fight!, the event’s host would also provide a title for people to make a new song for, to be debuted at the show.

Starting in the fourth year, the title-based format stopped being feasible (due to the growing number of titles and the rarity of any two participants having had any of them in common), and it switched to a concert-festival format, with each participant doing a short performance set, but the final “live fight” challenge (with the debut of a new song) remained.

In 2020, due to the COVID-19 pandemic, the format had to change; rather than meeting in person, we prerecorded videos that we would submit for presentation, and there would be a separate video presentation of the “live” fight songs, with a vote conducted online. The title in 2020 was “Better Luck Next Year,” which was… optimistic about how soon the pandemic would subside enough for us to reconvene in person.

2021 was still online-only, and the title selected for the live fight was “Behind a Mask.”

When this challenge was ongoing, I had recently moved from Seattle to a nearby small town in order to have more space for gardening and working on music, although I had not yet set up my new recording studio.

Inspiration

With the context of the world and everyone’s isolation, I was struck by just how much everyone had withdrawn and started to rely on social media for all interpersonal contact, and with how much this reliance on social media was distorting peoples' attitudes to one another.

At the same time, in this isolation, many people were forced to spend a lot of time alone, which leads to introspection and self-discovery. At the same time, Snapchat’s “gender swap” filter was leading a lot of people to experience gender euphoria for the first time, making them realize just how much dysphoria they were normally under without realizing it.

Meanwhile, so-called “deepfake” technology was getting extremely powerful and scary, and this was leading to a huge uptick in the number of scams being perpetrated on unsuspecting/unaware people, who didn’t realize that the person they thought they were dealing with did not exist.

The various algorithms that were impacting everyone’s lives were changing the way we thought about ourselves and others.

Lyrical meaning

Feeling lost in an elephant’s dream

This is in reference to how elephants purportedly never forget, and how the Internet is essentially a non-forgetting collection of facts that can be recalled at will, perhaps inaccurately. The specific phrase comes from the name of an open-source short film.

Caught up in the digital stream

This is, of course, in reference to how awash everyone is in a never-ending stream of social media posts, especially when trapped inside with nothing better to do than to sit at the computer.

Cave of shadows, can’t say what I mean

This is a reference to Plato’s Allegory of the Cave, in combination with the acts of self-censorship people have to undergo in order to remain “polite” or to avoid hurting others' feelings, especially with regards to invalidating other peoples' truths.

A million miles far from home

Being stuck inside made one’s home feel like an unfamiliar, oppressive place. Many of us just wanted to get back to simpler times.

Best foot forward, take a step back

A reference to the Overton window and its related allegory: “Meet me halfway,” says the reasonable person; you step forward, they take a step back, and then once again say, “Meet me halfway.”

Play defense while on the attack

Bad actors have a tendency to play the victim while being the ones who are committing an offense. Figuring out who is really wronged, and what steps we must take to address that injustice, can be quite challenging.

Sowing discord, can’t cut them no slack

This line has a double meaning; the primary one is that people who spread misinformation and disinformation online have a tendency to then decry anyone who tries to correct their lies and mistruths, and how difficult it is to hold one’s ground in light of that.

The other meaning is that Discord and Slack very quickly took over as the communication platforms of choice, especially as alternatives to in-person communication and collaboration since offices had finally become recognized as unsafe breeding grounds for disease.

Everybody so alone

Because, you know. Isolation.

If everyone could see
themselves through some other eyes

Perhaps it was a wish for folks to try to understand where other people were coming from, perhaps it was a cry for a fresh perspective in general.

Innocent, caught up in the machinery
An algorithmic disguise

The entire premise of the song.

Making everything great again

A reference to a certain Presidential candidate and his tendency to always frame his positions as being a “common sense” return to greatness.

Forget about your family and friends

So many people I knew went down political rabbit holes never to emerge; factions formed, contact was broken off, so much hurt happened as a result of people getting caught up in particularly virulent cliques.

Looking for a means to an end

So many people got downright Machiavellian when it came to justifying their beliefs and opinions, especially through vilifying the “other.”

Just don’t want to be alone

Many destructive online communities became self-sustaining cliques where people are so caught up in their community that they dare not abandon the specific belief system that makes it a community to begin with.

Back in September, 2020, Dan Olson released an incredible video about the Flat Earth community which goes deep into this, and draws connections between it and some of the other communities I am indirectly addressing in this song.

Fragile gem, split in two

This line has another specific double meaning.

First of all, I saw a huge uptick in people who began to identify as plural, possibly because of how much time they were spending with themselves and where they started to notice (or perhaps build) lines around the different aspects of their personalities that make them whole. Sometimes it felt like people were purposefully breaking themselves apart, shattering into little shards that could never be reassembled.

Secondly, and more generally, this just referred to the much deeper political divide that had formed, separating society fully into diametrically-opposed camps.

Internal dialog of them versus you

This refers to both of the meanings of the previous line. Some of the plural systems were having a very difficult time agreeing with themselves. Radicalized people with strong political views would engage in straw man tactics to decide that people on the other side were inherently wrong because of what they believed each other to believe, even if those beliefs were not, you know, actually present.

Find acceptance with nothing to do
The voices there are all your own

And this continues the double meaning, with somewhat opposing meanings depending on which side you’re looking at.

If everyone could see
Themselves through some other eyes
Innocent, caught up in the machinery
An algorithmic disguise

No new meaning to this repeat of the chorus.

A deeper body ache

The stress of the Everything made a lot of chronic pain sufferers feel everything to the core.

And make a bad mistake
A million jumping on

Nobody ever wants to become the Internet’s main character. Make a fool of yourself in public by sharing an opinion, and you might just be branded a monster forever.

Sharing a bad take
Nobody can quite shake
The feeling that you’re gone

But sometimes people just dig in their heels, regardless of any impassioned pleas to the contrary, and sometimes everyone else just has to cut their losses and mourn the person they once knew.

[Caught up in/Woke up from] the elephant’s dream

This lyric was originally recorded as a repeat of the first line of the song, but when I perform it live it is now changed, to show a progression and an understanding that it’s possible to pull yourself out of these algorithmic traps.

Getting washed away by the digital stream

But being aware of the situation doesn’t itself solve the problem.

If it feels better to share another meme
Doesn’t make you less alone

A lot of people use memes and cultural touchpoints to bond with each other. There’s nothing wrong with that, but life needs to be more than just sharing the latest weird in-jokes with your circle, right?

Some notes

My explanation of the “fragile gem” section might be seen as saying something that I am not, and I want to go into further detail.

In no way am I calling for a centrist idea of civility between warring camps. Nor am I saying that there are correct ideas on “both sides.” Instead I’m expressing frustration on how there even are tribal “sides” to begin with, and how much the nature of the modern Internet and social media in particular have lead to this societal partitioning.

I am very much of the opinion that we need to do what’s best for each other, and that happens to put me firmly on one side of the divide that shouldn’t even exist to begin with. I’d love to see the gap get filled in, ideally with the corpses of the capitalist systems that created it to begin with. From where I’m standing I feel like things are too far-gone.

I hope I’m wrong, but things are likely going to get way worse before they can get better.

And regarding plurality, I am certainly not making a judgment call or claiming what’s right for people. I’m just observing that the state of the world has led to these internal fractures and that sometimes people are left duct-taping the pieces back together. Really, my belief is that everyone is made up of separate constituent parts, and what defines someone’s “self” as an integrated whole has no true objective basis. I just want to remind people that even if they are multiple individual shards, they still form a singular being, even if it doesn’t feel like it sometimes.

Making of the video

Most of the videos produced for the live fight were just people performing their song in front of a camera, sometimes with loved ones, sometimes overdubbing and editing things into a “live” assemblage. (If you like, you can see the entire event as it unfolded.)

I took a different approach, and in I went all-in on the “algorithmic disguise” aspect, using Snapchat filters to make myself appear to be a number of different things (including, at one point, an actual sockpuppet!), as well as a bunch of other video filters for various effects. All of the video footage was taken while I recorded those parts for the Song Fight! submission, although for the album I rerecorded the vocals. (Regrettably I hadn’t yet thought to make the “woke up” change at that point.)

As mentioned earlier, I had not yet set up my recording studio, so I recorded different parts of it in different rooms of my house. There was also a segment that I animated (the making of which was shown in a timelapse during the post-bridge instrumental). Some commenters thought that this was also a Snapchat filter (which I’d looked into doing and is the style I was going for), but I’d actually keyframed it manually and used EbSynth to interpolate the keyframes, with additional video effects to take stylistic advantage of where EbSynth fell short. I was specifically inspired by Joel Haver’s animation technique (and Joel was kind enough to watch my video and leave a nice comment on it).

The bit with Animal Crossing was in reference to how New Horizons came out at just the right time to become the game of the pandemic. By mid-2021 I was pretty bored with it, but I also happened to have an in-game wardrobe that reflected my real-life clothing and some accessories; I’m not sure how well it comes across in the video, but during the segment where I’m changing my clothes in-game, those changes are also reflected in real life.

I am not too ashamed to admit that playing with my appearance in Animal Crossing actually inspired some real-life stylistic changes at the time, too, including me adopting a new hairstyle for a while.

During the DDR segment I’m actually playing one of my own charts (specifically the one I made for Delicious Candy). I thought that imperfectly playing DDR to the lyrics “A deeper body ache / and make a bad mistake / a million jumping on” was too good to pass up. I was also purposeful at tearing the mask off (to breathe better) only for the camera to cut away just before my face becomes visible.

Aftermath

I did the entirety of the song — video and all — in just over two days.

A couple people complained about it being “over-produced” and losing sight of the purported Song Fight! Live ethos, even though I was working in the same time constraints as everyone else, while simultaneously dealing with not yet being fully set up in my new home. But people love to find things to complain about, I guess. (Which is what I’m doing right now, too.)

Neither the song nor the video won anything in Song Fight!, but it was the song that sparked me to finally complete the album Songs of Substance, which I’d been planning for a few years at that point. There’s some pretty good songs on that album, if I do say so myself! None of them are as deep or meaningful as “Behind a Mask,” but I’m particularly fond of A Long Plastic Hallway.