Documentation / Build a light show

How the light show works

From an audio file to light on the beat: the analysis, the score, the clock, the conductor and the environment.

Most music visualisers react: they listen to the sound that is playing and move a moment later. Rainy's light show is choreographed instead. The server listens to the whole song once, writes down what happens and when, and the browser plays that back in time with the audio. The show knows the drop is coming eight bars before it lands.

This page explains each step. You do not need it to use the show, and you want it if you are going to create an environment.

The pipeline

server, once per song
    audio file  →  analysis  →  score (stored per song)

browser, every frame
    clock          the song's time, locked to the audio
      ↓
    source         the score at that time → musical frame F
      ↓
    conductor      intensity, palette, dimmer, flash limiter → show state S
      ↓
    environment    draws S its own way

Everything up to and including the conductor is shared. An environment only decides what the state of the show looks like, which is why every environment hits the same beat.

The analysis

When a song arrives in the library, a background worker on the server decodes it and measures:

Everything is measured relative to the song itself. A quiet acoustic track still uses the full range of the lights, and a "drop" in a ballad is its loudest lift. How loud and percussive the song is in absolute terms goes into the profile's intensity.

The result is the score: a small JSON document stored once per song. The analyser has a version number, and a score from an older version is redone when it is next needed. Part of the analysis can run on an NVIDIA graphics card; see GPU acceleration.

The clock

The browser never listens to the audio to run a scored show. It reads the player's position, smooths it, and adds your Sync Offset. That is the show's clock.

Every part of the show is computed from that time: "what is happening at 1:32.48?" rather than "what changed since the last frame?". This has consequences you can see:

• Seeking lands on exactly the right look, forwards or backwards.
• Pausing holds the show still, dimmed, and it resumes in place.
• A dropped frame cannot put the show out of step. The next frame asks the same question again.

The frame

Each frame, the score is read at the clock's time to make a musical frame: the energy in each band right now, which drums just hit, the position within the beat and the bar, the current section and how far through it the song is.

Because the score covers the whole song, the frame also looks ahead: how long until the next section, until the next drop, how far through a build the song is, whether a silence is about to hit. That look-ahead is what lets the lights rise through a build and cut to black in the gap before a drop.

The live fallback

A song that has not been analysed yet has no score. Until it has one, Rainy listens to the audio as it plays and estimates the same frame in real time: band energy, drum hits and a guessed beat. There is no look-ahead, so the show follows the music instead of anticipating it. The player switches to the score as soon as the analysis finishes.

The conductor

The conductor turns the musical frame into the state of the show, and it is the same for every environment:

• Intensity: how hard the show goes, from the song's profile or from your Intensity setting.
• Pulses: a kick, snare or hi-hat becomes an envelope that snaps up and fades in a fraction of a second.
• Phrases: sections are counted in phrases of eight bars, so the look can change within a long section without changing at random.
• Palette: five colours derived from the section's key and mode, the song's genre and your accent colour. It glides from one section's palette to the next, and snaps on a drop.
• Master dimmer: full while playing, low while paused, black during a silence, and a short dip in the last beat before a drop.
• Flash limiter: every flash an environment wants goes through the conductor, which allows at most three full flashes a second. With Reduce Flashing on, they become soft colour pulses. Because the limit lives here, it holds for every environment.

The lyrics

When lyrics are on stage, a shared timeline says what is sung when: lines, the start and end of each word, backing vocals (the words in parentheses) kept apart from the main line, and instrumental breaks. Word times are exact once the song's lyrics have been analysed, and estimated from the line's length before that.

How the words look is up to the environment. Classic sets them under the artwork with follow spots; Voyage builds them in the air over the road. Both light the same word at the same moment.

The environment

An environment receives the state of the show every frame and draws it. Rainy is its host:

• It gives the environment its own layer over the stage and fades it in and out.
• It arranges the player around it: as it is (stage) or folded away (immersive).
• It tells it the song's score, title and cover when they change.
• It removes everything the environment added when it leaves.
• If the environment fails, Rainy takes it off stage and continues in Classic.

Classic is built in. Any other environment is a package, loaded only when someone picks it. See Light show environments.

Keeping it smooth

The show measures how long its frames take and adjusts as it goes:

• It has three quality levels. If frames run long it steps down (lower resolution, fewer effects); when there is room again, it steps back up.
• Low Performance Mode fixes the lowest level and draws 30 frames a second.
• If the show still struggles at the lowest level for about ten seconds, it offers Low Performance Mode once.

Each environment decides what a quality level means for it: Classic drops haze and glow, Voyage lowers its resolution and its bloom.

Related

• Light show
• Light show environments
• Create an environment
• Environment API reference
• Lyrics