Aside from Reactional’s big Unite reveal, where we retrofitted the fantastic Unity Spaceship demo scene to react to music in a myriad of ways (you can read about that in part 1 of this blog series here), we wanted to create an experience for the Unity community that really got to the heart of the power of the Reactional Engine.
It had to be fun, interactive and fully engaging for the Unite audience. So we created a world first by using the Engine to sync live gameplay with a live DJ set, and, send live game data to the lighting desk, triggering a light show based on game play. All in real time. Nothing pre-baked.
Reactional chief product officer Jonas Kjellberg tells more about how we created the Musical Shooter Game Experience.
As part of our previous demo work, I had built a Musical Shooter prototype based on another Unity demo scene, famously known as 3D Game Kit. If you are a Unity developer you will likely already be familiar with this demo. And as you probably know, it is a third person action platformer, showcasing multiple Unity features.
It was perfect for this live gameplay first.
At this point you might want to watch the film. It’s only seven minutes.
So, how did we do it?
Adapting to FPS
First order of business was to reimagine this demo but with a first person camera instead. This involved adding a first person controller along with a rigidbody with camera attached.
Of course the most important part of an FPS is the shooting mechanic. The original demo does not contain any shooting mechanics. But it does however contain a gun model which is not used in the original scene. In the spirit of reusing only the assets found in the demo, I tried to creatively rotate and scale this model in order to have a bunch of different gun types.
To further differentiate between them I created different colored projectiles using the particle system. Essentially creating a few different types of Rail guns.
Musical Shooting
In order to maximise the musical impact to best showcase the Reactional system, I decided that the guns would only be allowed to fire on beat. This is a different take on “rhythm shooter” genre; where Metal Hellsinger awards you for firing on beat. Our version would check to see whether the Fire input was active at the time of every beat, and if so automatically fire a shot.
Most of the logic for this comes from this snippet:
currentBeat = Reactional.Playback.MusicSystem.GetCurrentBeat();
nextBeat = Reactional.Playback.MusicSystem.GetNextBeat(currentGun.fireRate);
offset = 0.125f;
if (nextBeat < currentBeat + offset && prevquant != nextBeat)
{
shootBullet(currentGun);
prevquant = nextBeat;
}In essence we compare the CurrentBeat with the time of the NextBeat. The latter is a quantized time for the occurrence of a future beat division. As you play through the scene you have the chance to pick up better (faster shooting) weapons, each that has a lower musical division.
GetNextBeat(2f)for instance will report the beat that corresponds to the next occurrence of every 2 beats. ie, if the CurrentBeat-time is 2.36 the NextBeat is 4. Essentially that means that we are 1.64 beats away ( 4 – 2.36 )
In comparing nextBeat with currentBeat above, we also add a slight offset to the currentBeat. We need this in order to actually schedule the shot before the beat has occurred. The main reason for doing this is that we want:
- The shot to land ON beat.
- The gunshot sound effect to play in perfect sync with the music.
- To schedule a Musical Stinger when you kill a creature.
A quick showcase of the weapons firing on different subdivisions. White 1/2, Orange 1/4, Pink 1/8-dotted, Blue 1/8, Green 1/16.
Musical Stingers
What really sets Reactional apart as a system is that we don’t think of musical gameplay as just something to do with Rhythm. As most of you are well aware the rhythm game genre has a long tradition, but sadly musical effects in terms of pitched musical sounds are extremely rare. In case they do exist they are often specifically created for a specific song.
Of course, with Reactional you can set up your game exactly as you wish. We chose to be on beat with this particular gaming experience.
Music Analysis
Since all of the music in the Reactional Music Delivery Platform has gone through an AI music analysis pipeline, we have a lot of data on not only the musical timings, but also every chord being played in the music. With this data we can pair our Interactive Music system to procedurally generate musical gestures and stingers that fit to the commercial track, regardless what the track is.
For Unite 2023 we also had to match the DJ’s music set to the gameplay, so we had to ensure we were aware of what tracks might be used throughout the evening in order that the Engine would enable live syncing to gameplay and lighting. This is key as nothing in what we did on the night was pre-baked.
Interactive Theme
Our system is two-fold, as we talked about in Part 1 of this series as well, we not only have a music playback solution for music tracks (paired with analysis data), we also have created a state of the art procedural music generation system that plays back music note-by-note using in game instruments.
Compositions created for this system we call Interactive Themes.
A big difference between this and other generative solutions is that the composer is in complete control, and can compose their music as they normally would, but then add a ton of interactive aspects to it, allowing the music to constantly transform as the gameplay progresses.
We believe that generative music will play an important part in game development.
These Interactive Themes can both be a full game soundtrack for unprecedented interactivity, following the action much like a film score. They can also contain only musical stingers. The real power comes from combining the Interactive Theme with a licensed track. That way we can trigger a musical stinger at any point, as it will automatically conform to the tempo and harmonicity of the current chord of the track.
Pulsing World
To make everything in the world come to life and “dance”, I added a simple script to the environment game objects that essentially scaled the transforms of these objects at different subdivisions of the music. For this I opted to use animation curves and the Reactional beat clock to achieve this.
float frequency = 2f
float scaledValue = _animationCurve.Evaluate(currBeat / frequency % 1f) * Mathf.InverseLerp(CullDistance, 0, dist))
gameObject.transform.localScale = _originalScale *(1f + scaledValue);The above code is approximately what is used. For each gameobject I also had a culling functionality where I could set a maximum distance where an object was allowed to “dance”, and where the effect tapers off in the distance. For instance the “grass” will only dance if you are close to it.
Real World Pulses
Just like I mention in the first part of this series, we wanted to extend this experience out into the crowd and the party space. To do this we built a network solution tapping into the Reactional system. Using this we could send messages to the outside world using the OSC-protocol. We paired my laptop – the macbook I am typing this on right now that actually ran the whole show – to the lighting console, a grandMA3, and send a ton of messages to it for every musical and gameplay event in the game. Using that data our amazing lighting designer mapped up these events to different scene macros. This included:
- Flashes for every shot, with a different colour per weapon (matching on screen), from different positions.
- Big flash when killing a creature.
- Another when picking up better weapons.
- Whole room going red whenever a creature hurt the player.
- Song specific programming.
The Aftermath
All and all this was a fantastic project. Technically everything just worked, even on rehearsals. There is always an impeding sense of doom when trying to pull something like this off live, but seeing it all come together was such a joy. As I was on stage duty I unfortunately missed out on the dancing and drinking, but I had quite the view over all the partygoers going ballistic when the score was about to be beat each round. I woke up with a big smile on my face, and while waiting for my flight a got a nice reminder of the night before – A laptop full of metal debris ashes from the pyro machines. Damn, what a night!
If you have already seen the Unite film and the Games Challenge you will know exactly what I mean. If you haven’t, enjoy.
We’d love to hear from you as well as we build our community of partners, developers and composers working with the Reactional Engine, Reactional Platform and, in the new year, our Composer solution.
It all added up to an incredible experience at Unite. I don’t think I have ever seen a primarily development focused community that engaged in something at such an event. And to those of you who took part, thank you.
About the author
Jonas Kjellberg is a games music composer and CPO at Reactional Music.
