Gestrument and the Philharmonia Orchestra make Interactive Music in the Pioneering Live Digital Production “Dream” by the Royal Shakespeare Company

February 8, 2021

With the latest gaming and theatre technology and an interactive score that responds to the actors’ movements – made possible with Reactional Music (previously Gestrument) technology – Dream gives us a completely new way to experience theatre.

On Friday 12th – Saturday 20th of March 2021 the Royal Shakespeare Company, in collaboration with Manchester International Festival, Marshmallow Laser Feast and Philharmonia Orchestra will stage 10 digital live performances to explore the future possibilities of how you can experience theatre from a remote location. 

Dream is a 50-minute online live performance based on Shakespeare’s A Midsummer Night’s Dream. You can join on your phone, tablet or desktop computer. View the show for free, or buy a ticket to participate as a firefly to take the stage together with the actors. 

Actors controlling the music with their movements

The music is a vital part of the experience of live performances. In Dream, the Philharmonia Orchestra recorded music by Maurice Ravel, Jesper Nordin (founder of Gestrument) and Esa-Pekka Salonen with 100 musicians led by conductor Esa-Pekka Salonen, just before the pandemic struck. Together with the Gestrument technology the recordings are made interactive making the actors also act as musicians, creating the music in real time with their movements. This creates a dynamic live soundtrack that adapts to the expressions of the performers, but will always be in perfect sync with the pre-recorded orchestral score. 

Luke Ritchie, the Philharmonia’s Executive Producer on the project, said: 

“The Philharmonia’s R&D work with Jesper Nordin and [Reactional Music] Gestrument has implications for future musical experiences whether they be online, in-game, as a location-based experience or a live performance in the concert hall. In Dream the actors will use Gestrument to perform music; in the future we want audiences to be playing this role as well, removing traditional barriers between performers and participants, and opening up a new world for the orchestra.”

Live performance and gaming technology come together

The Royal Shakespeare Company has built and led a collaboration with partners using their specialist skills in staging live performance, gaming, technology and research to shape what could become another way to experience theatre. Using advanced technology with motion capture, facial rigging, LED backdrops, and a virtual forest scene rendered in Epic Games’ Unreal Engine the live performances are mixed with pre-recorded animation sequences.

Interactive music in Dream

The audience is led by Puck who takes them from the real world into the digital world. As fireflies the audience can guide Puck through the forest using the movement of their cursor.  The actors perform and respond to audience interaction. This makes each performance unique, as the audience behave differently at each event.

A web-player enables the effective distribution of real-time content from individual audience members to the Unreal Engine server and back to the audience. This new software allows the level of dynamic, real-time interaction working with up to 2000 people in a live environment. 

Tech enables new ways to connect with a remote audience

When the coronavirus traveled the world, so did the shut-down of many cultural events. Now an entire industry is trying to look for new ways to connect with their audience and real-time performances to no longer be bound by physical location. 

Gregory Doran, Artistic Director of the Royal Shakespeare Company said: 

“This R&D has been in train for over 18 months, before we faced the challenges of the global pandemic. It now feels more urgent and relevant for the cultural sector having experienced the impact on our industry. Looking at how audiences can experience live performance in a theatre space, and importantly doing that from anywhere they happen to be at performance time. We’re thinking big – how can tech and live performance create a shared experience between a remote audience and a group of performers?” 

To book your tickets to Dream, go to

See full press release: 

See also:

About Dream

Dream is developed by the Royal Shakespeare Company with global technology partners, Epic Games and Intel. Using motion capture and computer game technologies, the production will deliver a wholly new interpretation of Shakespeare and an entirely new channel for theatre.

The project is funded within the Audience of the Future programme by UK Research and Innovation through the Industrial Strategy Challenge Fund.

More from News