AI & Music · Extended Reality
CrowdForge
Effects of audio and visual audience participation modalities on agency, collaboration, and engagement in AI-mediated mixed reality performances.
CrowdForge is a live set for guitarist, DJ, and audience. From their phones, audience members vote on tempo, genre, the guitarist’s tone, and the virtual band. Their votes steer a generative AI arranger and a mixed reality stage in real time, so the performers cede creative control and turn audience-induced chaos back into music.
Motivation
Traditional live music is largely one-directional, with minimal interaction between audience and performers. Technology-mediated audience participation (TMAP) lets audiences influence a performance, but balancing interaction with meaningful agency is hard. AI-generated music and mixed reality (MR) open new audio and visual ways to participate, yet little is known about how these modalities shape the audience’s sense of agency, collaboration, and engagement, or how they complement one another.
The performance system
Audience selections are aggregated into one collective configuration covering musical character, tempo, instrumentation, dancer movement, virtual performers, and stage environment. The audio parameters drive a fine-tuned language model that generates multitrack MIDI (using the REMI-z tokenization from our NeurIPS work), played live alongside a guitarist and DJ. The visual selections configure an MR stage populated by virtual musicians and dancers, viewed through Meta Quest 3 headsets and on projection.
The concert study
We ran a mixed-methods concert study comparing three interaction conditions: audio only, visual only, and combined audio-visual control.
Key findings
- Observability, not the number of controls, is the primary driver of audience agency.
- Audio control raised the audience’s sense of agency with large effects. Visual interaction gave easily observable feedback, while AI audio parameters offered macro-structural control over the music.
- Audiences used the generative AI controls to test musical boundaries, creating a productive adversarial tension between crowd and musicians.
- That tension pushed creative limits, deepened engagement, and produced moments of musical coherence when the performers resolved audience-induced challenges.
Video
Gallery
Publications
-
IEEE International Symposium on the Internet of Sounds (IS2), Cannes, France, 2026. Accepted
Related projects
Automatic Arrangement
A unified framework and neural network architecture that enables automatic multitrack music arrangement.
Rhythmic Conversations
Jamming with audience-controlled, improvising AI-driven drummers.
Stringesthesia
Dynamically shifting musical agency between audience and performer based on the performer’s neurally measured trust in an improvised performance.