AI & Music
Virtual AI Jam
AI-driven, animated virtual musicians for human-in-the-loop musical improvisation.
Virtual AI Jam is an animated, AI-driven virtual musician you can improvise with on an ordinary laptop. Instead of building costly robots, it gives AI musicianship a visual body, and it was refined through interviews with musicians.
Motivation
AI systems that interact with human musicians often lack any visual representation, or require building robots that play acoustic instruments. Visual representations can reveal performance expression, open new ways to collaborate, and support teaching. A virtual, animated musician makes this far more accessible and affordable.
The pipeline
A player’s MIDI notes are recorded locally (Ableton Live with Max for Live) and used as input to music generation models. The generated output plays through a virtual port that triggers sound samples in Ableton and animations in Unity. We shared the system publicly with a code repository so others can build on it.
Key findings
- Semi-structured interviews and thematic analysis informed successive improvements to the system.
- Many musicians preferred AI improvisation systems they could explicitly control over free-form improvisation with fully autonomous virtual musicians.
Gallery
Publications
-
Virtual AI Jam: AI-Driven Virtual Musicians for Human-in-the-Loop Musical Improvisation
AI Music Creativity Conference (AIMC), Brighton, UK, 2023. Open access
