AI & Music · Extended Reality
XR Jam
Design considerations for networked jamming with AI-driven musicians and remote players in extended reality.
XR Jam is a system for playing music in extended reality with AI-driven virtual musicians and remote players. To our knowledge, it was the first system to network human musicians together with AI-driven virtual musicians. It also explores haptic feedback for sensing where sounds come from.
The system
Built on an XR framework, XR Jam enables new forms of interaction, including object manipulation and social drumming with AI-driven virtual musicians (AIVMs) and remote players. Avatar-based interfaces help players overcome social inhibitions. A haptic floor mat conveys the direction of sounds to support spatial awareness.
Key findings
- Pilot studies suggested haptics can aid spatial localization of sounds, though front and back directions remained hard to tell apart.
- The system promotes musical interaction, social engagement, and spatial orientation. Synchronization and player experience need further refinement.
- Playing as avatars made musicians less self-conscious and more willing to improvise.
Gallery
Publications
-
XR Jam: Design Considerations for Music Networking with AI in XR
IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality (AIxVR), Los Angeles, CA, USA, pp. 97–101, 2024.
-
Networking AI-Driven Virtual Musicians in Extended Reality
IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Demo and Poster tracks, pp. 752–756, 2023.
Related projects
Networking AI-Driven Virtual Musicians in XR
Designing AI-driven music interaction for networked VR/AR/MR environments.
Virtual AI Jam
AI-driven, animated virtual musicians for human-in-the-loop musical improvisation.
AR Drum Circle
Networked musical interaction and real-time collaboration between remote drummers in augmented reality.