Torin Hopkins, PhD

AI & Music · Extended Reality

Networking AI-Driven Virtual Musicians in XR

Designing AI-driven music interaction for networked VR/AR/MR environments.

Read the paper (PDF)DOI

AI-driven virtual musicians (AIVMs) are more than backing tracks: they are companions in making music. This project networks AIVMs into online jam sessions, so remote human musicians and AI bandmates can play together in augmented reality.

Motivation

Networked music boomed during the COVID-19 pandemic, letting musicians meet regardless of geography. Research has focused on audio and, more recently, on video, point clouds, and avatars. Some platforms can generate accompaniment, but none let players invite AI-driven musicians into a live, networked jam alongside other people.

The system

  • Unity with Nreal AR glasses embodies players and AIVMs as avatars. The glasses’ cameras track hands so musical gestures can be shared.
  • Photon networking synchronizes hand tracking and the AI avatar’s gestures.
  • JackTrip carries near-instantaneous audio between players up to roughly 1,000 km apart.
  • An AI pipeline of LSTM networks trained on the Magenta Groove MIDI dataset generates music, routed through Ableton Live and Max for Live.

Key findings

  • At a public showcase, nearly 200 people saw the system and about 15 played with a networked virtual drummer.
  • Players generally felt their partner’s presence, stayed task-focused, and enjoyed the experience (6.25 of 7).
  • Feedback pointed to beat-transition timing and responsiveness as the next technical priorities.

Video

AI guitarist video

Gallery

A guitarist jamming with an AI-driven virtual guitarist and a remote drummer.
A guitarist jamming with an AI-driven virtual guitarist and a remote drummer.
Playing guitar with an AI-driven virtual guitarist.
Playing guitar with an AI-driven virtual guitarist.
The XR Jam system setup: MIDI instruments, Nreal glasses, JackTrip, and a haptic mat.
The XR Jam system setup: MIDI instruments, Nreal glasses, JackTrip, and a haptic mat.

Publications

  1. XR Jam: Design Considerations for Music Networking with AI in XR

    Torin Hopkins, Suibi Che-Chuan Weng, Rishi Vanukuru, Sasha Novack, Chad Tobin, Emma Wenzel, Amy Banić, Mark D. Gross, and Ellen Yi-Luen Do

    IEEE International Conference on Artificial Intelligence and eXtended and Virtual Reality (AIxVR), Los Angeles, CA, USA, pp. 97–101, 2024.

    AbstractPDFDOI

  2. Networking AI-Driven Virtual Musicians in Extended Reality

    Torin Hopkins, Rishi Vanukuru, Suibi Che-Chuan Weng, Chad Tobin, Amy Banić, and Ellen Yi-Luen Do

    IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct), Demo and Poster tracks, pp. 752–756, 2023.

    DOI

IEEE AIxVR 2024

XR Jam

Design considerations for networked jamming with AI-driven musicians and remote players in extended reality.

ProjectPDFDOI

Frontiers in Virtual Reality · 2022

AR Drum Circle

Networked musical interaction and real-time collaboration between remote drummers in augmented reality.

ProjectPDFDOI