Torin Hopkins, PhD

AI & Music · Neuroscience & BCI

BrAIn Jam

Controlling an AI-driven virtual drummer using neural signals (fNIRS) in real time, with a comparison of machine learning models for interpreting neural data.

Read the paper (PDF)DOI

BrAIn Jam is an AI drummer that listens to the brain. Expert drummers improvised with an embodied AI drummer while functional near-infrared spectroscopy (fNIRS) recorded their brain activity. Individualized models classified each player’s response in real time, and the AI changed its playing whenever the drummer’s brain signaled low rhythmic predictability.

Motivation

Improvising musicians exchange subtle cues that many intuit but that are hard to translate into computer musicianship. AI musicians traditionally "listen and react" via MIDI, audio, or video, which adds latency and ignores the embodied cues humans rely on. Using musical information from the brain had been explored only in limited contexts.

The study

The study had two phases. First, we trained individualized machine learning models on fNIRS data collected during a controlled experiment. Then we used those models to inform an embodied AI-driven virtual drummer in real-time improvised drumming collaboration. The fNIRS cap recorded 34 channels across prefrontal and temporoparietal cortex, and a real-time pipeline preprocessed and classified the signal to adjust the AI’s rhythm.

Key findings

  • We formalized rhythmic predictability as a quantifiable metric combining neural and machine learning data.
  • Improvising with embodied AI significantly activated the temporoparietal junction (TPJ) and regions associated with cognitive workload. The activation depended on whether beats were random or musical.
  • Random Forest models outperformed other models (including LSTM and SVM) at classifying the data, which makes them promising for real-time adaptive systems.
  • Individual preferences matter: some musicians valued the AI’s adaptiveness while others preferred consistency.
  • BrAIn Jam is the neural engine behind the Stringesthesia performance.

Video

BrAIn Jam video

Gallery

A drummer wearing an fNIRS cap improvises with the AI-driven virtual drummer.
A drummer wearing an fNIRS cap improvises with the AI-driven virtual drummer.
Setup: fNIRS device, player input, AI virtual musician display, and data collection backend.
Setup: fNIRS device, player input, AI virtual musician display, and data collection backend.
The embodied AI-driven virtual drummer.
The embodied AI-driven virtual drummer.
System pipeline for post-hoc analysis and real-time classification.
System pipeline for post-hoc analysis and real-time classification.

Publications

  1. BrAIn Jam: Neural Signal-Informed Adaptive System for Drumming Collaboration with an AI-Driven Virtual Musician

    Torin Hopkins, Ruojia Sun, Suibi Che-Chuan Weng, Shih-Yu Ma, James Crum, Leanne Hirshfield, and Ellen Yi-Luen Do

    Frontiers in Computer Science, 7:1570249, 2025. Open access

    AbstractPDFDOI

Audio Mostly 2023

Stringesthesia

Dynamically shifting musical agency between audience and performer based on the performer’s neurally measured trust in an improvised performance.

ProjectPDFDOI