Torin Hopkins, PhD

Neuroscience & BCI · HCI & Interfaces

Stringesthesia

Dynamically shifting musical agency between audience and performer based on the performer’s neurally measured trust in an improvised performance.

Read the paper (PDF)DOI

Stringesthesia is an improvised performance in which the performer’s brain decides how much of the music the audience controls. fNIRS, decoded in real time, reads the guitarist’s sense of "trust" in the audience and projects it as a glowing orb. As trust grows, more stations light up among the seats for playing drums and choosing the chords the guitarist must play.

Trust and musical agency

Interactive performances must decide how much musical agency, meaning control over the musical content, to give the audience versus the performer. Trust is central to that balance, and bio-sensing can make internal states visible. fNIRS offers better spatial resolution than EEG and had been little explored in creative musical settings.

The performance

  • A NIRSport 2 measured 50 channels over the temporoparietal junction, frontopolar area, dorsolateral prefrontal cortex, and medial prefrontal cortex, streamed through Lab Streaming Layer to MATLAB.
  • Trust was shown as the diameter of an orb of particles behind the performer: small when trust was low, large when high.
  • Six color controllers let audience members choose chords. Colors in spectral order mapped to the major-key chords I, ii, iii, IV, V, and vi, which the performer had memorized.
  • Six rhythmic instrument stations were miked to drive reactive visuals. Unity VFX Graph, MadMapper, and five projectors created the environment.
  • The 2 h 15 min concert was split into 14 sets of 2–9 minutes, with rests so the blood-oxygen signal could return to baseline.

Key findings

  • Audience members reported feeling connected to each other and to the performer, combining "the production quality of larger concerts" with "the intimacy of small performances".
  • When trust was low and fewer stations were active, the performer had more musical freedom, revealing a trade-off between audience interaction and artistic control.
  • Some listeners found the trust visualization too abstract, which suggests clearer explanation in future performances.
  • The neural control loop held up despite the motion and noise of a real stage.

Video

Stringesthesia performance video

Gallery

Overhead views of the performance space and projections.
Overhead views of the performance space and projections.
Audience members playing at an instrument station.
Audience members playing at an instrument station.
The guitarist performing while wearing the fNIRS device.
The guitarist performing while wearing the fNIRS device.
The trust orb: a larger diameter meant higher trust.
The trust orb: a larger diameter meant higher trust.
Audience color/chord controllers, reset (left) and with a color selected (right).
Audience color/chord controllers, reset (left) and with a color selected (right).

Publications

  1. Stringesthesia: Dynamically Shifting Musical Agency Between Audience and Performer Based on Trust in an Interactive and Improvised Performance

    Torin Hopkins, Emily Doherty, Netta Ofer, Suibi Che-Chuan Weng, Peter Gyory, Chad Tobin, Leanne Hirshfield, and Ellen Yi-Luen Do

    Audio Mostly (AM ’23), Edinburgh, UK, pp. 9–16, 2023.

    AbstractPDFDOI

  2. Stringesthesia: Dynamically Shifting Musical Agency Between Audience and Performer Based on Trust in an Interactive and Improvised Performance (Work in Progress)

    Torin Hopkins, Emily Doherty, Netta Ofer, Suibi Che-Chuan Weng, Peter Gyory, and Chad Tobin

    New Interfaces for Musical Expression (NIME), Mexico City, Mexico, pp. 116–122, 2023.

    AbstractPDF

  3. Stringesthesia: Interactive Musical Improvisation Using Functional Near-Infrared Spectroscopy

    Torin Hopkins, Emily Doherty, Netta Ofer, Suibi Che-Chuan Weng, Peter Gyory, Chad Tobin, Leanne Hirshfield, et al.

    Front Range Neuroscience Group, Poster Session, 2022.

Frontiers in Computer Science · 2025

BrAIn Jam

Controlling an AI-driven virtual drummer using neural signals (fNIRS) in real time, with a comparison of machine learning models for interpreting neural data.

ProjectPDFDOI

IEEE IS2 2026 · Accepted

CrowdForge

Effects of audio and visual audience participation modalities on agency, collaboration, and engagement in AI-mediated mixed reality performances.

Project