Torin Hopkins, PhD

Extended Reality · Neuroscience & BCI

How Late Is Too Late?

Investigating the interaction between visual and audio perceptual delays in networked music.

Read the paper (PDF)DOI
(A) The setup: a participant with AR glasses and a MIDI drum plays with a remote musician’s avatar. (B) The sound-animation latency threshold.
(A) The setup: a participant with AR glasses and a MIDI drum plays with a remote musician’s avatar. (B) The sound-animation latency threshold.

Real-time networked music needs near-instantaneous transmission, and visuals travel slower than audio. How much delay between a remote musician’s sound and their avatar’s animation can players notice, and how much can they tolerate? We measured it in augmented reality.

The study

Twenty-four participants played a hand drum with a prerecorded remote musician rendered as an avatar in AR. Across multiple trials we varied the audio-visual latency, from no delay up to 1,200 ms, during both mimicking and improvising tasks. We analyzed participants’ subjective responses together with the recorded musical data from each session.

Key findings

  • The minimum noticeable delay was between 160 ms and 320 ms.
  • Delay between sound and avatar animation became less tolerable at 320 ms but was never completely intolerable, even at 1,200 ms.
  • Delays in AR visuals were more tolerable than those reported for video streams. Network latency is not the main roadblock to meaningful remote collaboration, and immersive visuals help even when slightly delayed.

Gallery

Study setup with AR glasses and hand drum.
Study setup with AR glasses and hand drum.
Delay scores across latency levels for the mimic and improvise tasks.
Delay scores across latency levels for the mimic and improvise tasks.

Publications

  1. Studying the Effects of Network Latency on Audio-Visual Perception During an AR Musical Task

    Torin Hopkins, Suibi Che-Chuan Weng, Rishi Vanukuru, Emma Wenzel, Amy Banić, Mark D. Gross, and Ellen Yi-Luen Do

    IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Singapore, pp. 26–34, 2022.

    AbstractPDFDOI

  2. How Late is Too Late? Effects of Network Latency on Audio-Visual Perception During AR Remote Musical Collaboration

    Torin Hopkins, Suibi Che-Chuan Weng, Rishi Vanukuru, Emma Wenzel, Amy Banić, and Ellen Yi-Luen Do

    IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), pp. 686–687, 2022.

    AbstractPDFDOI

Frontiers in Virtual Reality · 2022

AR Drum Circle

Networked musical interaction and real-time collaboration between remote drummers in augmented reality.

ProjectPDFDOI