DualStream is a mobile augmented reality system for remote communication. Using the front and rear cameras of an ordinary phone, it captures, streams, and displays spatial representations of people and their surroundings.
Motivation
In-person interaction relies on our spatial perception of each other and of our surroundings. Video calls show real people but without spatial interaction. AR and VR experiences are immersive and spatial, but they usually replace real people and places with virtual characters and environments. DualStream bridges the two, centered on the device most people already carry.
The system
- The front-facing depth camera captures the user as a 3D hologram while the rear camera captures their surroundings, and the phone tracks the user’s position.
- Users move between representations with different levels of visuospatial fidelity: holograms, spatial video feeds, and environment snapshots.
- Environment snapshots create persistent shared spaces, so people can build blended spaces for collaboration.
Key findings
- In a formative evaluation, users valued being able to interact spatially and to move between representations.
- Participants could see DualStream fitting into their own remote communication practices in the near future.
- The findings point to new opportunities for widely accessible spatial communication tools built around the mobile phone.
Publications
-
DualStream: Spatially Sharing Selves and Surroundings using Mobile Devices and Augmented Reality
IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Sydney, Australia, pp. 138–147, 2023.
Related projects
AR Drum Circle
Networked musical interaction and real-time collaboration between remote drummers in augmented reality.
How Late Is Too Late?
Investigating the interaction between visual and audio perceptual delays in networked music.
Augmented Personification of Intelligent Music Tools
Personifying loopers and generative music tools as humanoid avatars in augmented reality, for creativity and collaboration.