I went to a meeting on Friday to discuss the development of a standard for describing spatial audio, called SpatDIF. There is now an official website and mailing list, so go check it out at www.spatdif.org.
The idea is quite nice, because it allows one to separate the contents of the 3D scene from the audio renderer. All that is required is to format messages for sound source positions and directivity in this OSC-style format:
Then, any SpatDIF-compatible renderer will be able to spatialize the sounds on the particular audio display that it is written for (eg, binaural, 5.1 surround, VBAP, Ambisonics, etc).
In the summer of 2008, we developed a prototype of a multi-player mobile audio game, called SoundPark. The objective of the game was to collect audio clips that were physically scattered throught the real world, and deposit them in a staging area for playback. Central to the design are the themes of highly coupled interaction and communication between players with different roles and an engaging blend of interaction with both the physical and virtual worlds. To this end, numerous technologies including locative sensing, miniature computing, and portable displays had to be integrated with a game middleware and audio scene rendering engine.
I’ve written a much more detailed description on audioscape.org if you are interested, but here is a little documentation video that we put together to illustrate the concepts:
My main activity is the development and programming for the Audioscape Project, which involves anything to do with spatial organization of audio content and sound processing. Generally speaking, we provide the ability to create virtual worlds that function as musical interfaces or installation.
Most recently, however, we have started to add the ability to include mobile and location-based environments, in the outdoors. By using tiny mobile computing devices, GPS, and ad-hoc wireless audio streaming, we have managed to create shared mixed-reality audio environments that multiple users can experience while roaming around outdoors.
On March 11th, 2008, I presented our mobile audio work at Interface[s] Montreal, which is a conference series held each year at the SAT. The theme for this night’s presentations was “Mobility”, and so I talked about the future use of audio in augmented- and mixed-reality applications.
Ménagerie Imaginaire is a musical piece that takes place in a shared virtual environment. However, this piece is rarely performed in the same fashion, since musical material is arranged throughout space and can be visited in a random and nonlinear manner. We typically perform this piece using real instruments & performers on stage, with avatar representations modelled in a virtual 3D world. The audience sees a rendered visualization of that world on screen and hears all sounds spatialized on a multi-channel audio system. The performers navigate through the scene and interact with spatially localized audio processing. Microphones capture sound from their instruments, and sensors are used to control the direction of audio propagation.
The piece was performed at the conference on New Interfaces for Musical Expression (NIME), in New York ( June 2007) and at the Pure Data Convention in Montreal (August 2007). Unfortunately, there were no quality recordings of these events, so we reproduced the performance and filmed it for the video that you can find below.
A reconstruction of the piece, filmed at La Société des Arts Technologiques (Sept 2007)
Rendered for 5.1 channel surround audio. Please inquire for a DVD copy.
4Dmix3 is an immersive multi-user remixing installation, that uses a ceiling-mounted camera to track many participants moving within a gallery space. The users can see their avatar representations in a mirror-like display, allowing them to localize themselves within a field of virtual sound sources. In fact, a plethora of musical loops are scattered throughout the environment, each of which are phase-locked to match the rhythms of the others.
In the end, the user experiences immersion in a virtual world, where movements guide virtual microphones through a forest of audiovisual sculptures. The sculptures emit sound that feed the users’ mix as he or she travels through through the musical landscape.
