I’ve started to work on a project called New Terrain of Apparition (NTA) with Luc Courchesne. The goal is to connect several hemispheric projection environments (called Panoscopes), so that users can meet and interact in a networked virtual environment. Several cameras are used to capture each user from various angles, allowing an accurate image to be displayed for each relative viewing angle in the virtual world. This means that even in an immersive 360 degree display, users can look each other in eyes and have a real time video chat. Audio and video transmission is handled by Scenic, and management of the virtual environment is handled by the SPIN Framework.
Below is a video of the first prototype, which will be exhibited as part of CODE Live 1 at the Vancouver Olympics:
We recently created an installation to demonstrate our component of the PropulseART project at the Society for Arts and Technology. The goal of the project at large is to connect remote concert venues with high quality video and multi-channel audio. Several open source software components have been released, under the name of Telesceno, which manage real time transmission of audiovisual data. Our Clickable Space authoring suite allows for the 3D modelling of performance spaces so that users from multiple locations can share and interact with each others stages.
Below is a proof-of-concept video about how Clickable Space operates:
On March 6th 2009, we held a networked event where DJs and VJs in Montreal and Vancouver performed simultaneously. Moreover, audience members in each location were able to contribute material in real time with their cell phones using Raw Materials, a software developed by Mike Wozniewski and Alexandre Quessy at the SAT. Below is a documentary video created by Mo Simpson that describes the event:
As you might know, my main colleague and research collaborator who works with me in the design of 3D musical interfaces is Zack Settel. In summer of 2008, he produced Blairatta Policeme, a volumetric concert piece for jazz trio, mobile chamner quartet, and audio-enhanced performance space. See his documentation for more about what this means. On that site, you’ll find a live recording by the CBC, as well as an excellent video that explains the main concepts.
Also most recently, Zack appears in a video interview (seulement en français) on the on the Télé-Québec site. In fact, you can also see me in some of the shots, apparently doing “research”.
As a collaboration between the SAT and Mobile Muse, we have created a component that allows visual artists to collect media content from their audiences during live events. The project is called RawMaterials, and is targetted at VJs using Modul8 for mixing, yet any software capable of receiving OSC messages can be used. Audience members can use their cell phones or other mobile devices to send SMS, MMS, microblob feeds such as Twitter, and even live video using Livecast software. The public can also upload content to social media sites such as Flickr, YouTube, Picasa, etc., and the system can be configured to aggregate content from those sources as well. This can even happen far before the event starts, so that the public can choose the raw materials that will be used during the performance.
In the summer of 2008, we developed a prototype of a multi-player mobile audio game, called SoundPark. The objective of the game was to collect audio clips that were physically scattered throught the real world, and deposit them in a staging area for playback. Central to the design are the themes of highly coupled interaction and communication between players with different roles and an engaging blend of interaction with both the physical and virtual worlds. To this end, numerous technologies including locative sensing, miniature computing, and portable displays had to be integrated with a game middleware and audio scene rendering engine.
I’ve written a much more detailed description on audioscape.org if you are interested, but here is a little documentation video that we put together to illustrate the concepts:
From May 13-15, I was at the Mobile Music Workshop in Vienna, which turned out to be a great forum for the exchange of ideas related to sonic design in mobile settings. There is a nice book published, which includes all the proceedings from this workshop since 2004. So rather than telling you about all of the projects, I’ll urge you to go to mobilemusicworkshop.org and buy one of these books
As for the poster that I presented, click the thumbnail below to see the larger version:
My main activity is the development and programming for the Audioscape Project, which involves anything to do with spatial organization of audio content and sound processing. Generally speaking, we provide the ability to create virtual worlds that function as musical interfaces or installation.
Most recently, however, we have started to add the ability to include mobile and location-based environments, in the outdoors. By using tiny mobile computing devices, GPS, and ad-hoc wireless audio streaming, we have managed to create shared mixed-reality audio environments that multiple users can experience while roaming around outdoors.
On March 11th, 2008, I presented our mobile audio work at Interface[s] Montreal, which is a conference series held each year at the SAT. The theme for this night’s presentations was “Mobility”, and so I talked about the future use of audio in augmented- and mixed-reality applications.
Ménagerie Imaginaire is a musical piece that takes place in a shared virtual environment. However, this piece is rarely performed in the same fashion, since musical material is arranged throughout space and can be visited in a random and nonlinear manner. We typically perform this piece using real instruments & performers on stage, with avatar representations modelled in a virtual 3D world. The audience sees a rendered visualization of that world on screen and hears all sounds spatialized on a multi-channel audio system. The performers navigate through the scene and interact with spatially localized audio processing. Microphones capture sound from their instruments, and sensors are used to control the direction of audio propagation.
The piece was performed at the conference on New Interfaces for Musical Expression (NIME), in New York ( June 2007) and at the Pure Data Convention in Montreal (August 2007). Unfortunately, there were no quality recordings of these events, so we reproduced the performance and filmed it for the video that you can find below.
A reconstruction of the piece, filmed at La Société des Arts Technologiques (Sept 2007)
Rendered for 5.1 channel surround audio. Please inquire for a DVD copy.
