Audio: Sound Design

Sound Design and Audio Production are the most overlooked part of any digital media project, requiring much experience, sensitivity and sensibility to move beyond the culture of "library sound." I have often been horrified by audio thoughtlessly attached to projects, that seems to bear no relation to the onscreen imagery or is of dreadful, kitsch quality.

An interesting possibility with data-based projects is the idea of "sonification" - where datasets, instead of  - or as well as - being transformed into visual representations, can also be usefully transformed into sound environments that deliver new forms of insight to the listener.

AVIARy:

A perhaps amusing example of sonification using Augmented Reality techniques is presented in the following video. What you see is a stage with four "consumers" sitting around a dinner table (bottom left). The table has a camera suspended above it, of which the video is projected onto a large screen, so the audience simultaneously has a top-down view. The performers are served "dishes' of AR fiducial markers, upon which images of the audience and various body parts are composited, driven by the AVIARy system (based upon ARToolkit), The ARToolkit uses pattern recognition algorithms to enable the recognition and tracking of fiducial markers (eg. cards with unique patterns upon them) by a camera attached to a computer, assigning 3D cartesian co-ordinates to the marker and then compositing computer-generated content - in real-time - upon the marker. This enables new types of interaction with virtual content, beyond the conventional mouse/keyboard/joystick-type interfaces - and in the case of AVIARy was extended to fully spatialised audio driving an OSC audio system.

Edible Audience by The Consumers (2005)

This performance arose as the consequence of the development of AVIARy - Audio Visual Interactive Augmented Reality - a system developed in collaboration between the University of Melbourne (Peter Morse, Tim Barrass) and the CSIRO Advanced Audio Interfaces Group (Stephen Barrass, Matt Adcock.) It utilises the ARToolkit developed by the HITLab, and was an experiment in audio augmented reality.

In concert with Alistair Riddell, Anita Fitton (ANU) and OnacloV from the Canberra school of Art we staged this rather weird event for the Liquid Architecture 6 Sound Festival at the National Gallery of Australia, Canberra.

The blurb went:
"To tantalise all senses, Liquid Architecture also presents Edible audience by The Consumers. Four guests dine on auditory aperitifs, feasting on the faces of the audience to produce a gastronomic sound-scape. This interactive sound art interface was programmed by Tim Barrass, and is performed by Stephen Barrass, Anita Fitton and Onaclov from the University of Canberra, and Alistair Riddell from the Australian National University. The Augmented Reality System (AVIARY) used to track the food was developed in collaboration with Peter Morse from the University of Melbourne."

As the video demonstrates the system was overly sensitive to illumination/shadow - we had very little time to finesse the performance, but the basic principle is there. Other interesting ideas we didn't have time to properly explore included such things as using vegemite-on-toast as our fiducial markers - this would have been fun and rendered them actually edible (by Australians at least.)