Leveraging Behavioral Models of Sounding Objects for Gesture-Controlled Sound Design

Introduction:

The authors of the paper present a new intuitive approach to Foley and sound design techniques through the use of "off the shelf" motion sensing devices that allow for quick and fluid interaction with sound trigging and sound modelling based on the everyday action of interacting with objects. The authors use the traditional methods of Foley and sound design and combine them with an exciting more natural workflow.

Review

Foley artists and sound designers spend a long time automating effects, synchronising sounds, scrolling through banks and recording new sounds. This process can be very slow, normally requires manual setup and all the parameters are inputed through the keyboard and mouse. The authors of the paper have presented a faster, more natural way for Foley artists and sound designers to interacte with and create their soundscapes.

With the inventions of commercial motion detection devices and OSC (Open Sound Control) the possibilities for interaction with digital sound has widely opened. The authors of the paper have designed a toolbox of plugins that work off the basic behavioural models of physical sounding objects that combines the characteristic methods of a Foley workflow with the newly available digital sound design methods.

The core principal behind this system was to design a set of models that mimic the behavioural patterns of physical sounding objects. The user must interact with the controller (eg. Wii, Kinect, Move) in the same way they would interact with the physical object that they are modelling. The system is broken down into several pseudo-physical models that focus on instantly giving the sound designer a wide verity of sounds and modulation techniques.

Below is a video of the Wii controlling the same DAW (Ableton Live) that the authors used for testing .

I find this paper very interesting. This is currently a very busy area, especially in the field of digital music performance. The authors of the paper have broke their system down into a intuitive set of models that are aimed particularly at sound designers. This is the first purpose built system that uses motion detection technology I have seen for Foley artists and sound designers. It tested well with professional sound designers who said "the system was particularly suitable for exploring the possibilities of the sounds and quickly recording ideas". User testers were drawn to the fun factor of the system but did express the need for a high accuracy in the final sound.

For more information on David Black and to review many of the other sound design projects he has worked on visit: http://www.daveblack.org/

Reference:

Kristian Gohlke, David Black, and Jörn Loviscach. 2010. Leveraging behavioral models of sounding objects forgesture-controlled sound design. In Proceedings of the fifth international conference on Tangible, embedded, and embodied interaction (TEI '11). ACM, New York, NY, USA, 245-248. DOI=10.1145/1935701.1935750 http://doi.acm.org/10.1145/1935701.1935750

Hypo Chrysos: Mapping in Interactive Action Art Using Bioacoustic Sensing

Introduction:

Marco Donnarumma presents to us Hypo Chrysos. Hypo Chrysos is a work of action art and biophysical media. Audio and visual content is driven by continuous bioacoustic signals. These bioacoustic signals comprise of blood, muscle sound bursts and bone crackles as a result of pulling heavy weights attached to the performers arms.. The bioacoustic signals are amplified giving a low frequency sound and then distorted to give mid range and high frequencies. All the content is created in real time and solely depends on how the performer interacts with the weights on stage.

Body:

All audio and visual content is created in real time by the performers body using the Xth-sense. The Xth-sense, a new instrument and recently voted "world’s most innovative new musical instrument" by the Georgia Tech Center for Music Technology (US),  works by attaching wearable biosensors to the body. These biosensors work with a digital framework that processes real-time acoustic biosignals. These biosignals are then amplified and played out over several speakers.  The performer straps two weights (30kg combined) to their arms. The music is created by the performer pulling these weights around the stage. The performer must force themselves through the pain until the piece is completed. The strain level of the performer's body defines the piece of music being played back to the audience. At first the build of the bioacoustic sounds from the viens and muscles build slowly, delivering low punchy sounds. As the sounds build they are distorted and fed back into the back into the piece creating higher sounds. 

Watch Marco Donnarumma perform live here: Hypo Chrysos | Action art for vexed body and biophysical media (Xth Sense)

Hypo Chrysos is a fascinating and inspiring piece, to achieve a musical performance the performer must go through a certain level of pain and suffering. I see it almost as a physical metaphor to the hard work that musicians go through to bring a new piece of music to the world. This piece of art also represents inner musical workings of the body in a way never heard before.


Additional information: Marco Donnarumma's piece is inspired by "the sixth Bolgia of Dante’s Inferno, located in one of the lowest of the circles of hell. Here, the poet encounters the hypocrites walking along wearing gilded cloaks filled with lead. It was Dante’s punishment for the falsity hidden behind their behaviour; a malicious use of reason which he considered unique to human beings."