Interfacing Gestural Data from Instrumentalists

This paper presents preliminary work on a system for capturing gestures from music instrumentalists. A saxophone player performed an étude from standard repertoire in two different manners: firstly, constraining their physical gestures to pro- duce sound and execute the written music; secondly, to exaggerate the gestures to include expressions of emotions. The author used a “non-invasive” way to capture the performance gesture using wearable IMU devices with sensor fusion. Using open source 3D creation software, the author extracted motion paths from the data generated by the performer. A 3D Kernel Density Estimation (KDE) algorithm was implemented to create a visualization of density of the trajectories of the head, left elbow and left hand. Analyzing the gestural space of a work of music to develop composition strategies when interfacing the machine and the performer in real time electroacoustic music is highlighted.

Article in Art Music Review

Compositional Strategies in Spectral Spatialization
Martin Jaroszewicz
PhD. Dissertation
Department of Music
University of California Riverside, 2015
Committee: Dr. Paulo Chagas, UCR
Dr. Miller Puckette, UCSD
Dr. Ian Dicke, UCR





Contact me for a copy of my Dissertation in PDF

Abstract

The use of space as a structural part of a composition is a complex issue that involves rethinking the idea of the sound object – as described by Pierre Schaeffer in his Treatise on Musical Objects (1966) – and involves considering movement as a means to manipulate spectrum in the frequency domain. In addition, the contemporary composer who writes acousmatic music needs to consider the materiality of the studio where music is created. These interrelated factors and others, such as the means of reproduction, the acoustics of the venue, the choice of loudspeakers and eventually the software that executes the algorithms for complex calculations of movement and other parameters of sound spatialization, are part of the techné involved in the creative process.¹ Thus, the choice of sound material and the aesthetics of movement should be considered to be elements intrinsically related to the technical means of the performance as spatialization produces an impact on the timbral footprint of sound.

One of the motivations behind the research conducted at EARS was to answer Agostino Di Scipio’s question: “How can I design the tools that are necessary to realize my own idea of composition ?” (Scipio 1995a). The other one, was to realize some ideas inspired by Borges’ story The Library of Babel, which became the starting point behind the Laberintos series of electroacoustic études, each one related to one aspect of working with space and using the electronic music techniques described in this thesis.

There is a discussion involving the meaning of working with space from the point of view of a composer of contemporary electroacoustic music. This dissertation deals with strategies for working with sounds objects whose trajectories are predetermined by the composer during the composition process, as opposed to using space as a resonant body to music. A new definition of the sound object that considers its and its relationship with space is discussed. It views sound objects from a different perspective than Pierre Schaeffer, considering them as volumetric objects that occupy a space like any object or , a view in tandem with speculative realism (Harman 2011). These sound objects are real; they can travel in space and acquire a tangible property.

I describe a unique approach for dealing with sound trajectories in spatialized music using 3D modeling software. Although, none of these applications were designed for music or sound and lack any synthesis capabilities besides the creation of basic waveforms, it is possible to extract the data containing the coordinates of a 3D virtual space in the form of a simple text file. In addition, Blender² features a game engine that can be used to send Open Sound Control (OSC) messages in real time using the Python scripting language. As an alternative, I have created several externals written in C for the Pure Data language for the creation of sound trajectories using ideas taken from parametric design. These objects offer the possibility to work using algorithmic and stochastic approaches to spatialization simplifying the compositional process. The technical part of this dissertation deals with the tools I have developed at EARS and how can be implemented using the techniques exemplified with my own work.

Another aspect of this thesis is the description of the systems available for spatialization in music and how they differ from commercial systems. At EARS, I built and worked with tools that were specifically designed for composition and spatialization thus contributing to the techné and the aesthetics that influenced my musical ideas. The result of the research conducted at EARS and the experienced gained working with Wave Field Synthesis (WFS) systems ³ not only generated new apparatus but suggested a new rethinking of composition in space.

From a technical point of view, among the programming languages for music and sound, I found that Pure Data⁴, Supercollider⁵, Faust, and Chuck are suitable open source tools for the composer working with electronic music as they are portable and available for most platforms including Mac, Linux and Windows.⁶ In addition, Pure Data, with its visual approach, is a great tool for quickly sketching musical ideas and for demonstrating the theory and technique of electronic music without writing lines of code.