Grant: Strategic Projects - Group
University of Victoria
$59,100.00
Intelligent human-computer interaction in multimedia performance
FY 2009-10
Summary
We envision a mechanical robot playing acoustic musical instruments on stage, responding and improvising in
real time to gesture and sound from a human performer. The project consists of five areas: (I) designing and
building robotic musical instruments including harmonium, tampura, sitar, and spinning instruments; (2)
real-time extraction of a composite representation of performers' musical gestures from the fusion ofmultiple
sensor and audio streams; (3) machine learning to distill gestural data into higher-level musical understanding
and to allow the robot's responses to evolve based on what it "hears"; (4) intelligent multiple-agent-based
interaction to model complex interactions between perfonners; and (5) integration of all of the above into
innovative multimedia compositions and improvised musical performances on stage, which will surely be a
challenging test-bed for robustness and inspiration for ongoing musical and technological refinement.
We review prior work in musical robotics, including our own solenoid-based drummer, noting that almost
every example is an idiosyncratic, limited, one-of-a-kind system for a specific artistic project. We intend to
create a sustainable framework that will support many artists and diverse works as well as to evaluate various
design options systematically. Our goal is not merely to produce another robotic instrument to play notes under
direct external control, but to produce a robotic musician that in some sense can hear, understand, and react to
human performer in real-time, grounded in a representation of specific musical knowledge from Indian,
African, Cuban, and Western "art music" cultures. In the course of this work we will undertake research
challenges including performer characterization and identification, the learning of melodic and rhythmic
patterns, domain-specific pattern-based beat tracking and the modeling of anticipation. Finally, the resulting
system will be visually engaging in addition to musically interesting, providing a performance element that is
often lacking in computer music.
Program
Strategic Projects - Group
Recipient
Location
British Columbia, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
383849-2009
Name nom
Driessen, Peter
Department département
Electrical and Computer Engineering
Organization id
7
Institution établissement
University of Victoria
Province en
British Columbia
Province fr
Colombie-Britannique
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2009
Competition year année de concours
2009
Award amount
59100
Program id
STPGP
Program nane en
Strategic Projects - Group
Program name fr
Projets stratégiques - De groupe
Group en
Research partnerships programs
Group fr
Programmes de partenariat
Committee code
1032
Committee name en
New Media Initiative
Committee name fr
Initiative en nouveaux médias
Area of application code
1200
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Advancement of knowledge
Area of application fr
Avancement des connaissances
Research subject code
7000
Research subject group en
Other studies in natural sciences and engineering
Research subject group fr
Autres études en sciences naturelles et en génie
Research subject en
Other studies in natural sciences and engineering
Research subject fr
Autres études en sciences naturelles et en génie
Partie
1
Nb partie
3
Application title
Intelligent human-computer interaction in multimedia performance
Keyword
NA
Application summary
We envision a mechanical robot playing acoustic musical instruments on stage, responding and improvising in
real time to gesture and sound from a human performer. The project consists of five areas: (I) designing and
building robotic musical instruments including harmonium, tampura, sitar, and spinning instruments; (2)
real-time extraction of a composite representation of performers' musical gestures from the fusion ofmultiple
sensor and audio streams; (3) machine learning to distill gestural data into higher-level musical understanding
and to allow the robot's responses to evolve based on what it "hears"; (4) intelligent multiple-agent-based
interaction to model complex interactions between perfonners; and (5) integration of all of the above into
innovative multimedia compositions and improvised musical performances on stage, which will surely be a
challenging test-bed for robustness and inspiration for ongoing musical and technological refinement.
We review prior work in musical robotics, including our own solenoid-based drummer, noting that almost
every example is an idiosyncratic, limited, one-of-a-kind system for a specific artistic project. We intend to
create a sustainable framework that will support many artists and diverse works as well as to evaluate various
design options systematically. Our goal is not merely to produce another robotic instrument to play notes under
direct external control, but to produce a robotic musician that in some sense can hear, understand, and react to
human performer in real-time, grounded in a representation of specific musical knowledge from Indian,
African, Cuban, and Western "art music" cultures. In the course of this work we will undertake research
challenges including performer characterization and identification, the learning of melodic and rhythmic
patterns, domain-specific pattern-based beat tracking and the modeling of anticipation. Finally, the resulting
system will be visually engaging in addition to musically interesting, providing a performance element that is
often lacking in computer music.