Canada Spends Logo

Explore our Budget 2026 analysis

Grant: Strategic Projects - Group

University of Victoria

$59,100.00

Intelligent human-computer interaction in multimedia performance

FY 2011-12

Summary
We envision a mechanical robot playing acoustic musical instruments on stage, responding and improvising in real time to gesture and sound from a human performer. The project consists of five areas: (I) designing and building robotic musical instruments including harmonium, tampura, sitar, and spinning instruments; (2) real-time extraction of a composite representation of performers' musical gestures from the fusion ofmultiple sensor and audio streams; (3) machine learning to distill gestural data into higher-level musical understanding and to allow the robot's responses to evolve based on what it ""hears""; (4) intelligent multiple-agent-based interaction to model complex interactions between perfonners; and (5) integration of all of the above into innovative multimedia compositions and improvised musical performances on stage, which will surely be a challenging test-bed for robustness and inspiration for ongoing musical and technological refinement. We review prior work in musical robotics, including our own solenoid-based drummer, noting that almost every example is an idiosyncratic, limited, one-of-a-kind system for a specific artistic project. We intend to create a sustainable framework that will support many artists and diverse works as well as to evaluate various design options systematically. Our goal is not merely to produce another robotic instrument to play notes under direct external control, but to produce a robotic musician that in some sense can hear, understand, and react to human performer in real-time, grounded in a representation of specific musical knowledge from Indian, African, Cuban, and Western ""art music"" cultures. In the course of this work we will undertake research challenges including performer characterization and identification, the learning of melodic and rhythmic patterns, domain-specific pattern-based beat tracking and the modeling of anticipation. Finally, the resulting system will be visually engaging in addition to musically interesting, providing a performance element that is often lacking in computer music.
Program
Strategic Projects - Group
Location
British Columbia, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data

As published

Application id
383849-2009
Name nom
Driessen, Peter
Department département
Electrical and Computer Engineering
Organization id
7
Institution établissement
University of Victoria
Province en
British Columbia
Province fr
Colombie-Britannique
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2011
Competition year année de concours
2009
Award amount
59100
Program id
STPGP
Program nane en
Strategic Projects - Group
Program name fr
Projets stratégiques - De groupe
Group en
Research partnerships programs
Group fr
Programmes de partenariat
Committee code
1032
Committee name en
New Media Initiative
Committee name fr
Initiative en nouveaux médias
Area of application code
1200
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Advancement of knowledge
Area of application fr
Avancement des connaissances
Research subject code
7000
Research subject group en
Other studies in natural sciences and engineering
Research subject group fr
Autres études en sciences naturelles et en génie
Research subject en
Other studies in natural sciences and engineering
Research subject fr
Autres études en sciences naturelles et en génie
Partie
3
Nb partie
3
Application title
Intelligent human-computer interaction in multimedia performance
Keyword
NSERC
Application summary
We envision a mechanical robot playing acoustic musical instruments on stage, responding and improvising in real time to gesture and sound from a human performer. The project consists of five areas: (I) designing and building robotic musical instruments including harmonium, tampura, sitar, and spinning instruments; (2) real-time extraction of a composite representation of performers' musical gestures from the fusion ofmultiple sensor and audio streams; (3) machine learning to distill gestural data into higher-level musical understanding and to allow the robot's responses to evolve based on what it ""hears""; (4) intelligent multiple-agent-based interaction to model complex interactions between perfonners; and (5) integration of all of the above into innovative multimedia compositions and improvised musical performances on stage, which will surely be a challenging test-bed for robustness and inspiration for ongoing musical and technological refinement. We review prior work in musical robotics, including our own solenoid-based drummer, noting that almost every example is an idiosyncratic, limited, one-of-a-kind system for a specific artistic project. We intend to create a sustainable framework that will support many artists and diverse works as well as to evaluate various design options systematically. Our goal is not merely to produce another robotic instrument to play notes under direct external control, but to produce a robotic musician that in some sense can hear, understand, and react to human performer in real-time, grounded in a representation of specific musical knowledge from Indian, African, Cuban, and Western ""art music"" cultures. In the course of this work we will undertake research challenges including performer characterization and identification, the learning of melodic and rhythmic patterns, domain-specific pattern-based beat tracking and the modeling of anticipation. Finally, the resulting system will be visually engaging in addition to musically interesting, providing a performance element that is often lacking in computer music.
University of Victoria - $59,100 Grant (2011-12) | Canada Spends