Canada Spends Logo

Explorez notre analyse du budget 2026

Grant: Discovery Grants Program - Individual

University of Victoria

$46,000.00

Computational modelling of sonic mimicry for music pedagogy, revitalization, and search

EF 2023-24

Résumé
Sonic mimicry is fundamental for learning to communicate. Children learn new words by imitating the sounds produced by other speakers. Music teaching frequently involves a student trying to repeat what a teacher plays. Mimicry interesting because although the two signals (source and target) are perceived as similar they are produced by different sources. It consists of two tasks: 1) transforming the source signal  to sonically "match" the target signal, 2) judging whether the two signals produced are sonically similar. Both tasks are simple for humans, but challenging for existing deep learning models. Our research focus is to develop sonic mimicry systems based on deep learning. The developed models will be applied in three interdisciplinary contexts: byzantine chant, jazz soloing, and indigenous song/language revitalization.  Byzantine chant was added by UNESCO in 2019 to its list of intangible cultural heritage. Its teaching has been based on oral transmission and sonic mimicry for centuries. Our research has the potential to create more realistic synthetic cantors as well as provide insights about the singing nuances of expert performers and facilitate interaction with large chant databases. Improvised solos are an integral part of jazz. Existing computer improvisation systems typically work by analyzing a corpus of transcribed solos and then generating new solos using machine learning models. They only focus on pitch/duration generation but do not take into account the variations in expression.  Better sonic mimicry models can result in virtual musicians that sound as rich and expressive as the human musicians they are based on. Songs play an important part in language learning. Indigenous language revitalization is constrained by the limited number of fluent speakers. Effective sonic mimicry systems can be used in a variety of ways to support interactive learning especially of accent and prosody. For example, when a student is learning a new word or song phrase based on a recording of an elder a variety of new examples with different voices could be generated. A computer system for judging sonic similarity can be used to identify how close the student rendering is to an original. Many recent advances in Artificial Intelligence (AI) have originated in Canada and there is also growing number of companies involved with it. Human interaction with AI systems is an active research area with a lot of interest from both academia and industry. The research on sonic mimicry systems based on deep learning described in this proposal has the potential to lead to important research insights on making human AI interaction more effective. In addition to advancing the state-of-the-art  in music and audio AI applications it will also provide insights about more effective AI interactions in other areas. This research also has important social implications in terms of preservation of language, culture, and heritage as well as computer-assisted learning.
Programme
Discovery Grants Program - Individual
Emplacement
British Columbia, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data

Tel que publié

Application id
RGPIN-2023-05029
Name nom
Tzanetakis, George
Department département
Computer Science
Organization id
7
Institution établissement
University of Victoria
Province en
British Columbia
Province fr
Colombie-Britannique
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2023
Competition year année de concours
2023
Award amount
46000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
1207
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Information, computer and communication technologies
Area of application fr
Technologie de l'information, des ordinateurs et des réseaux de communications
Research subject code
2514
Research subject group en
Electrical and electronic engineering
Research subject group fr
Digital signal processing
Research subject en
Génie électrique et électronique
Research subject fr
Traitement des signaux numériques
Application title
Computational modelling of sonic mimicry for music pedagogy, revitalization, and search
Keyword
artificial intelligence; computational ethnomusicology; computer-assisted music pedagogy; deep learning; human-in-the loop AI; language learning; music information retrieval
Application summary
Sonic mimicry is fundamental for learning to communicate. Children learn new words by imitating the sounds produced by other speakers. Music teaching frequently involves a student trying to repeat what a teacher plays. Mimicry interesting because although the two signals (source and target) are perceived as similar they are produced by different sources. It consists of two tasks: 1) transforming the source signal  to sonically "match" the target signal, 2) judging whether the two signals produced are sonically similar. Both tasks are simple for humans, but challenging for existing deep learning models. Our research focus is to develop sonic mimicry systems based on deep learning. The developed models will be applied in three interdisciplinary contexts: byzantine chant, jazz soloing, and indigenous song/language revitalization.  Byzantine chant was added by UNESCO in 2019 to its list of intangible cultural heritage. Its teaching has been based on oral transmission and sonic mimicry for centuries. Our research has the potential to create more realistic synthetic cantors as well as provide insights about the singing nuances of expert performers and facilitate interaction with large chant databases. Improvised solos are an integral part of jazz. Existing computer improvisation systems typically work by analyzing a corpus of transcribed solos and then generating new solos using machine learning models. They only focus on pitch/duration generation but do not take into account the variations in expression.  Better sonic mimicry models can result in virtual musicians that sound as rich and expressive as the human musicians they are based on. Songs play an important part in language learning. Indigenous language revitalization is constrained by the limited number of fluent speakers. Effective sonic mimicry systems can be used in a variety of ways to support interactive learning especially of accent and prosody. For example, when a student is learning a new word or song phrase based on a recording of an elder a variety of new examples with different voices could be generated. A computer system for judging sonic similarity can be used to identify how close the student rendering is to an original. Many recent advances in Artificial Intelligence (AI) have originated in Canada and there is also growing number of companies involved with it. Human interaction with AI systems is an active research area with a lot of interest from both academia and industry. The research on sonic mimicry systems based on deep learning described in this proposal has the potential to lead to important research insights on making human AI interaction more effective. In addition to advancing the state-of-the-art  in music and audio AI applications it will also provide insights about more effective AI interactions in other areas. This research also has important social implications in terms of preservation of language, culture, and heritage as well as computer-assisted learning.