Canada Spends Logo

Explorez notre analyse du budget 2026

Grant: Discovery Grants Program - Individual

Ryerson University

$20,518.00

Multi-dimensional and multi-modal integration in perception of vocal music

EF 2009-10

Résumé
Auditory researchers have long known that lip-reading is a normal part of everyday speech and that the integration of audio and visual channels determines our perception of the basic units of speech (i.e., phonemes). For example, if someone is presented with an auditory /ba/ and a visual /ga/, they will generally report an illusory percept of /da/. Similar evidence for audio-visual integration in the perception of vocal music has recently been found by the applicant. This evidence comes from experiments with stimuli that are highly controlled as well as more naturalistic experiments involving audio-video recordings of great vocal performers such as Judy Garland. Experiments in the first phase of the proposed research will use motion analysis and visually-based judgments of musical stimuli to characterize facial movements that are reliably associated with specific units of production in vocal music. Experiments in the second phase will assess properties and possible mechanisms of integration in vocal music with the aim of developing a preliminary model of multi-dimensional and multi-modal integration. This model would be of significant interest to music science and may be of importance to leading edge issues within cognitive neuroscience. Insofar as vocal music and prosody are based on similar components, this model may also be useful for advancing our scientific understanding of speech prosody (i.e., intonation). Finally, building on the work in the first two phases of this grant, a third phase of experiments will lead to a proof-of-concept for an innovative assistive technology. The technology would use a visual display (e.g., an iPod) to project either a close-up view of a singer's face or a point-light animation that captures the critical time-varying aspects of facial movement. The efficacy of the proposed technology to enhance the experience of vocal music in hearing impaired listeners will be tested.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Ontario, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data

Tel que publié

Application id
341583-2007
Name nom
Russo, Frank
Department département
Psychology
Organization id
43
Institution établissement
Ryerson University
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2009
Competition year année de concours
2007
Award amount
20518
Program id
RGPIN
Program nane en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
Discovery grants programs
Group fr
Programmes de subventions à la découverte
Committee code
12
Committee name en
Psychology: Brain, Behaviour & Cognitive Sciences
Committee name fr
Psychologie: Cerveau, comportement et science cognitive
Area of application code
1210
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Psychology
Area of application fr
Psychologie
Research subject code
6301
Research subject group en
Psychology
Research subject group fr
Psychologie
Research subject en
Sensory systems and perception
Research subject fr
Systèmes sensoriels et perception
Partie
3
Nb partie
5
Application title
Multi-dimensional and multi-modal integration in perception of vocal music
Keyword
NA
Application summary
Auditory researchers have long known that lip-reading is a normal part of everyday speech and that the integration of audio and visual channels determines our perception of the basic units of speech (i.e., phonemes). For example, if someone is presented with an auditory /ba/ and a visual /ga/, they will generally report an illusory percept of /da/. Similar evidence for audio-visual integration in the perception of vocal music has recently been found by the applicant. This evidence comes from experiments with stimuli that are highly controlled as well as more naturalistic experiments involving audio-video recordings of great vocal performers such as Judy Garland. Experiments in the first phase of the proposed research will use motion analysis and visually-based judgments of musical stimuli to characterize facial movements that are reliably associated with specific units of production in vocal music. Experiments in the second phase will assess properties and possible mechanisms of integration in vocal music with the aim of developing a preliminary model of multi-dimensional and multi-modal integration. This model would be of significant interest to music science and may be of importance to leading edge issues within cognitive neuroscience. Insofar as vocal music and prosody are based on similar components, this model may also be useful for advancing our scientific understanding of speech prosody (i.e., intonation). Finally, building on the work in the first two phases of this grant, a third phase of experiments will lead to a proof-of-concept for an innovative assistive technology. The technology would use a visual display (e.g., an iPod) to project either a close-up view of a singer's face or a point-light animation that captures the critical time-varying aspects of facial movement. The efficacy of the proposed technology to enhance the experience of vocal music in hearing impaired listeners will be tested.