Canada Spends Logo

Explore our Budget 2026 analysis

Grant: Discovery Grants Program - Individual

Ryerson University

$20,518.00

Multi-dimensional and multi-modal integration in perception of vocal music

FY 2011-12

Summary
Auditory researchers have long known that lip-reading is a normal part of everyday speech and that the integration of audio and visual channels determines our perception of the basic units of speech (i.e., phonemes). For example, if someone is presented with an auditory /ba/ and a visual /ga/, they will generally report an illusory percept of /da/. Similar evidence for audio-visual integration in the perception of vocal music has recently been found by the applicant. This evidence comes from experiments with stimuli that are highly controlled as well as more naturalistic experiments involving audio-video recordings of great vocal performers such as Judy Garland. Experiments in the first phase of the proposed research will use motion analysis and visually-based judgments of musical stimuli to characterize facial movements that are reliably associated with specific units of production in vocal music. Experiments in the second phase will assess properties and possible mechanisms of integration in vocal music with the aim of developing a preliminary model of multi-dimensional and multi-modal integration. This model would be of significant interest to music science and may be of importance to leading edge issues within cognitive neuroscience. Insofar as vocal music and prosody are based on similar components, this model may also be useful for advancing our scientific understanding of speech prosody (i.e., intonation). Finally, building on the work in the first two phases of this grant, a third phase of experiments will lead to a proof-of-concept for an innovative assistive technology. The technology would use a visual display (e.g., an iPod) to project either a close-up view of a singer's face or a point-light animation that captures the critical time-varying aspects of facial movement. The efficacy of the proposed technology to enhance the experience of vocal music in hearing impaired listeners will be tested.
Program
Discovery Grants Program - Individual
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data

As published

Application id
341583-2007
Name nom
Russo, Frank
Department département
Psychology
Organization id
43
Institution établissement
Ryerson University
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2011
Competition year année de concours
2007
Award amount
20518
Program id
RGPIN
Program nane en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
Discovery grants programs
Group fr
Programmes de subventions à la découverte
Committee code
12
Committee name en
Psychology: Brain, Behaviour & Cognitive Sciences
Committee name fr
Psychologie: Cerveau, comportement et science cognitive
Area of application code
1210
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Psychology
Area of application fr
Psychologie
Research subject code
6301
Research subject group en
Psychology
Research subject group fr
Psychologie
Research subject en
Sensory systems and perception
Research subject fr
Systèmes sensoriels et perception
Partie
5
Nb partie
5
Application title
Multi-dimensional and multi-modal integration in perception of vocal music
Keyword
NSERC
Application summary
Auditory researchers have long known that lip-reading is a normal part of everyday speech and that the integration of audio and visual channels determines our perception of the basic units of speech (i.e., phonemes). For example, if someone is presented with an auditory /ba/ and a visual /ga/, they will generally report an illusory percept of /da/. Similar evidence for audio-visual integration in the perception of vocal music has recently been found by the applicant. This evidence comes from experiments with stimuli that are highly controlled as well as more naturalistic experiments involving audio-video recordings of great vocal performers such as Judy Garland. Experiments in the first phase of the proposed research will use motion analysis and visually-based judgments of musical stimuli to characterize facial movements that are reliably associated with specific units of production in vocal music. Experiments in the second phase will assess properties and possible mechanisms of integration in vocal music with the aim of developing a preliminary model of multi-dimensional and multi-modal integration. This model would be of significant interest to music science and may be of importance to leading edge issues within cognitive neuroscience. Insofar as vocal music and prosody are based on similar components, this model may also be useful for advancing our scientific understanding of speech prosody (i.e., intonation). Finally, building on the work in the first two phases of this grant, a third phase of experiments will lead to a proof-of-concept for an innovative assistive technology. The technology would use a visual display (e.g., an iPod) to project either a close-up view of a singer's face or a point-light animation that captures the critical time-varying aspects of facial movement. The efficacy of the proposed technology to enhance the experience of vocal music in hearing impaired listeners will be tested.