Grant: Discovery Grants Program - Individual
Queen's University
$34,000.00
Audiovisual Speech Perception
FY 2016-17
Summary
Understanding speech is a vital part of our daily life, for social, emotional, and informational purposes. Although audition plays an important role in perceiving speech information, vision is also an integral part of this process. It has been recognized for many years that viewing someone's face during a conversation can boost our speech comprehension. Thus, speech perception, in its most natural form (face-to-face), is visual as well as auditory. This grant will support a research program that aims at understanding how the visual and acoustic linguistic aspects of speech are bound during the perception of spoken language.
Specifically, during this grant period, my students and I will explore the perceptual and neural mechanisms involved in audiovisual speech perception using three methodological approaches. The first involves studies that address the temporal window over which audiovisual speech integration occurs. Since the invention of talking movies, researchers and filmmakers have known that strict temporal overlapping between the video and sound is by no means necessary for people to perceive them as a single event. In other words, people cannot really tell the tracks are out-of-sync over a broad range of temporal offsets. We will explore this phenomenon as a method to understand how the nervous system combines information from different senses. The second component addresses the role of gaze fixations on audiovisual speech perception. Active vision determines what parts of the face are selectively processed and these studies address how this selection influences speech perception. Specifically, we want to determine how information gathering (gaze behaviour) is influenced by the temporal synchrony between the modalities. The final research area involves the use of functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) to study the neural representation of audiovisual speech. The speech stimuli used in all components of the research program are created using innovative animation and computer-based editing procedures.
The overall research program will make significant contributions to our understanding of speech information processing and the neurological mechanisms that carry out this remarkable perceptual skill. In recent years, we have used our controlled animated stimuli to determine the temporal and spatial conditions required for audiovisual integration. With fMRI and the study of neurological patients we have begun to identify the complex network of neurological structures that contribute to this perception. Because the program combines different methodologies and techniques, it provides a rich training environment for students.
Program
Discovery Grants Program - Individual
Recipient
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
RGPIN-2014-04985
Name nom
Munhall, Kevin
Department département
Psychology - Psychology
Organization id
29
Institution établissement
Queen's University
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2016
Competition year année de concours
2014
Award amount
34000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
Discovery grants programs
Group fr
Programmes de subventions à la découverte
Committee code
1502
Committee name en
Biological Systems and Functions
Committee name fr
Systèmes et fonctions biologiques
Area of application code
1200
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Advancement of knowledge
Area of application fr
Avancement des connaissances
Research subject code
6301
Research subject group en
Psychology
Research subject group fr
Psychologie
Research subject en
Sensory systems and perception
Research subject fr
Systèmes sensoriels et perception
Partie
3
Nb partie
5
Application title
Audiovisual Speech Perception
Keyword
NSERC
Application summary
Understanding speech is a vital part of our daily life, for social, emotional, and informational purposes. Although audition plays an important role in perceiving speech information, vision is also an integral part of this process. It has been recognized for many years that viewing someone's face during a conversation can boost our speech comprehension. Thus, speech perception, in its most natural form (face-to-face), is visual as well as auditory. This grant will support a research program that aims at understanding how the visual and acoustic linguistic aspects of speech are bound during the perception of spoken language.
Specifically, during this grant period, my students and I will explore the perceptual and neural mechanisms involved in audiovisual speech perception using three methodological approaches. The first involves studies that address the temporal window over which audiovisual speech integration occurs. Since the invention of talking movies, researchers and filmmakers have known that strict temporal overlapping between the video and sound is by no means necessary for people to perceive them as a single event. In other words, people cannot really tell the tracks are out-of-sync over a broad range of temporal offsets. We will explore this phenomenon as a method to understand how the nervous system combines information from different senses. The second component addresses the role of gaze fixations on audiovisual speech perception. Active vision determines what parts of the face are selectively processed and these studies address how this selection influences speech perception. Specifically, we want to determine how information gathering (gaze behaviour) is influenced by the temporal synchrony between the modalities. The final research area involves the use of functional magnetic resonance imaging (fMRI) and electroencephalography (EEG) to study the neural representation of audiovisual speech. The speech stimuli used in all components of the research program are created using innovative animation and computer-based editing procedures.
The overall research program will make significant contributions to our understanding of speech information processing and the neurological mechanisms that carry out this remarkable perceptual skill. In recent years, we have used our controlled animated stimuli to determine the temporal and spatial conditions required for audiovisual integration. With fMRI and the study of neurological patients we have begun to identify the complex network of neurological structures that contribute to this perception. Because the program combines different methodologies and techniques, it provides a rich training environment for students.