Canada Spends Logo

Explore our Budget 2026 analysis

Grant: Discovery Grants Program - Individual

Carleton University

$25,000.00

Joint audio-visual signal processing for video conferencing

FY 2011-12

Summary
We are seeing the emergence of new signal processing techniques that combine multiple physiological modalities such as hearing and in vision. These modalities can be in the form of speech, facial movements, and gestures and used to improve signal processing results and human-computer interaction. Areas that can take advantage of multimodal processing includes audiovisual speech recognition, audiovisual speech enhancement, audiovisual speaker detection, audiovisual speech separation, audiovisual affect-sensitive recognition (emotion recognition), and gesture-speech recognition for human-computer interaction. The broader scope of the research program I propose is to develop multimodal techniques for advanced video conferencing. The initial focus is on single channel audiovisual speech separation and underdetermined multi-channel audiovisual speech separation where one or more microphones are placed within a room to record the speech along with one or more video cameras to record lip motion. Other aspects such as audiovisual speaker detection and tracking, and audiovisual speech enhancement will also be part of the research. The problem to address is the ""cocktail party effect"". Consider that you are at a cocktail party where many smaller groups of people are talking. Even though we hear other people talking in the room, humans are quite adept at focusing on one particular speaker and ignoring the speech of the other talkers. The goal of the speech separation is to develop signal processing techniques that can filter out the speech of the other talkers will keeping the speech of just one talker, like in the ""cocktail party effect"". It is expected that including information about lip motion will improve the speech separation over acoustic-only solutions, which degrade when any room reverberation and high levels of noise exist.
Program
Discovery Grants Program - Individual
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data

As published

Application id
249590-2009
Name nom
Dansereau, Richard
Department département
Systems and Computer Engineering
Organization id
23
Institution établissement
Carleton University
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2011
Competition year année de concours
2009
Award amount
25000
Program id
RGPIN
Program nane en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
Discovery grants programs
Group fr
Programmes de subventions à la découverte
Committee code
1079
Committee name en
Electrical and Computer Engineering
Committee name fr
Génie électrique et de l'informatique
Area of application code
514
Area of application group en
Manufacturing processes and products
Area of application group fr
Procédés et produits industriels
Area of application en
Communications equipment
Area of application fr
Matériel de télécommunications
Research subject code
2514
Research subject group en
Electrical and electronic engineering
Research subject group fr
Génie électrique et électronique
Research subject en
Digital signal processing
Research subject fr
Traitement des signaux numériques
Partie
3
Nb partie
5
Application title
Joint audio-visual signal processing for video conferencing
Keyword
NSERC
Application summary
We are seeing the emergence of new signal processing techniques that combine multiple physiological modalities such as hearing and in vision. These modalities can be in the form of speech, facial movements, and gestures and used to improve signal processing results and human-computer interaction. Areas that can take advantage of multimodal processing includes audiovisual speech recognition, audiovisual speech enhancement, audiovisual speaker detection, audiovisual speech separation, audiovisual affect-sensitive recognition (emotion recognition), and gesture-speech recognition for human-computer interaction. The broader scope of the research program I propose is to develop multimodal techniques for advanced video conferencing. The initial focus is on single channel audiovisual speech separation and underdetermined multi-channel audiovisual speech separation where one or more microphones are placed within a room to record the speech along with one or more video cameras to record lip motion. Other aspects such as audiovisual speaker detection and tracking, and audiovisual speech enhancement will also be part of the research. The problem to address is the ""cocktail party effect"". Consider that you are at a cocktail party where many smaller groups of people are talking. Even though we hear other people talking in the room, humans are quite adept at focusing on one particular speaker and ignoring the speech of the other talkers. The goal of the speech separation is to develop signal processing techniques that can filter out the speech of the other talkers will keeping the speech of just one talker, like in the ""cocktail party effect"". It is expected that including information about lip motion will improve the speech separation over acoustic-only solutions, which degrade when any room reverberation and high levels of noise exist.