Canada Spends Logo

Explore our Budget 2026 analysis

Grant: Discovery Grants Program - Individual

University of Waterloo

$27,000.00

A multi-modal sensor fusion architecture for audio-visual speech understanding

FY 2008-09

Summary
Modern speech recognition systems remain challenged in noisy environments despite all the technical advances that have been made to enhance their capabilities. There has been significant research work to address the effect of noise and word ambiguities on recognition performance. For speech recognition systems to be of practical use in noisy environments, such as in automobile, crowded areas and/or simultaneous human computer discourse applications, the issue of robustness must be addressed. The applicant maintains that this can possibly be accomplished by utilizing other sensing modalities to complement the acoustic signal of the speech. An example in line with this strategy is to fuse visual lip movements and expressions with the acoustic signal of the speech so as to maximize information gathering about the words uttered and to minimize the impact of acoustic noise. Understanding speech from visual information is an attractive approach that has captured the interest of many researchers to improve speech recognition performance. Not only does this approach have the potential to improve speech recognition performance but it may also be possible for speech impaired people to interact with devices in human machine interfacing applications. In this research project the applicant investigates the integration of audio and visual lip movement features (during speech signal production) to detect and recognize spoken phrases. A multi-modal fusion system will be designed. The system should be able to acquire visual and acoustic signals of speech and fuse them to detect and recognize words spoken by the speaker in a noisy environment. The system will also be tuned and tested in situations where the speaker is speech impaired. It is expected that the proposed research project will result in the training of 6 highly qualified personnel in human machine interfacing- an area of strategic importance to the Canadian economy.
Program
Discovery Grants Program - Individual
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data

As published

Application id
184129-2007
Name nom
Basir, Otman
Department département
Electrical & Computer Engineering
Organization id
33
Institution établissement
University of Waterloo
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2008
Competition year année de concours
2007
Award amount
27000
Program id
RGPIN
Program nane en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
Discovery grants programs
Group fr
Programmes de subventions à la découverte
Committee code
334
Committee name en
Commun.,Computers & Components
Committee name fr
Communic.,ordin. et composants
Area of application code
700
Area of application group en
Transportation systems and services
Area of application group fr
Services et systèmes de transport
Area of application en
Transportation systems and services
Area of application fr
Services et systèmes de transport
Research subject code
2605
Research subject group en
Robotics
Research subject group fr
Robotique
Research subject en
Pattern analysis and machine intelligence
Research subject fr
Analyse de formes et intelligence artificielle
Partie
2
Nb partie
5
Application title
A multi-modal sensor fusion architecture for audio-visual speech understanding
Keyword
NA
Application summary
Modern speech recognition systems remain challenged in noisy environments despite all the technical advances that have been made to enhance their capabilities. There has been significant research work to address the effect of noise and word ambiguities on recognition performance. For speech recognition systems to be of practical use in noisy environments, such as in automobile, crowded areas and/or simultaneous human computer discourse applications, the issue of robustness must be addressed. The applicant maintains that this can possibly be accomplished by utilizing other sensing modalities to complement the acoustic signal of the speech. An example in line with this strategy is to fuse visual lip movements and expressions with the acoustic signal of the speech so as to maximize information gathering about the words uttered and to minimize the impact of acoustic noise. Understanding speech from visual information is an attractive approach that has captured the interest of many researchers to improve speech recognition performance. Not only does this approach have the potential to improve speech recognition performance but it may also be possible for speech impaired people to interact with devices in human machine interfacing applications. In this research project the applicant investigates the integration of audio and visual lip movement features (during speech signal production) to detect and recognize spoken phrases. A multi-modal fusion system will be designed. The system should be able to acquire visual and acoustic signals of speech and fuse them to detect and recognize words spoken by the speaker in a noisy environment. The system will also be tuned and tested in situations where the speaker is speech impaired. It is expected that the proposed research project will result in the training of 6 highly qualified personnel in human machine interfacing- an area of strategic importance to the Canadian economy.