Canada Spends Logo

Explorez notre analyse du budget 2026

Grant: Discovery Grants Program - Individual

Concordia University

$25,000.00

Multimodal synchronous spatial-temporal acquisition of facial features and tongue for medical applications

EF 2016-17

Résumé
High quality geometry acquisition of time varying surfaces, also known as spatio-temporal geometry reconstruction or 4D surface reconstruction, is a fundamental problem in the fields of computer graphics and computer vision with a very wide range of applications in several medical fields, engineering, tele-presence, entertainment, fashion, and e-commerce. One general goal of my research that I have pursued during my PhD and my post-doctoral studies is to develop fundamental spatio-temporal acquisition methods and integrate them seamlessly in various application domains such as garment capture, face performance capture, free viewpoint video, tele-presence, gaze correction to name a few. A key ingredient to these acquisition methodologies is the combination of multiple sensors to achieve high fidelity reconstruction results. Depending on the application domain, more sensor modalities can be aggregated together: video cameras, hybrid depth+color cameras, temperature cameras, CT-scan, ultra-sound data, MRI data etc. These new data types can provide additional complementary information that can help the reconstruction process. Such multi-modal acquisition systems have received to date relatively little examination from the research community, but they have a great potential particularly in the medical field. During my tenure at ETH Zurich we developed several such multi-modal acquisition methods with applications in real-time performance capture for tele-presence applications and multi-modal acquisition from 3D facial scans and CT-scans with applications in surgery assessment and evaluation. This proposal builds up on this work and following my expertise in multi-modal acquisition systems we will develop new methods that fuse the data from new sensor modalities for a variety of medical applications. There are many synergies and research questions on spatio-temporal acquisition with applications in speech and language therapy, assessment and evaluation, therapy in children with autism spectrum disorder, biofeedback and neurofeedback therapy of facial paralysis and facial kinesis, therapy and computer aided lip-reading for hearing impaired to name a few. For instance, fusing together ultrasound data and video data for real-time synchronous acquisition of tongue and facial performance can lead to better understanding of speech and language disorders and can provide new therapy tools. Combining electromyography (EMG) sensing technology that can measure electrical activity in facial muscles with spatio-temporal 3D acquisition of faces can lead to a better understanding of facial paralysis and related disorders as well as more effective 3D-assessment, evaluation, documentation, training program and therapy progress protocol. Accurate 3D lip motion acquisition can be used to document the lip movements to get optimal exploit of optical speech sound characteristics in speech and language therapies. The proposed research focuses on these three important problems: (a) Fuse ultrasound imaging with spatio-temporal 3D reconstruction of faces for synchronous simultaneous reconstruction of the tongue and face performance during speech (b) Fuse EMG data with with spatio-temporal 3D reconstruction of faces for a better understanding of the correlation between facial muscle and facial expressions and (c) Practical lip movement acquisition during speech using inexpensive consumer level hardware.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Quebec, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data

Tel que publié

Application id
RGPIN-2014-05884
Name nom
Popa, Tiberiu
Department département
Computer Science and Software Engineering
Organization id
70
Institution établissement
Concordia University
Province en
Québec
Province fr
Québec
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2016
Competition year année de concours
2014
Award amount
25000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
Discovery grants programs
Group fr
Programmes de subventions à la découverte
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
801
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Computer software
Area of application fr
Logiciel
Research subject code
2707
Research subject group en
Information technology
Research subject group fr
Technologie de l'information
Research subject en
Graphics
Research subject fr
Infographie
Partie
3
Nb partie
5
Application title
Multimodal synchronous spatial-temporal acquisition of facial features and tongue for medical applications
Keyword
NSERC
Application summary
High quality geometry acquisition of time varying surfaces, also known as spatio-temporal geometry reconstruction or 4D surface reconstruction, is a fundamental problem in the fields of computer graphics and computer vision with a very wide range of applications in several medical fields, engineering, tele-presence, entertainment, fashion, and e-commerce. One general goal of my research that I have pursued during my PhD and my post-doctoral studies is to develop fundamental spatio-temporal acquisition methods and integrate them seamlessly in various application domains such as garment capture, face performance capture, free viewpoint video, tele-presence, gaze correction to name a few. A key ingredient to these acquisition methodologies is the combination of multiple sensors to achieve high fidelity reconstruction results. Depending on the application domain, more sensor modalities can be aggregated together: video cameras, hybrid depth+color cameras, temperature cameras, CT-scan, ultra-sound data, MRI data etc. These new data types can provide additional complementary information that can help the reconstruction process. Such multi-modal acquisition systems have received to date relatively little examination from the research community, but they have a great potential particularly in the medical field. During my tenure at ETH Zurich we developed several such multi-modal acquisition methods with applications in real-time performance capture for tele-presence applications and multi-modal acquisition from 3D facial scans and CT-scans with applications in surgery assessment and evaluation. This proposal builds up on this work and following my expertise in multi-modal acquisition systems we will develop new methods that fuse the data from new sensor modalities for a variety of medical applications. There are many synergies and research questions on spatio-temporal acquisition with applications in speech and language therapy, assessment and evaluation, therapy in children with autism spectrum disorder, biofeedback and neurofeedback therapy of facial paralysis and facial kinesis, therapy and computer aided lip-reading for hearing impaired to name a few. For instance, fusing together ultrasound data and video data for real-time synchronous acquisition of tongue and facial performance can lead to better understanding of speech and language disorders and can provide new therapy tools. Combining electromyography (EMG) sensing technology that can measure electrical activity in facial muscles with spatio-temporal 3D acquisition of faces can lead to a better understanding of facial paralysis and related disorders as well as more effective 3D-assessment, evaluation, documentation, training program and therapy progress protocol. Accurate 3D lip motion acquisition can be used to document the lip movements to get optimal exploit of optical speech sound characteristics in speech and language therapies. The proposed research focuses on these three important problems: (a) Fuse ultrasound imaging with spatio-temporal 3D reconstruction of faces for synchronous simultaneous reconstruction of the tongue and face performance during speech (b) Fuse EMG data with with spatio-temporal 3D reconstruction of faces for a better understanding of the correlation between facial muscle and facial expressions and (c) Practical lip movement acquisition during speech using inexpensive consumer level hardware.