Grant: Strategic Projects - Group

We propose to advance the state of the art of articulatory speech synthesis research by building and evaluating neural activation-based models to drive a parameterized, 3D biomechanical-articulatory speech synthesizer. Our industry partner, CTF (http://www.ctfmeg.com) provides expertise in magnetoencephalography (MEG) and multimodal sensing of the brain and neural pathways, contributing to developing novel low degree-of-freedom (DoF) speech control strategies. The proposed research targets ICT - Human Interaction with Digital Information, as we focus on fundamental aspects of speech-based natural user interfaces. We plan to work backwards from our biomechanically driven speech synthesizer to determine control strategies that take advantage of the coupling afforded by biomechanics to reduce the DoF needed for producing natural sounding speech. Creating low DoF control representations is an important step for creating talking heads that move and sound natural for avatar-type interfaces needed for natural user interfaces. We have three central goals for this project: 1) improving our existing biomechanics-to-speech model, 2) developing neural control topologies, and 3) building control models for the coupled biomechanical-neural system. These goals will converge to synthesize a neurally controlled talking head for use in natural user interface evaluations. As a capstone demonstration, we will explore having a person speak in real-time using the neurally controlled talking head while in the MEG. Our team of established researchers span the fields of expertise for this project. Drs. Fels, Stavness and Gick have published together on the biomechanics of the oral, pharyngeal and laryngeal complex and articulatory control of speech. CTF engineers have over a decade of experience advancing the state of the art of MEG for sensing neural activation. Our team's world-class modeling, simulation, HCI, speech synthesis control and MEG expertise provides a leading-edge opportunity for HQP to learn techniques for 3D physics-based simulation, advanced speech control, machine learning and neural sensing.