Grant: Strategic Projects - Group

We propose to create a new system that uses hand gestures to synthesize audiovisual speech and song by means of an intermediate conversion of hand gestures to articulator (e.g., tongue, jaw, lip, vocal chords) parameters of a three-dimensional vocal tract model. The system, called DIVA (DIgital Ventriloquized Actor), will be adaptive which will reduce learning time and increase actors' ability to master the production of complex expressive vocal behavior. The articulatory synthesis system, ArtiSynth, will incorporate new techniques for fast simulation of both vowels and consonants. DIVAs will be used in three composed stage works of increasing complexity to be performed in Canada and internationally, starting with one performer initially and culminating in three performers simultaneously using their natural voices as well as teh hand-based synthesizer. Training performances will be used to study the processes associated with skill acquisition, the coordination of multiple "voices" within and among performers, and the intelligibility and realism of this new form of audio/visual production. This project is a opportunity to blend highly-skilled artistry, sophisticated technlogical innovation, and rigorous scientific investigation into a new means of expression using face and voice that will stimulate audiences as well as capture the imaginations of other artists.