Grant: Discovery Grants Program - Individual

Speech emotion recognition is a rapidly growing field in machine learning, leading to a new generation of human-computer interfaces. Past research has focused on identifying emotion from tone of voice, yet this overlooks a rich source of expressive information - our breath. Here I propose a novel paradigm - Affective Breath Recognition (ABR) - the classification of emotion from audible respiration. In conversation, audible breath - sigh, gasp, huff, or yawn - may encode emotional meaning yet our ability to decode this information is unknown. For example, "huff, I'm fine" conveys dissatisfaction while "I'm fine" suggests contentment. My research will investigate the sound signatures of expressive breath to develop new speech emotion recognition technologies. Incorporating respiratory features into emotion classifiers may boost recognition accuracy. ABR is an attractive feature for today's smart speakers as it requires only a microphone. Yet virtual agents like Alexa and Siri do not incorporate ABR. The reason is straightforward - ABR is a new technology. This research will develop new machine learning models that can provide ABR. Developing new methods for ABR will require a dataset of expressive breath recordings. While many such datasets exist for speech communication, no dataset exists to study emotional respiration. My group previously developed the RAVDESS - the first database of audio-visual expressive speech. The RAVDESS is the de facto standard in speech emotion recognition, with tens of thousands of applications across industry, academia, and the public sector. We aim to repeat this success with BreathCA - the first dataset of expressive audible breath and cardiorespiratory recordings. Identifying the acoustic features of affective breath presents new research challenges. Traditional speech analysis relies heavily on phonation - voiced speech produced through vibration of the vocal folds. Respiration, though, is voiceless, and will require a new set of techniques to quantify the expressed emotion. Our group has developed extensive experience in voice analysis. In our second objective, we will identify new acoustic signatures of affective respiration. In our final objective, we will build new machine learning models for ABR. By using a novel paradigm, this work will deepen our scientific understanding of affective respiratory acoustics and its role in speech communication. 13 trainees will receive specialized training to develop high-demands skills in data science and machine learning. The technology outcomes of this research are well placed for integration into today's smart speakers technologies. Through ABR and BreathCA, this program of research will directly support the federal government's Pan-Canada Artificial Intelligence Strategy by establishing Canada as home to a new subdiscipline in emotion analytics, where the next generation human-machine interfaces will be sensitive to the sound of sigh.