Grant: Discovery Grants Program - Individual

Affective trends can directly influence our short- and long-term mental health. Understanding these trends may give us the opportunity to interrupt detrimental habits or develop beneficial ones. In this proposal, we present a research road map for the development of a cloud-based framework that continuously assesses the emotional status of users. The framework will consist of pre-devised components that permit the analysis of affective information at various scopes and granularities. The framework relies on 1) personal computing devices and wearable sensors to collect data and 2) cloud servers running machine learning models to make sense of the obtained information. The framework associates estimated emotions with a context that may provide useful indications about the roots of emotional trends. Hence, users can surmise the possible causes of positive or negative affective changes. The framework monitors emotions at three levels: -Micro-monitoring: Refers to affective monitoring at the individual level. The user's emotions are estimated from multiple modalities, including audio, video, and physiological channels. The affective data is supplemented by a context that includes information about the user's behavioral patterns. -Meso-monitoring: Corresponds to affective monitoring for a group of individuals that share circumstances that may affect their emotional wellbeing (e.g., military personnel or firefighters in the same unit). -Macro-monitoring: Denotes monitoring at the metropolitan, regional, or national level. Data is scraped from social media to track public sentiment and the prevalence of mental illnesses in a population. The framework will provide a common visualization module that presents the assessed emotional information with respect to contextual data. The short-term objectives of this project are: 1- Developing multi-modal emotion recognition methods using state-of-the-art deep learning approaches. 2- Developing natural language processing methods capable of analyzing the affective content of social media posts and speech transcripts. 3 - Developing non-contact physiological signal measurement methods using a combination of signal processing and deep learning techniques. 4 - Developing visual analytics modules that convey patterns in emotions with respect to contextual data. 5 - Developing a cloud-based framework that integrates the proposed contributions. The framework can be adopted for diverse applications.