Grant: Discovery Grants Program - Individual
Simon Fraser University
$46,000.00
Cognitive based computational modelling of human expression
FY 2023-24
Summary
The growing success of dialogue systems research makes conversational agents a perfect candidate for becoming a standard in human computer interaction. The naturalness of communicative acts allows for providing a comfortable ground for the users to interact with. There have been many advances on using multiple communication channels in dialogue systems in the way of simulating humaneness in an artificial agent. However, engineering the usability of the system might have contradicting priorities compared with modeling an emotive believable character agent. One issue is to be able to find a good balance of the intensity and frequency of multimodal affective feedback to guide the dialogue flow. The timing and the quality of the feedback can have varying effects on different users. In this work, we propose a framework to incorporate affective input from the user while not sacrificing the perceived interactivity. We use conversational mirroring mechanisms as a way to generate baselines for interaction, that can be used to dynamically guide the user towards the end goal. The proposed research program will study the human behavior while interacting with these assistive technologies using natural interaction methods in order to create an Embodied Conversational Agent (ECA) that enables users to efficiently achieve their goals. The proposed research will develop new theories, models, algorithms, and technologies to make it easier for users to interact more naturally with less cognitive effort. This work will also improve the fundamental understanding of how such systems affect humans. Highly qualified personnel (HQP) will be trained in artificial intelligence (AI) and human computer interaction (HCI), particularly in natural interaction models. Our system uses a fully body realistic 3D avatar with full facial and gesture parameterized animated control. Building on our virtual avatar system that interacts with humans we will use multimodal affective interaction techniques in a complex framework to coordinate the different techniques in a meaningful way. An example of achieving this type of coordination can be seen in previous work that controls gaze, bodily and facial gestures. Our Objectives are: 1. Developing a framework for an affective conversational agent that can recognize and express multimodal emotional cues. 2. Understanding user perception and expectations of using natural interaction methods and bio-feedback mechanisms while interacting with artificial agents. 3. Understanding the effect of the visual and auditory cues on affective human-computer interaction. 4. Apply the observed user parameters to the framework to provide a personalized, user-aware system that can answer varying user needs. 5. With foundational system fork to specific cognitive social models for ECAs including empathy systems and caregiver system.
Program
Discovery Grants Program - Individual
Recipient
Location
British Columbia, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
RGPIN-2019-06767
Name nom
DiPaola, Stephen
Department département
Interactive Arts and Technology
Organization id
5
Institution établissement
Simon Fraser University
Province en
British Columbia
Province fr
Colombie-Britannique
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2023
Competition year année de concours
2019
Award amount
46000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
801
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Computer software
Area of application fr
Logiciel
Research subject code
6304
Research subject group en
Psychology
Research subject group fr
Cognitive science -- fundamental
Research subject en
Psychologie
Research subject fr
Science cognitive -- principes fondamentaux
Application title
Cognitive based computational modelling of human expression
Keyword
Artificial Intelligence; Cognitive Science; Computer Graphics; Conversational Agents; Deep Learning; Empathy Modelling; Facial Animation; Facial Recognition; Intelligent Agents; Real-time Character Animation
Application summary
The growing success of dialogue systems research makes conversational agents a perfect candidate for becoming a standard in human computer interaction. The naturalness of communicative acts allows for providing a comfortable ground for the users to interact with. There have been many advances on using multiple communication channels in dialogue systems in the way of simulating humaneness in an artificial agent. However, engineering the usability of the system might have contradicting priorities compared with modeling an emotive believable character agent. One issue is to be able to find a good balance of the intensity and frequency of multimodal affective feedback to guide the dialogue flow. The timing and the quality of the feedback can have varying effects on different users. In this work, we propose a framework to incorporate affective input from the user while not sacrificing the perceived interactivity. We use conversational mirroring mechanisms as a way to generate baselines for interaction, that can be used to dynamically guide the user towards the end goal. The proposed research program will study the human behavior while interacting with these assistive technologies using natural interaction methods in order to create an Embodied Conversational Agent (ECA) that enables users to efficiently achieve their goals. The proposed research will develop new theories, models, algorithms, and technologies to make it easier for users to interact more naturally with less cognitive effort. This work will also improve the fundamental understanding of how such systems affect humans. Highly qualified personnel (HQP) will be trained in artificial intelligence (AI) and human computer interaction (HCI), particularly in natural interaction models. Our system uses a fully body realistic 3D avatar with full facial and gesture parameterized animated control. Building on our virtual avatar system that interacts with humans we will use multimodal affective interaction techniques in a complex framework to coordinate the different techniques in a meaningful way. An example of achieving this type of coordination can be seen in previous work that controls gaze, bodily and facial gestures. Our Objectives are: 1. Developing a framework for an affective conversational agent that can recognize and express multimodal emotional cues. 2. Understanding user perception and expectations of using natural interaction methods and bio-feedback mechanisms while interacting with artificial agents. 3. Understanding the effect of the visual and auditory cues on affective human-computer interaction. 4. Apply the observed user parameters to the framework to provide a personalized, user-aware system that can answer varying user needs. 5. With foundational system fork to specific cognitive social models for ECAs including empathy systems and caregiver system.