Grant: Discovery Grants Program - Individual

Multimodality plays a vital role in creating delightful experiences. By stimulating senses beyond vision and hearing, creators can engage users on multiple experiential levels. Despite the awareness of the importance of multimodality in achieving rich immersive experiences, significant challenges hold back its thoughtful integration in mainstream entertainment media. Indeed, academics and industry experts have expressed difficulties associated with the design for novel hardware platforms and for the intricacies of multimodal perceptual integration. As a consequence, their adoption typically carries risks that, at best, leads to teams avoiding their use entirely and, at worst, to the deployment of products that actively hurt its public perception and that of multimodal experiences. My long-term research goal is to support the democratization of multimodal interaction design by building an open platform that becomes the one-stop shop for tools that facilitate the study, creation, deployment and evaluation of rich multisensory interfaces. In the next 5 years, my team and I will work towards this goal from three complementary perspectives: Axis 1 - Systems to Facilitate Multimodal System Study and Use To facilitate the adoption of multimodal interactions, my team and I will develop a suite of hardware and software tools aimed at attenuating key methodological challenges associated with the optimal use and study of multimodal systems by end-users. Axis 2 - Multimodal Stimuli Authoring Support To empower both novice and expert practitioners in designing multimodal stimuli that meet specific experiential goals and minimize their need to manipulate low-level stimuli characteristics, my team and I will work towards establishing the theoretical and practical foundations of generative AI agents-based multisensory experience design and evaluation support. Axis 3 - Multimodal Interactions and Perception Using tools developed in A1 and A2, we will design and evaluate novel multimodal interaction techniques that will not only showcase our platform, but also advance our understanding of multimodal perceptual integration in realistic application scenarios. The creation of user-friendly hardware and software tools in my research program aims to broaden participation in multimodal interactions across academia and industry. These tools will make it easier for a wider range of researchers, students and industry practitioners to engage with and contribute to advancements in human-computer interactions, promoting innovation and productivity. The proposed platform will serve as catalysts for innovation, opening doors to new possibilities and applications in various sectors, driving forward the adoption of multimodal interactions in mainstream entertainment.