Grant: Discovery Launch Supplement

Everyday functions often require effective integration of multiple sensory inputs. For example, we read lips when having a conversation with others, in addition to listening. Similarly, we listen for unexpected sounds (such as sirens) when driving a car, in addition to scanning the road. Past studies of multisensory integration have revealed that integrating simple inputs conveyed by different senses often leads to benefits in human perception and performance. Here we are interested in such benefits of multisensory integration can be extended to other types of multisensory stimuli, in particularly those that are connected by abstract associations. To give an example of the abstract associations human make to make sense of their multisensory world: a spiky shape is often paired with a higher-pitched sound or a nonsense word like `kiki,' while a rounded shape is often paired with a lower-pitched sound or `bouba.' These crossmodal correspondences are consistently observed in diverse individuals, cultures, and ages, but whether they enhance perception remains unclear. To address this gap, we propose a series of experiments to investigate the effects of crossmodal correspondence on perception and when they occur. We will develop a new experimental protocol using multisensory illusions (when observers' percepts differ from reality due to multisensory integration) in Aim 1. We will measure and test whether observers are more likely to experience the illusion when the stimulus has crossmodal correspondences. Using this new and other existing experimental protocols, we will compare and contrast the effects of sensory uncertainty (Aim 2), akin to situations where listening is more challenging in a noisier environment. Finally, we will use these protocols to study the effects of cognitive processing resources (Aim 3), akin to situations where a human observer is occupied by other mental processes. This research program will provide empirical data for psychologists and neuroscientists to understand multisensory integration better using a diverse set of stimuli. Knowledge gained from this proposal is expected to advance potential applications of crossmodal correspondences. These applications could be in education, user experience, and technology development. The research program is also expected to benefit highly qualified personnel in a small liberal arts university such that they can acquire skills and experience in experimental psychology, a relevant discipline in Natural Sciences and Engineering.