Grant: Discovery Grants Program - Individual
Institut national de la recherche scientifique
$39,000.00
Immersive Multisensory Media: from Quality of Experience to Quality of Life
FY 2023-24
Summary
Immersive communication aims to enable natural experiences and interactions with remote people and environments in ways that blur the line between real and virtual environments. Today, our need for more effective ways to communicate with remote people has never been greater. Existing immersive technologies and applications, however, rely only on two senses: vision and hearing. To enable true immersive communications, we need to involve other senses. Imagine yourself watching a movie where you feel the heat from an explosion or a TV ad where you can smell the food. With advances in 5G, the Tactile Internet, and neurotechnology, the next decade will see the Internet of Senses revolution, where mind, touch, smells, taste, sight, and sound will come together. Ultimately, the success or failure of any new immersive technology will rely on the human experience it provides, and not on the technology it uses. Human factors, such as sense of presence, immersion, cybersickness, and cognitive states all contribute to the user's perceived immersive media experience (IMEx). In turn, objective (instrumental) measurement of these factors could allow for immersive content and sensory stimulation to be updated in real-time, thus maximizing the per-user experience. Measurement of such human factors, however, is extremely challenging and this is where the proposed Research Program comes in. In particular, several short-term objectives have been defined, grouped under three main themes. First, we propose to develop instrumental measures of several factors, including presence, immersion, cybersickness, and emotion, and then gauge their effects on overall perceived audio-visual IMEx. We propose a combination of artifact-robust psychophysiological and behavioural metrics, biosignal enhancement/compression techniques, and user-aware machine learning algorithms. Applications for single and multiple users working collaboratively will be developed, thus building on emerging hyperscanning innovations in cross-subject synchrony and task performance. Second, we propose to expand the developed tools to encompass multisensory immersive media, where haptics and smells will be combined. Lastly, the developed algorithms will all come together and two multisensory applications to improve the quality of life of Canadians will be developed: neurofeedback for anxiety treatment and Alzheimer's disease progression monitoring. The proposed research program will emphasize excellence in training of highly qualified personnel (HQP), who will be part of an inclusive, diverse, and equitable training environment. Given the cross-disciplinary nature of the program, HQP will be exposed to a mix of multisensory signal processing, serious gaming, human factors, and machine learning, thus equipping them with skills and expertise that are in high demand in today's competitive job market. The research program will provide significant economical, technological and societal benefits to Canada.
Program
Discovery Grants Program - Individual
Location
Quebec, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
RGPIN-2021-03246
Name nom
Falk, Tiago
Department département
Energie, Materiaux, Telecommunications
Organization id
56
Institution établissement
Institut national de la recherche scientifique
Province en
Québec
Province fr
Québec
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2023
Competition year année de concours
2021
Award amount
39000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1510
Committee name en
Electrical and Computer Engineering
Committee name fr
Génie électrique et informatique
Area of application code
802
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Information systems and technology
Area of application fr
Systèmes d'information et technologie de l'information
Research subject code
1900
Research subject group en
Biomedical engineering
Research subject group fr
Biomedical engineering
Research subject en
Génie biomédical
Research subject fr
Génie biomédical
Application title
Immersive Multisensory Media: from Quality of Experience to Quality of Life
Keyword
haptics; human factors; immersive media experience; immersive technology; multimedia signal processing; multisensorial media; olfaction; quality of experience; quality of life; virtual reality
Application summary
Immersive communication aims to enable natural experiences and interactions with remote people and environments in ways that blur the line between real and virtual environments. Today, our need for more effective ways to communicate with remote people has never been greater. Existing immersive technologies and applications, however, rely only on two senses: vision and hearing. To enable true immersive communications, we need to involve other senses. Imagine yourself watching a movie where you feel the heat from an explosion or a TV ad where you can smell the food. With advances in 5G, the Tactile Internet, and neurotechnology, the next decade will see the Internet of Senses revolution, where mind, touch, smells, taste, sight, and sound will come together. Ultimately, the success or failure of any new immersive technology will rely on the human experience it provides, and not on the technology it uses. Human factors, such as sense of presence, immersion, cybersickness, and cognitive states all contribute to the user's perceived immersive media experience (IMEx). In turn, objective (instrumental) measurement of these factors could allow for immersive content and sensory stimulation to be updated in real-time, thus maximizing the per-user experience. Measurement of such human factors, however, is extremely challenging and this is where the proposed Research Program comes in. In particular, several short-term objectives have been defined, grouped under three main themes. First, we propose to develop instrumental measures of several factors, including presence, immersion, cybersickness, and emotion, and then gauge their effects on overall perceived audio-visual IMEx. We propose a combination of artifact-robust psychophysiological and behavioural metrics, biosignal enhancement/compression techniques, and user-aware machine learning algorithms. Applications for single and multiple users working collaboratively will be developed, thus building on emerging hyperscanning innovations in cross-subject synchrony and task performance. Second, we propose to expand the developed tools to encompass multisensory immersive media, where haptics and smells will be combined. Lastly, the developed algorithms will all come together and two multisensory applications to improve the quality of life of Canadians will be developed: neurofeedback for anxiety treatment and Alzheimer's disease progression monitoring. The proposed research program will emphasize excellence in training of highly qualified personnel (HQP), who will be part of an inclusive, diverse, and equitable training environment. Given the cross-disciplinary nature of the program, HQP will be exposed to a mix of multisensory signal processing, serious gaming, human factors, and machine learning, thus equipping them with skills and expertise that are in high demand in today's competitive job market. The research program will provide significant economical, technological and societal benefits to Canada.