Canada Spends Logo

Explore our Budget 2026 analysis

Grant: Discovery Grants Program - Individual

Queen's University

$36,000.00

Development of Adaptive Mixed Reality Simulation for Training and Education Using Multimodal Machine Learning

FY 2024-25

Summary
Imagine a future where training and education is tailored to each individual student. The proposed research will develop Dynamically Adaptive Mixed Reality Simulation for training and education, particularly for immediate applications in aviation and medical training where simulation is already widely used as an educational tool. Physiological data from a variety of biosensors will be used in a machine learning pipeline to investigate and classify cognitive load (the amount of working memory resources being used), a key construct for optimizing performance in training and education. The modalities used will include well established metrics like electroencephalography (EEG) and eye tracking while incorporating more novel sensors like functional near-infrared spectroscopy (fNIRS). Deep learning and multimodal machine learning approaches will be analyzed and developed to determine how best to combine physiological signals for the creation of higher order classifiers. The data from the machine learning pipeline will be fed back into a mixed reality (MR) simulation. The simulation architecture will facilitate the modification of the user experience to either increase or decrease the difficulty based on the results of the classifier. This process will result in a dynamically adaptive simulation that is personalized, providing improved learning outcomes, and reducing time to competency. The key objectives of this project are to generate a multimodal open-sourced data set from wearable physiological sensors with labels of cognitive load. This data set will be used in conjunction with other open-source datasets to develop a multimodal machine learning classifier of cognitive load for real time applications. Information from the machine learning pipeline will be used to inform the mixed reality simulation that will be specifically designed for adaptation. The successful outcomes of this project can be applied across many domains. Aviation pilot training will be the specific application that will be evaluated as part of this research program. It is anticipated that dynamically adaptive simulation has the potential to reduce the time required for pilots to achieve competency, thus reducing training time and helping to address the worldwide pilot shortage. This research program will train 6 HQP including 1 PhD, 4 Masters and 1 undergraduate student. HQP will be trained as excellent, interdisciplinary researchers who understand the impact of this work for academia, industry, and society. Through my training program and simultaneous professional development, HQP will be uniquely qualified to contribute to the growing fields of machine learning and mixed reality (for educational purposes and beyond), in academia or industry.
Program
Discovery Grants Program - Individual
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data

As published

Application id
RGPIN-2023-05184
Name nom
Hungler, Paul
Department département
Faculty of Engineering and Applied Science
Organization id
29
Institution établissement
Queen's University
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2023
Award amount
36000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
1103
Area of application group en
Health, education and social services
Area of application group fr
Santé publique, éducation et services sociaux
Area of application en
Learning and education
Area of application fr
Apprentissage et éducation
Research subject code
2805
Research subject group en
Artificial intelligence
Research subject group fr
Intelligence artificielle
Research subject en
Learning and inference theories
Research subject fr
Théories de l'apprentissage et l'inférence
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Development of Adaptive Mixed Reality Simulation for Training and Education Using Multimodal Machine Learning
Keywords
adaptive learning; machine learning; multimodal; human computer interface; simulation; mixed reality; virtual reality; augmented reality; training and education
Application summary
Imagine a future where training and education is tailored to each individual student. The proposed research will develop Dynamically Adaptive Mixed Reality Simulation for training and education, particularly for immediate applications in aviation and medical training where simulation is already widely used as an educational tool. Physiological data from a variety of biosensors will be used in a machine learning pipeline to investigate and classify cognitive load (the amount of working memory resources being used), a key construct for optimizing performance in training and education. The modalities used will include well established metrics like electroencephalography (EEG) and eye tracking while incorporating more novel sensors like functional near-infrared spectroscopy (fNIRS). Deep learning and multimodal machine learning approaches will be analyzed and developed to determine how best to combine physiological signals for the creation of higher order classifiers. The data from the machine learning pipeline will be fed back into a mixed reality (MR) simulation. The simulation architecture will facilitate the modification of the user experience to either increase or decrease the difficulty based on the results of the classifier. This process will result in a dynamically adaptive simulation that is personalized, providing improved learning outcomes, and reducing time to competency. The key objectives of this project are to generate a multimodal open-sourced data set from wearable physiological sensors with labels of cognitive load. This data set will be used in conjunction with other open-source datasets to develop a multimodal machine learning classifier of cognitive load for real time applications. Information from the machine learning pipeline will be used to inform the mixed reality simulation that will be specifically designed for adaptation. The successful outcomes of this project can be applied across many domains. Aviation pilot training will be the specific application that will be evaluated as part of this research program. It is anticipated that dynamically adaptive simulation has the potential to reduce the time required for pilots to achieve competency, thus reducing training time and helping to address the worldwide pilot shortage. This research program will train 6 HQP including 1 PhD, 4 Masters and 1 undergraduate student. HQP will be trained as excellent, interdisciplinary researchers who understand the impact of this work for academia, industry, and society. Through my training program and simultaneous professional development, HQP will be uniquely qualified to contribute to the growing fields of machine learning and mixed reality (for educational purposes and beyond), in academia or industry.