Grant: Discovery Grants Program - Individual
Queen's University
$36,000.00
Development of Adaptive Mixed Reality Simulation for Training and Education Using Multimodal Machine Learning
FY 2023-24
Summary
Imagine a future where training and education is tailored to each individual student. The proposed research will develop Dynamically Adaptive Mixed Reality Simulation for training and education, particularly for immediate applications in aviation and medical training where simulation is already widely used as an educational tool. Physiological data from a variety of biosensors will be used in a machine learning pipeline to investigate and classify cognitive load (the amount of working memory resources being used), a key construct for optimizing performance in training and education. The modalities used will include well established metrics like electroencephalography (EEG) and eye tracking while incorporating more novel sensors like functional near-infrared spectroscopy (fNIRS). Deep learning and multimodal machine learning approaches will be analyzed and developed to determine how best to combine physiological signals for the creation of higher order classifiers. The data from the machine learning pipeline will be fed back into a mixed reality (MR) simulation. The simulation architecture will facilitate the modification of the user experience to either increase or decrease the difficulty based on the results of the classifier. This process will result in a dynamically adaptive simulation that is personalized, providing improved learning outcomes, and reducing time to competency. The key objectives of this project are to generate a multimodal open-sourced data set from wearable physiological sensors with labels of cognitive load. This data set will be used in conjunction with other open-source datasets to develop a multimodal machine learning classifier of cognitive load for real time applications. Information from the machine learning pipeline will be used to inform the mixed reality simulation that will be specifically designed for adaptation. The successful outcomes of this project can be applied across many domains. Aviation pilot training will be the specific application that will be evaluated as part of this research program. It is anticipated that dynamically adaptive simulation has the potential to reduce the time required for pilots to achieve competency, thus reducing training time and helping to address the worldwide pilot shortage. This research program will train 6 HQP including 1 PhD, 4 Masters and 1 undergraduate student. HQP will be trained as excellent, interdisciplinary researchers who understand the impact of this work for academia, industry, and society. Through my training program and simultaneous professional development, HQP will be uniquely qualified to contribute to the growing fields of machine learning and mixed reality (for educational purposes and beyond), in academia or industry.
Program
Discovery Grants Program - Individual
Recipient
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
RGPIN-2023-05184
Name nom
Hungler, Paul
Department département
Faculty of Engineering and Applied Science
Organization id
29
Institution établissement
Queen's University
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2023
Competition year année de concours
2023
Award amount
36000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
1103
Area of application group en
Health, education and social services
Area of application group fr
Santé publique, éducation et services sociaux
Area of application en
Learning and education
Area of application fr
Apprentissage et éducation
Research subject code
9999
Research subject group en
Not available
Research subject group fr
Not available
Research subject en
Non disponible
Research subject fr
Non disponible
Application title
Development of Adaptive Mixed Reality Simulation for Training and Education Using Multimodal Machine Learning
Keyword
Adaptive Learning; Augmented Reality; Human Computer Interface; Machine Learning; Mixed Reality; Multimodal; Simulation; Training and Education; Virtual Reality
Application summary
Imagine a future where training and education is tailored to each individual student. The proposed research will develop Dynamically Adaptive Mixed Reality Simulation for training and education, particularly for immediate applications in aviation and medical training where simulation is already widely used as an educational tool. Physiological data from a variety of biosensors will be used in a machine learning pipeline to investigate and classify cognitive load (the amount of working memory resources being used), a key construct for optimizing performance in training and education. The modalities used will include well established metrics like electroencephalography (EEG) and eye tracking while incorporating more novel sensors like functional near-infrared spectroscopy (fNIRS). Deep learning and multimodal machine learning approaches will be analyzed and developed to determine how best to combine physiological signals for the creation of higher order classifiers. The data from the machine learning pipeline will be fed back into a mixed reality (MR) simulation. The simulation architecture will facilitate the modification of the user experience to either increase or decrease the difficulty based on the results of the classifier. This process will result in a dynamically adaptive simulation that is personalized, providing improved learning outcomes, and reducing time to competency. The key objectives of this project are to generate a multimodal open-sourced data set from wearable physiological sensors with labels of cognitive load. This data set will be used in conjunction with other open-source datasets to develop a multimodal machine learning classifier of cognitive load for real time applications. Information from the machine learning pipeline will be used to inform the mixed reality simulation that will be specifically designed for adaptation. The successful outcomes of this project can be applied across many domains. Aviation pilot training will be the specific application that will be evaluated as part of this research program. It is anticipated that dynamically adaptive simulation has the potential to reduce the time required for pilots to achieve competency, thus reducing training time and helping to address the worldwide pilot shortage. This research program will train 6 HQP including 1 PhD, 4 Masters and 1 undergraduate student. HQP will be trained as excellent, interdisciplinary researchers who understand the impact of this work for academia, industry, and society. Through my training program and simultaneous professional development, HQP will be uniquely qualified to contribute to the growing fields of machine learning and mixed reality (for educational purposes and beyond), in academia or industry.