Grant: Discovery Grants Program - Individual
École de technologie supérieure
$29,000.00
Combined biplanar X-ray and RGBD images 3D reconstruction of the musculoskeletal system for augmented reality clinical follow-up.
FY 2023-24
Summary
Imagine going to the hospital, taking a 1 min X-ray scan like the ones at the airport security and when you see the doctor, she/he has a tablet and sees your bones in the right positions in your body via a precise 3D augmented reality (AR) app. The doctor asks you to walk to identify the origin of that pain in your right knee and tells you immediately, while showing you your knee joint (the bones) superimposed on the image of your real knee, what the problem is. Or going to the ER with a fractured arm bone and seeing the precise depiction of the fracture inside your arm on the tablet, which the doctor is using to plan the best approach in putting the bone back together. This is the kind of system that we intend to make possible by building specialized 3D models of bones, developing algorithms to construct them from optical and X-ray images, and building an AR application to show the bones in the right place on a live video feed. Our research group at the LIO-ÉTS-CRCHUM has been working over the years building 3D models from low-dose biplanar X-ray images to facilitate the extraction of clinical parameters needed to diagnose conditions of the musculoskeletal system, e.g. scoliosis, or to analyze posture or walking disorders. We have developed expertise in the identification of bones, the rigid and non-rigid 2D/3D registration of 3D models and 2D images, pose estimation and AR applications development. We intend to expand our research efforts to build a multimodal acquisition system combining X-ray and image+depth (RGBD) imaging devices to construct a personalized 3D model of the patient that can be used later for real-time analysis of the musculoskeletal system. The proposed methodology is based on the use of deep learning approaches to identify the bones in the images and modify a combined statistical 3D model of the bones and the body envelope so it corresponds to the morphology of the patient. We will also develop a robust pose estimation method able to precisely place the bone images into an RGBD image of the patient. This process will be performed by learning, from a calibrated and synchronized pair of X-ray and RGBD images, the relationship between the shape of the envelope and the location of bones. An AR clinical application to show the model superimposed on the image of the patient and the relevant clinical parameters will be developed to show the potential applications of the system. Being a certified Living Lab, we include all the stakeholders to the research from the beginning. We count on the participation of clinical and industrial collaborators to ensure that the algorithms and methodologies that we develop are applicable in clinical routine and amenable to commercialization by medical equipment companies. We intend to transfer these promising technologies to industrial collaborators, to make them available to all doctors and allow faster and more accurate analysis of the musculoskeletal system for the benefit of the patients.
Program
Discovery Grants Program - Individual
Recipient
Location
Quebec, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
RGPIN-2021-04293
Name nom
Vazquez, Carlos
Department département
Génie logiciel et des technologies de l'information
Organization id
53
Institution établissement
École de technologie supérieure
Province en
Québec
Province fr
Québec
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2023
Competition year année de concours
2021
Award amount
29000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
518
Area of application group en
Manufacturing processes and products
Area of application group fr
Procédés et produits industriels
Area of application en
Medical equipment and apparatus
Area of application fr
Appareils et instruments médicaux
Research subject code
2603
Research subject group en
Robotics
Research subject group fr
Computer vision
Research subject en
Robotique
Research subject fr
Vision artificielle
Application title
Combined biplanar X-ray and RGBD images 3D reconstruction of the musculoskeletal system for augmented reality clinical follow-up.
Keyword
3D model reconstruction; Augmented reality; RGBD imaging; X-ray medidal imaging
Application summary
Imagine going to the hospital, taking a 1 min X-ray scan like the ones at the airport security and when you see the doctor, she/he has a tablet and sees your bones in the right positions in your body via a precise 3D augmented reality (AR) app. The doctor asks you to walk to identify the origin of that pain in your right knee and tells you immediately, while showing you your knee joint (the bones) superimposed on the image of your real knee, what the problem is. Or going to the ER with a fractured arm bone and seeing the precise depiction of the fracture inside your arm on the tablet, which the doctor is using to plan the best approach in putting the bone back together. This is the kind of system that we intend to make possible by building specialized 3D models of bones, developing algorithms to construct them from optical and X-ray images, and building an AR application to show the bones in the right place on a live video feed. Our research group at the LIO-ÉTS-CRCHUM has been working over the years building 3D models from low-dose biplanar X-ray images to facilitate the extraction of clinical parameters needed to diagnose conditions of the musculoskeletal system, e.g. scoliosis, or to analyze posture or walking disorders. We have developed expertise in the identification of bones, the rigid and non-rigid 2D/3D registration of 3D models and 2D images, pose estimation and AR applications development. We intend to expand our research efforts to build a multimodal acquisition system combining X-ray and image+depth (RGBD) imaging devices to construct a personalized 3D model of the patient that can be used later for real-time analysis of the musculoskeletal system. The proposed methodology is based on the use of deep learning approaches to identify the bones in the images and modify a combined statistical 3D model of the bones and the body envelope so it corresponds to the morphology of the patient. We will also develop a robust pose estimation method able to precisely place the bone images into an RGBD image of the patient. This process will be performed by learning, from a calibrated and synchronized pair of X-ray and RGBD images, the relationship between the shape of the envelope and the location of bones. An AR clinical application to show the model superimposed on the image of the patient and the relevant clinical parameters will be developed to show the potential applications of the system. Being a certified Living Lab, we include all the stakeholders to the research from the beginning. We count on the participation of clinical and industrial collaborators to ensure that the algorithms and methodologies that we develop are applicable in clinical routine and amenable to commercialization by medical equipment companies. We intend to transfer these promising technologies to industrial collaborators, to make them available to all doctors and allow faster and more accurate analysis of the musculoskeletal system for the benefit of the patients.