Grant: Discovery Grants Program - Individual
The University of British Columbia
$62,000.00
Visual 3D Reconstruction and Reasoning
FY 2024-25
Summary
Much of what we do as human beings rely on seeing the world with our eyes, constructing a mental model of the physical world, and then reasoning about it. This visual 3D reconstruction and reasoning capability is at the core of many things that we do, from things as simple as walking around, to things more complicated such as driving, or even more so as in designing new products or performing surgery. While essential, the degree of proficiency in these 3D reconstruction and reasoning tasks varies greatly among individuals and could be a limiting factor in what one can do, even with years of training. Thus, I aim to develop novel visual 3D reconstruction and reasoning methods, with the long-term vision of providing expert-level assistance to the public, augmenting human capabilities.
By doing so, I hope to allow people to do more, and to do better. For example, one would not need to be a CAD expert to design their own personal toy for their children, a car designer would not need to wait and consult an aerodynamics expert to test their design, and a surgeon would not need to rely solely on experience and would be able to manipulate the implant configurations during surgery with accurate feedback for improved outcome. From simple to complicated tasks, my research will enable and facilitate 3D reconstruction and reasoning for everyone.
A necessity towards this vision is to have 3D reconstruction and reasoning methods that can be deployed in the wild, without expert intervention-in other words, methods that are generalizable, robust, and accurate. Hence, I will focus on improving the current state-of-the-art methods, specifically via two short-term objectives:
Specialized machine learning frameworks for 3D computer vision. Recent research trends in 3D computer vision, including my own, have shown that utilizing specialized machine learning frameworks-for example, those that embed physics-is highly effective. This is because specialized architectures typically aim to limit the role of machine learning to small portions of the 3D visual processing pipeline, reducing the scope of things that need to be learned. This allows less data to be used for training when achieving generalizable models. I will thus investigate novel frameworks that are specialized for 3D computer vision.
Leveraging prior knowledge within large models. Complementary to the first objective, another way to enhance 3D reconstruction and reasoning is to use more data. This comes at a challenge as collecting data, especially 3D data, is costly and often requires specialized equipment. One very recent alternative, which has been led by our research group, is to repurpose large generative models-trained with billions of images-for 3D inference tasks. As in large language models such as ChatGPT, these models have within them emergent properties beyond the image generation task that they were trained for. I will investigate how to probe and utilize these hidden properties.
Program
Discovery Grants Program - Individual
Recipient
Location
British Columbia, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
RGPIN-2024-03942
Name nom
Yi, Kwang Moo
Department département
Computer Science - Computer Science
Organization id
2
Institution établissement
The University of British Columbia
Province en
British Columbia
Province fr
Colombie-Britannique
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2024
Award amount
62000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
801
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Computer software
Area of application fr
Logiciel
Research subject code
2603
Research subject group en
Robotics
Research subject group fr
Robotique
Research subject en
Computer vision
Research subject fr
Vision artificielle
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Visual 3D Reconstruction and Reasoning
Keywords
3d computer vision; visual 3d reconstruction; visual 3d reasoning; image correspondences; structure from motion; neural rendering; neural radiance fields; widebaseline stereo; visual localization; point cloud processing
Application summary
Much of what we do as human beings rely on seeing the world with our eyes, constructing a mental model of the physical world, and then reasoning about it. This visual 3D reconstruction and reasoning capability is at the core of many things that we do, from things as simple as walking around, to things more complicated such as driving, or even more so as in designing new products or performing surgery. While essential, the degree of proficiency in these 3D reconstruction and reasoning tasks varies greatly among individuals and could be a limiting factor in what one can do, even with years of training. Thus, I aim to develop novel visual 3D reconstruction and reasoning methods, with the long-term vision of providing expert-level assistance to the public, augmenting human capabilities.
By doing so, I hope to allow people to do more, and to do better. For example, one would not need to be a CAD expert to design their own personal toy for their children, a car designer would not need to wait and consult an aerodynamics expert to test their design, and a surgeon would not need to rely solely on experience and would be able to manipulate the implant configurations during surgery with accurate feedback for improved outcome. From simple to complicated tasks, my research will enable and facilitate 3D reconstruction and reasoning for everyone.
A necessity towards this vision is to have 3D reconstruction and reasoning methods that can be deployed in the wild, without expert intervention-in other words, methods that are generalizable, robust, and accurate. Hence, I will focus on improving the current state-of-the-art methods, specifically via two short-term objectives:
Specialized machine learning frameworks for 3D computer vision. Recent research trends in 3D computer vision, including my own, have shown that utilizing specialized machine learning frameworks-for example, those that embed physics-is highly effective. This is because specialized architectures typically aim to limit the role of machine learning to small portions of the 3D visual processing pipeline, reducing the scope of things that need to be learned. This allows less data to be used for training when achieving generalizable models. I will thus investigate novel frameworks that are specialized for 3D computer vision.
Leveraging prior knowledge within large models. Complementary to the first objective, another way to enhance 3D reconstruction and reasoning is to use more data. This comes at a challenge as collecting data, especially 3D data, is costly and often requires specialized equipment. One very recent alternative, which has been led by our research group, is to repurpose large generative models-trained with billions of images-for 3D inference tasks. As in large language models such as ChatGPT, these models have within them emergent properties beyond the image generation task that they were trained for. I will investigate how to probe and utilize these hidden properties.