Grant: Alliance Grants
The University of British Columbia
$67,786.00
Extreme Distillation of Monocular Geometry Estimators
EF 2024-25
Résumé
3D reconstruction has seen a dramatic breakthrough in the last few years, with the introduction of inverse rendering methods that tie graphics and vision together. More recently, large AI models trained on massive datasets have enabled a breakthrough in 3D reconstruction---the ability to accurately estimate depth from a single image. These monocular depth estimators represent a significant advance in computer vision, allowing 3D inference from minimal input. Just in the last couple of months, we have seen a series of new models that can predict accurate 3D geometry from a single image, up to a level where they argue metric accuracy. Particularly, these models provide estimates through a single forward pass, and are thus efficient to run, compared to the inverse rendering methods that require inference-time optimization.
In this project, we thus focus on distilling monocular depth estimators into a compact and efficient model that can be used for downstream tasks, such as novel-view rendering and semantic mapping. Importantly, our focus here will be on efficient inference speed, such that the model can be used in real-time settings, such as in AR/VR applications. To do so, we will develop a model that processes a random subsample of the input pixels, which can then be evaluated extremely efficiently. We will then further develop a method to aggregate the results to recover high-resolution estimates, and further imbue the model with semantic understanding.
Programme
Alliance Grants
Bénéficiaire
Emplacement
British Columbia, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data
Tel que publié
Application id
602074-2024
Name nom
Yi, Kwang Moo KM
Department département
Computer Science
Organization id
2
Institution établissement
The University of British Columbia
Province en
British Columbia
Province fr
Colombie-Britannique
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2024
Award amount
67786
Program id
ALLRP
Program name en
Alliance Grants
Program name fr
Subventions Alliance
Group en
RESEARCH PARTNERSHIPS
Group fr
PARTENARIATS DE RECHERCHE
Committee code
96
Committee name en
RPP Internal Decision Cttee
Committee name fr
Cté de décision interne - PPR
Area of application code
802
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Information systems and technology
Area of application fr
Systèmes d'information et technologie de l'information
Research subject code
2603
Research subject group en
Robotics
Research subject group fr
Robotique
Research subject en
Computer vision
Research subject fr
Vision artificielle
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Extreme Distillation of Monocular Geometry Estimators
Keywords
2d to 3d; 3d computer vision; 3d reconstruction; ensemble; foundational models; gaussian splatting; knowledge distillation; monocular depth estimation; semantic segmentation; sub-sampling
Application summary
3D reconstruction has seen a dramatic breakthrough in the last few years, with the introduction of inverse rendering methods that tie graphics and vision together. More recently, large AI models trained on massive datasets have enabled a breakthrough in 3D reconstruction---the ability to accurately estimate depth from a single image. These monocular depth estimators represent a significant advance in computer vision, allowing 3D inference from minimal input. Just in the last couple of months, we have seen a series of new models that can predict accurate 3D geometry from a single image, up to a level where they argue metric accuracy. Particularly, these models provide estimates through a single forward pass, and are thus efficient to run, compared to the inverse rendering methods that require inference-time optimization.
In this project, we thus focus on distilling monocular depth estimators into a compact and efficient model that can be used for downstream tasks, such as novel-view rendering and semantic mapping. Importantly, our focus here will be on efficient inference speed, such that the model can be used in real-time settings, such as in AR/VR applications. To do so, we will develop a model that processes a random subsample of the input pixels, which can then be evaluated extremely efficiently. We will then further develop a method to aggregate the results to recover high-resolution estimates, and further imbue the model with semantic understanding.