Grant: Alliance Grants
University of Toronto
$288,208.00
Efficient Understanding and Inpainting in Learned 3D Reconstruction
FY 2024-25
Summary
Recent innovations in computer vision and graphics have led to the emergence of new transformative representations for 3D scenes and objects using learned parameters. These techniques, such as Gaussian Splatting, have many benefits and state-of-art performance in tasks such as high-quality 3D reconstruction using 2D images, for content creation and manipulation for graphics tasks such as gaming and immersive experiences in AR/VR and for medical imaging. For many interesting and important use-cases, a mere high quality representation of a captured scene is insufficient. Editing (the ability to alter and edit the representation of the 3D scene or object) and understanding (enabling tasks such as object detection, collision detection, that require semantic understanding) techniques are required to fully unlock their potential. However, many of these tasks require fast real-time performance and it is necessary to consider resource constraints for real-world deployment, such as the limited compute capabilities of mobile devices. This proposal aims to develop efficient representations that (1) enable high-speed 3D reconstruction, (2) are semantic-rich for a wider range of use cases such as segmentation, object detection, and semantic reasoning, and (3) enable scene editing for realistic modifications of captured scenes, for data-driven representations such as Gaussian Splatting. The outcomes of this project will have direct application in various domains such as photography and imaging, content creation, entertainment, augmented/virtual reality, education, and many more. Additionally, during the course of this research HQP will be trained in state-of-art technologies in visual computing and thus be valuable assets to companies, startups, and the technology sector in Canada.
Program
Alliance Grants
Recipient
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
590812-2023
Name nom
Vijaykumar, Nandita N
Department département
Computer and Mathematical Sciences (Scarborough)
Organization id
31
Institution établissement
University of Toronto
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2023
Award amount
288208
Program id
ALLRP
Program name en
Alliance Grants
Program name fr
Subventions Alliance
Group en
RESEARCH PARTNERSHIPS
Group fr
PARTENARIATS DE RECHERCHE
Committee code
96
Committee name en
RPP Internal Decision Cttee
Committee name fr
Cté de décision interne - PPR
Area of application code
1207
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Information, computer and communication technologies
Area of application fr
Technologie de l'information, des ordinateurs et des réseaux de communications
Research subject code
2603
Research subject group en
Robotics
Research subject group fr
Robotique
Research subject en
Computer vision
Research subject fr
Vision artificielle
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Efficient Understanding and Inpainting in Learned 3D Reconstruction
Keywords
editing; efficient processing; gaussian splatting; inpainting; learned reconstruction; mobile devices; performance optimization
Application summary
Recent innovations in computer vision and graphics have led to the emergence of new transformative representations for 3D scenes and objects using learned parameters. These techniques, such as Gaussian Splatting, have many benefits and state-of-art performance in tasks such as high-quality 3D reconstruction using 2D images, for content creation and manipulation for graphics tasks such as gaming and immersive experiences in AR/VR and for medical imaging. For many interesting and important use-cases, a mere high quality representation of a captured scene is insufficient. Editing (the ability to alter and edit the representation of the 3D scene or object) and understanding (enabling tasks such as object detection, collision detection, that require semantic understanding) techniques are required to fully unlock their potential. However, many of these tasks require fast real-time performance and it is necessary to consider resource constraints for real-world deployment, such as the limited compute capabilities of mobile devices. This proposal aims to develop efficient representations that (1) enable high-speed 3D reconstruction, (2) are semantic-rich for a wider range of use cases such as segmentation, object detection, and semantic reasoning, and (3) enable scene editing for realistic modifications of captured scenes, for data-driven representations such as Gaussian Splatting. The outcomes of this project will have direct application in various domains such as photography and imaging, content creation, entertainment, augmented/virtual reality, education, and many more. Additionally, during the course of this research HQP will be trained in state-of-art technologies in visual computing and thus be valuable assets to companies, startups, and the technology sector in Canada.