Grant: Alliance Grants
University of Toronto
$189,000.00
Efficient and Versatile Neural Representations and Pipelines for Visual Computing
FY 2023-24
Summary
Implicit neural representations (INRs) and implicit radiance fields, e.g., neural radiance fields (NeRFs) have emerged as a promising technique to represent visual data such as 3D objects/scenes, videos, and images. Compared to traditional methods, a neural network is used to learn various signals in the scene such as color, density, surface properties, etc. The benefits of using NeRFs are in their ability to represent complex scenes with low memory overheads and the differentiability of the representation, which enables extrapolation of data based on sparse input data. Neural radiance fields enable generation of high quality photo-realistic novel views from these scene representations. Implicit representations have promising use cases in enabling high-quality 3D reconstruction using 2D images, for content creation and manipulation for graphics tasks such as gaming and immersive experiences in AR/VR and for medical imaging. The benefits and versatility of neural representations indicates an emerging change in the key representation for 3D visual data. This change necessitates rethinking existing systems and pipelines for processing, storing, and transmitting visual data. This proposal aims to tackle three important challenges: (1) Large storage requirements and no existing support for storing, transmitting, and retrieving 3D scenes stored as neural representations in databases; (2) Long training/rendering latencies and the need for significant compute power and memory resources; and (3) Algorithmic constraints such as inability to support scene editing and relighting for complex scenes with shiny surfaces. Research on novel systems and architectures neural radiance fields can have wide-ranging impact across various sectors in Canada including gaming and immersive experiences (AR/VR), medical imaging, 3D reconstruction, photo-realistic editing, and autonomous vehicles. This research project would also directly lead to the training of HQP in state-of-art transformative technologies who would then be valuable assets to companies, startups, and the technology sector in Canada.
Program
Alliance Grants
Recipient
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
586624-2023
Name nom
Vijaykumar, Nandita NV
Department département
Computer and Mathematical Sciences (Scarborough)
Organization id
31
Institution établissement
University of Toronto
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2023
Competition year année de concours
2023
Award amount
189000
Program id
ALLRP
Program name en
Alliance Grants
Program name fr
Subventions Alliance
Group en
RESEARCH PARTNERSHIPS
Group fr
PARTENARIATS DE RECHERCHE
Committee code
96
Committee name en
RPP Internal Decision Cttee
Committee name fr
Cté de décision interne - PPR
Area of application code
1207
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Information, computer and communication technologies
Area of application fr
Technologie de l'information, des ordinateurs et des réseaux de communications
Research subject code
2603
Research subject group en
Robotics
Research subject group fr
Computer vision
Research subject en
Robotique
Research subject fr
Vision artificielle
Application title
Efficient and Versatile Neural Representations and Pipelines for Visual Computing
Keyword
3D reconstruction; accelerators; computer graphics; computer vision; databases; data compression; efficient systems; neural radiance fields; novel view synthesis; visual computing
Application summary
Implicit neural representations (INRs) and implicit radiance fields, e.g., neural radiance fields (NeRFs) have emerged as a promising technique to represent visual data such as 3D objects/scenes, videos, and images. Compared to traditional methods, a neural network is used to learn various signals in the scene such as color, density, surface properties, etc. The benefits of using NeRFs are in their ability to represent complex scenes with low memory overheads and the differentiability of the representation, which enables extrapolation of data based on sparse input data. Neural radiance fields enable generation of high quality photo-realistic novel views from these scene representations. Implicit representations have promising use cases in enabling high-quality 3D reconstruction using 2D images, for content creation and manipulation for graphics tasks such as gaming and immersive experiences in AR/VR and for medical imaging. The benefits and versatility of neural representations indicates an emerging change in the key representation for 3D visual data. This change necessitates rethinking existing systems and pipelines for processing, storing, and transmitting visual data. This proposal aims to tackle three important challenges: (1) Large storage requirements and no existing support for storing, transmitting, and retrieving 3D scenes stored as neural representations in databases; (2) Long training/rendering latencies and the need for significant compute power and memory resources; and (3) Algorithmic constraints such as inability to support scene editing and relighting for complex scenes with shiny surfaces. Research on novel systems and architectures neural radiance fields can have wide-ranging impact across various sectors in Canada including gaming and immersive experiences (AR/VR), medical imaging, 3D reconstruction, photo-realistic editing, and autonomous vehicles. This research project would also directly lead to the training of HQP in state-of-art transformative technologies who would then be valuable assets to companies, startups, and the technology sector in Canada.