Canada Spends Logo

Explorez notre analyse du budget 2026

Grant: Discovery Grants Program - Individual

Simon Fraser University

$46,000.00

Neural fields for 3D visual perception

EF 2023-24

Résumé
Animals are phenomenal at spatial perception - employing their various senses to understand their surroundings. Current day technology pales in comparison. We do not yet trust machines to discern a 3D scene better than humans. Fully autonomous driving, for instance, is still a pipe dream. What has been holding us back? In the era of artificial intelligence, algorithms are learnt from data. Yet, when compared to images or video, data in 3D computer vision has been relatively scarce and of low-quality. They do not perfectly reproduce reality in a way that is indistinguishable to the human eye. They are also relatively small and their acquisition requires custom hardware and well trained operators. Furthermore, there are many, often incompatible, representations of 3D information. My research program aims to build technology to capture 3D data more effectively, and computational models to reason about the acquired data. I plan to achieve this by casting spatial perception on "fields" - functions that can be queried in space. When fields are stored within the weights of an artificial neural network we call them "neural fields" - an area that I helped pioneer and that has been growing exponentially. With neural fields, we can now capture reality photorealistically, and from data captured from everyday digital cameras. There are many technical challenges that still prevent neural fields from reaching their true potential. Capturing a scene accurately requires hundreds of images. In contrast, our brains are capable of leveraging similar past observations and often perceive space from as little as a single image. When we perform an action, we also perceive the environment with varying degrees of uncertainty and devise ways to reduce it. And when in familiar environments, we can tap into our knowledge together with our senses before making a decision. My research program works towards mimicking these capabilities in machines. The research will impact science by providing better techniques for data acquisition (faster, cheaper, more reliable), which will open the doors to training of large scale 3D artificial intelligence. My team will develop mathematical models to represent past experience; whether a quantity that has been stored can be trusted; and how to analyze neural fields to extract useful information, whether it is to query the content of the scene or guide a robot towards fulfilling an objective. We will impact the industry by releasing our software, enabling others to build upon our work. As we further develop techniques that better understand space, risky tasks could be performed by machines rather than people. The accurate data we acquire will benefit the Canadian VFX industry, which can then focus on storytelling rather than realism. And as this research is based on cutting-edge research in machine learning and visual computing, it will also provide the Canadian market with the needed highly-qualified personnel.
Programme
Discovery Grants Program - Individual
Emplacement
British Columbia, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data

Tel que publié

Application id
RGPIN-2023-05617
Name nom
Tagliasacchi, Andrea
Department département
Computer Science (St. George Campus) - Computer Science (St. George Campus)
Organization id
5
Institution établissement
Simon Fraser University
Province en
British Columbia
Province fr
Colombie-Britannique
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2023
Competition year année de concours
2023
Award amount
46000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
1207
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Information, computer and communication technologies
Area of application fr
Technologie de l'information, des ordinateurs et des réseaux de communications
Research subject code
2707
Research subject group en
Information technology
Research subject group fr
Graphics
Research subject en
Technologie de l'information
Research subject fr
Infographie
Application title
Neural fields for 3D visual perception
Keyword
3D deep learning; coordinate neural networks; geometric deep learning; geometry processing; neural fields; neural fields processing toolbox; priors for spatial perception; semantic understanding; spatial perception; uncertainty modeling in neural fields
Application summary
Animals are phenomenal at spatial perception - employing their various senses to understand their surroundings. Current day technology pales in comparison. We do not yet trust machines to discern a 3D scene better than humans. Fully autonomous driving, for instance, is still a pipe dream. What has been holding us back? In the era of artificial intelligence, algorithms are learnt from data. Yet, when compared to images or video, data in 3D computer vision has been relatively scarce and of low-quality. They do not perfectly reproduce reality in a way that is indistinguishable to the human eye. They are also relatively small and their acquisition requires custom hardware and well trained operators. Furthermore, there are many, often incompatible, representations of 3D information. My research program aims to build technology to capture 3D data more effectively, and computational models to reason about the acquired data. I plan to achieve this by casting spatial perception on "fields" - functions that can be queried in space. When fields are stored within the weights of an artificial neural network we call them "neural fields" - an area that I helped pioneer and that has been growing exponentially. With neural fields, we can now capture reality photorealistically, and from data captured from everyday digital cameras. There are many technical challenges that still prevent neural fields from reaching their true potential. Capturing a scene accurately requires hundreds of images. In contrast, our brains are capable of leveraging similar past observations and often perceive space from as little as a single image. When we perform an action, we also perceive the environment with varying degrees of uncertainty and devise ways to reduce it. And when in familiar environments, we can tap into our knowledge together with our senses before making a decision. My research program works towards mimicking these capabilities in machines. The research will impact science by providing better techniques for data acquisition (faster, cheaper, more reliable), which will open the doors to training of large scale 3D artificial intelligence. My team will develop mathematical models to represent past experience; whether a quantity that has been stored can be trusted; and how to analyze neural fields to extract useful information, whether it is to query the content of the scene or guide a robot towards fulfilling an objective. We will impact the industry by releasing our software, enabling others to build upon our work. As we further develop techniques that better understand space, risky tasks could be performed by machines rather than people. The accurate data we acquire will benefit the Canadian VFX industry, which can then focus on storytelling rather than realism. And as this research is based on cutting-edge research in machine learning and visual computing, it will also provide the Canadian market with the needed highly-qualified personnel.