Canada Spends Logo

Explore our Budget 2026 analysis

Grant: Discovery Grants Program - Individual

University of Toronto

$40,000.00

Neural Signal Representations for Physics-Based Machine Learning and Active 3D Imaging

FY 2022-23

Summary
In recent years, machine learning techniques have become powerful tools for processing and understanding visual data. However, while these techniques offer new and exciting capabilities, they often fail to generalize outside of their training datasets. On the other hand, many conventional, physics-based modeling pipelines are informed by decades of research and demonstrate robust performance across their input space. The two long term objectives of my research program are: 1. studying and developing a new machine learning framework, known as neural signal representations, that marries the capabilities of neural networks with the robustness of physics-based models. 2. exploiting these capabilities for applications in active 3D imaging, with new methods to recover geometry in challenging conditions: at single photon signal levels and with multiply scattered light. Neural signal representations are an emerging method for representing and optimizing signals; they have recently become popular after demonstrating state-of-the-art performance for multiview 3D reconstruction and neural rendering. The idea is that, rather than storing signals as discrete samples, signals are stored directly in the weights of a neural network. Key advantages of this framework are that it can be flexibly incorporated into physics-based models, the network can learn priors over a space of signals, and large scale, high-dimensional signals can be represented with a far smaller memory footprint compared to conventional arrays. Still, there are many open questions about the behavior of neural representations, and there are serious practical limitations relating to efficiency and scalability. My work will (1) advance the theory and interpretability of neural signal representations, and (2) develop practical and scalable architectures that enable new capabilities in computer vision, physics-based modeling, and 3D reconstruction (Research Aim 1). Active imaging systems, such as lidar (light detection and ranging), have achieved widespread adoption for 3D imaging and scene reconstruction. They capture 3D geometry by emitting a pulse of light and measuring the precise time it takes for light to reflect back from an object. While these systems are used in consumer electronics (iPhone, iPad), self-driving cars, robots, remote sensing systems, and biomedical imaging devices, they have a number of failure modes. For example, current lidar systems fail when the reflected light is too weak or becomes scrambled, such as when imaging at long distances or in situations where the emitted light scatters multiple times before returning. My work will leverage machine learned priors and emerging, single-photon-sensitive detectors to create a new class of 3D imaging systems that (1) recover high-resolution 3D geometry from single photons, and which (2) can recover 3D shape using light that scatters multiple times around corners, behind occluders, or through fog (Research Aim 2).
Program
Discovery Grants Program - Individual
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data

As published

Application id
RGPIN-2022-04829
Name nom
Lindell, David
Department département
Computer Science (St. George Campus) - Computer Science (St. George Campus)
Organization id
31
Institution établissement
University of Toronto
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2022
Competition year année de concours
2022
Award amount
40000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
802
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Information systems and technology
Area of application fr
Systèmes d'information et technologie de l'information
Research subject code
2600
Research subject group en
Robotics
Research subject group fr
Robotique
Research subject en
Robotics
Research subject fr
Robotique
Installment
inst:1
Partie
1
Nb partie
5
Application title
Neural Signal Representations for Physics-Based Machine Learning and Active 3D Imaging
Application summary
In recent years, machine learning techniques have become powerful tools for processing and understanding visual data. However, while these techniques offer new and exciting capabilities, they often fail to generalize outside of their training datasets. On the other hand, many conventional, physics-based modeling pipelines are informed by decades of research and demonstrate robust performance across their input space. The two long term objectives of my research program are: 1. studying and developing a new machine learning framework, known as neural signal representations, that marries the capabilities of neural networks with the robustness of physics-based models. 2. exploiting these capabilities for applications in active 3D imaging, with new methods to recover geometry in challenging conditions: at single photon signal levels and with multiply scattered light. Neural signal representations are an emerging method for representing and optimizing signals; they have recently become popular after demonstrating state-of-the-art performance for multiview 3D reconstruction and neural rendering. The idea is that, rather than storing signals as discrete samples, signals are stored directly in the weights of a neural network. Key advantages of this framework are that it can be flexibly incorporated into physics-based models, the network can learn priors over a space of signals, and large scale, high-dimensional signals can be represented with a far smaller memory footprint compared to conventional arrays. Still, there are many open questions about the behavior of neural representations, and there are serious practical limitations relating to efficiency and scalability. My work will (1) advance the theory and interpretability of neural signal representations, and (2) develop practical and scalable architectures that enable new capabilities in computer vision, physics-based modeling, and 3D reconstruction (Research Aim 1). Active imaging systems, such as lidar (light detection and ranging), have achieved widespread adoption for 3D imaging and scene reconstruction. They capture 3D geometry by emitting a pulse of light and measuring the precise time it takes for light to reflect back from an object. While these systems are used in consumer electronics (iPhone, iPad), self-driving cars, robots, remote sensing systems, and biomedical imaging devices, they have a number of failure modes. For example, current lidar systems fail when the reflected light is too weak or becomes scrambled, such as when imaging at long distances or in situations where the emitted light scatters multiple times before returning. My work will leverage machine learned priors and emerging, single-photon-sensitive detectors to create a new class of 3D imaging systems that (1) recover high-resolution 3D geometry from single photons, and which (2) can recover 3D shape using light that scatters multiple times around corners, behind occluders, or through fog (Research Aim 2).