Grant: Discovery Grants Program - Individual
University of Toronto
$40,000.00
Neural Signal Representations for Physics-Based Machine Learning and Active 3D Imaging
EF 2023-24
Résumé
In recent years, machine learning techniques have become powerful tools for processing and understanding visual data. However, while these techniques offer new and exciting capabilities, they often fail to generalize outside of their training datasets. On the other hand, many conventional, physics-based modeling pipelines are informed by decades of research and demonstrate robust performance across their input space. The two long term objectives of my research program are: 1. studying and developing a new machine learning framework, known as neural signal representations, that marries the capabilities of neural networks with the robustness of physics-based models. 2. exploiting these capabilities for applications in active 3D imaging, with new methods to recover geometry in challenging conditions: at single photon signal levels and with multiply scattered light. Neural signal representations are an emerging method for representing and optimizing signals; they have recently become popular after demonstrating state-of-the-art performance for multiview 3D reconstruction and neural rendering. The idea is that, rather than storing signals as discrete samples, signals are stored directly in the weights of a neural network. Key advantages of this framework are that it can be flexibly incorporated into physics-based models, the network can learn priors over a space of signals, and large scale, high-dimensional signals can be represented with a far smaller memory footprint compared to conventional arrays. Still, there are many open questions about the behavior of neural representations, and there are serious practical limitations relating to efficiency and scalability. My work will (1) advance the theory and interpretability of neural signal representations, and (2) develop practical and scalable architectures that enable new capabilities in computer vision, physics-based modeling, and 3D reconstruction (Research Aim 1). Active imaging systems, such as lidar (light detection and ranging), have achieved widespread adoption for 3D imaging and scene reconstruction. They capture 3D geometry by emitting a pulse of light and measuring the precise time it takes for light to reflect back from an object. While these systems are used in consumer electronics (iPhone, iPad), self-driving cars, robots, remote sensing systems, and biomedical imaging devices, they have a number of failure modes. For example, current lidar systems fail when the reflected light is too weak or becomes scrambled, such as when imaging at long distances or in situations where the emitted light scatters multiple times before returning. My work will leverage machine learned priors and emerging, single-photon-sensitive detectors to create a new class of 3D imaging systems that (1) recover high-resolution 3D geometry from single photons, and which (2) can recover 3D shape using light that scatters multiple times around corners, behind occluders, or through fog (Research Aim 2).
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Ontario, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data
Tel que publié
Application id
RGPIN-2022-04829
Name nom
Lindell, David
Department département
Computer Science
Organization id
31
Institution établissement
University of Toronto
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2023
Competition year année de concours
2022
Award amount
40000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
802
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Information systems and technology
Area of application fr
Systèmes d'information et technologie de l'information
Research subject code
2707
Research subject group en
Information technology
Research subject group fr
Graphics
Research subject en
Technologie de l'information
Research subject fr
Infographie
Application title
Neural Signal Representations for Physics-Based Machine Learning and Active 3D Imaging
Keyword
3D imaging; active imaging; computational imaging; computer vision; lidar; machine learning; neural networks; neural rendering; non-line-of-sight imaging; single-photon imaging
Application summary
In recent years, machine learning techniques have become powerful tools for processing and understanding visual data. However, while these techniques offer new and exciting capabilities, they often fail to generalize outside of their training datasets. On the other hand, many conventional, physics-based modeling pipelines are informed by decades of research and demonstrate robust performance across their input space. The two long term objectives of my research program are: 1. studying and developing a new machine learning framework, known as neural signal representations, that marries the capabilities of neural networks with the robustness of physics-based models. 2. exploiting these capabilities for applications in active 3D imaging, with new methods to recover geometry in challenging conditions: at single photon signal levels and with multiply scattered light. Neural signal representations are an emerging method for representing and optimizing signals; they have recently become popular after demonstrating state-of-the-art performance for multiview 3D reconstruction and neural rendering. The idea is that, rather than storing signals as discrete samples, signals are stored directly in the weights of a neural network. Key advantages of this framework are that it can be flexibly incorporated into physics-based models, the network can learn priors over a space of signals, and large scale, high-dimensional signals can be represented with a far smaller memory footprint compared to conventional arrays. Still, there are many open questions about the behavior of neural representations, and there are serious practical limitations relating to efficiency and scalability. My work will (1) advance the theory and interpretability of neural signal representations, and (2) develop practical and scalable architectures that enable new capabilities in computer vision, physics-based modeling, and 3D reconstruction (Research Aim 1). Active imaging systems, such as lidar (light detection and ranging), have achieved widespread adoption for 3D imaging and scene reconstruction. They capture 3D geometry by emitting a pulse of light and measuring the precise time it takes for light to reflect back from an object. While these systems are used in consumer electronics (iPhone, iPad), self-driving cars, robots, remote sensing systems, and biomedical imaging devices, they have a number of failure modes. For example, current lidar systems fail when the reflected light is too weak or becomes scrambled, such as when imaging at long distances or in situations where the emitted light scatters multiple times before returning. My work will leverage machine learned priors and emerging, single-photon-sensitive detectors to create a new class of 3D imaging systems that (1) recover high-resolution 3D geometry from single photons, and which (2) can recover 3D shape using light that scatters multiple times around corners, behind occluders, or through fog (Research Aim 2).