Grant: Discovery Grants Program - Individual
University of Toronto
$80,000.00
Computational Sensing for Extreme Computer Vision
FY 2023-24
Summary
The main goal of my research program over the next five years will be to study and develop novel computational cameras and sensing techniques that confer extreme vision capabilities to the systems employing them. My work aims to push dynamic 2D and 3D imaging to its limits along multiple axes---3D accuracy, low-light performance, speed, dynamic range, robustness, non-line-of-sight imaging ability, etc---by exploiting two emerging families of computational cameras: The first---time-resolved SPADs---have pixels sensitive enough to detect individual photons; they are used widely for depth imaging, biological imaging, and looking around corners, and their pixels output a never-ending stream of highly-precise timestamps for the photons they detect. This stream is usually discarded as too massive to process but our initial investigations suggest that the information contained in the timestamp streams makes these cameras extemely powerful: able to measure distances down to microns when combined with a laser, and able to passively record video at speeds almost a billion faster than conventional video cameras. The second---cameras with pixelwise-programmable exposure---was pioneered by an interdisciplinary team that I built at the University of Toronto, with a vision to provide a "swiss-army-knife sensor" for high-performance 2D and 3D imaging. Now, after more than seven years of pushing the state of the art in speed, pixel size and resolution with new CMOS sensor designs and new camera prototypes, we finally have programmable VGA-resolution sensors with one-of-a-kind capabilities for high-speed, robust 2D and 3D vision. We seek to unlock the potential of these cameras by (1) laying the foundations for 2D and 3D computer vision at the photon limit, (2) using the flexibility offered by our programmable cameras to solve hard visual inference tasks and (3) integrating sensing, optics and computation even further, to expand our imaging systems' capabilities.
Program
Discovery Grants Program - Individual
Recipient
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
RGPIN-2023-05609
Name nom
Kutulakos, Kiriakos
Department département
Computer Science
Organization id
31
Institution établissement
University of Toronto
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2023
Competition year année de concours
2023
Award amount
80000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
802
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Information systems and technology
Area of application fr
Systèmes d'information et technologie de l'information
Research subject code
2707
Research subject group en
Information technology
Research subject group fr
Graphics
Research subject en
Technologie de l'information
Research subject fr
Infographie
Application title
Computational Sensing for Extreme Computer Vision
Keyword
3D shape acquisition; adaptive imaging; computational cameras; computational sensors; computer vision in the dark; extreme computer vision; high-dynamic range imaging; high-speed imaging; light detection and ranging; single-photon imaging
Application summary
The main goal of my research program over the next five years will be to study and develop novel computational cameras and sensing techniques that confer extreme vision capabilities to the systems employing them. My work aims to push dynamic 2D and 3D imaging to its limits along multiple axes---3D accuracy, low-light performance, speed, dynamic range, robustness, non-line-of-sight imaging ability, etc---by exploiting two emerging families of computational cameras: The first---time-resolved SPADs---have pixels sensitive enough to detect individual photons; they are used widely for depth imaging, biological imaging, and looking around corners, and their pixels output a never-ending stream of highly-precise timestamps for the photons they detect. This stream is usually discarded as too massive to process but our initial investigations suggest that the information contained in the timestamp streams makes these cameras extemely powerful: able to measure distances down to microns when combined with a laser, and able to passively record video at speeds almost a billion faster than conventional video cameras. The second---cameras with pixelwise-programmable exposure---was pioneered by an interdisciplinary team that I built at the University of Toronto, with a vision to provide a "swiss-army-knife sensor" for high-performance 2D and 3D imaging. Now, after more than seven years of pushing the state of the art in speed, pixel size and resolution with new CMOS sensor designs and new camera prototypes, we finally have programmable VGA-resolution sensors with one-of-a-kind capabilities for high-speed, robust 2D and 3D vision. We seek to unlock the potential of these cameras by (1) laying the foundations for 2D and 3D computer vision at the photon limit, (2) using the flexibility offered by our programmable cameras to solve hard visual inference tasks and (3) integrating sensing, optics and computation even further, to expand our imaging systems' capabilities.