Grant: Research Tools and Instruments
University of Toronto
$149,627.00
Single-Photon Cameras for Extreme Computer Vision and Computational Astronomy
FY 2022-23
Summary
Cameras capture images by collecting photons: discrete packets of light that arrive to the sensor individually, stochastically, and asynchronously. While this physical process is abstracted away by the vast majority of computer vision algorithms, an emerging class of single-photon cameras provides means to precisely detect and catalog the arrival times of individual photons. Within the past five years, the growing availability of single-photon cameras has led to numerous never-seen-before imaging capabilities, from vision systems that can see in near-total darkness and around corners, to high-precision 3D cameras for automobiles and smartphones, to new biomedical imaging techniques for seeing below the skin. However, the single-photon cameras currently used by various laboratories around the world, including our own, have a critical limitation: they have only one pixel. Capturing an image with these cameras requires mechanical scanning across the field of view, a slow and tedious process which places profound limits on imaging applications. For example, despite their extremely fast temporal resolution, single- pixel, single-photon cameras cannot be used to image dynamic scenes, e.g., settings where the scene or camera moves or where the illumination itself is changing. Instead, these cameras are restricted to imaging static scenes in laboratory settings where scanning and image acquisition can take minutes to hours. Moreover, current single-photon cameras have restricted bandwidth, limiting the total number of photons that can be collected within a given period of time. Motivated by the recently announced commercial availability of 2D single-photon cameras, we propose to build a novel experimental testbed to push dynamic computer vision, 2D imaging, and 3D sensing to an entirely new level of speed, sensitivity, and accuracy. Specifically, this proposal requests 2D single-photon cameras of two distinct types: (1) a single-photon camera with 23 pixels that individually detect and stream out up to 7.8 million high-precision photon arrival times per second, and (2) a high-resolution 512x512 camera that outputs 100,000 binary frames of photon detections per second. The first type of camera is a 2D generalization of single-pixel, single-photon cameras, offering a 20x improvement in photon count rate, dramatically reduced scan times, and extreme imaging capabilities: low-light, high-speed videography; dynamic, single-photon 3D imaging; and passive localization and mapping in the dark. The second type of camera is best understood as a high-frame-rate, high-resolution, zero-noise binary video camera; while it does not output photon timing information, its single-photon sensitivity, high speed, and high spatial resolution enable new forms of real-time visual processing, and may lead to revolutionary new types of Earth-based astronomical telescopes in a collaborative research direction described in the proposal.
Program
Research Tools and Instruments
Recipient
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
RTI-2023-00153
Name nom
Kutulakos, Kiriakos
Department département
Computer Science (St. George Campus) - Computer Science (St. George Campus)
Organization id
31
Institution établissement
University of Toronto
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2022
Competition year année de concours
2023
Award amount
149627
Program id
RTI
Program name en
Research Tools and Instruments
Program name fr
Outils et instruments de recherche
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1607
Committee name en
Computer, Mathematical and Statistical Sciences (RTI)
Committee name fr
Sciences informatiques, mathématiques et statistiques (OIR)
Area of application code
1207
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Information, computer and communication technologies
Area of application fr
Technologie de l'information, des ordinateurs et des réseaux de communications
Research subject code
2600
Research subject group en
Robotics
Research subject group fr
Robotique
Research subject en
Robotics
Research subject fr
Robotique
Installment
inst:1
Partie
1
Nb partie
1
Application title
Single-Photon Cameras for Extreme Computer Vision and Computational Astronomy
Application summary
Cameras capture images by collecting photons: discrete packets of light that arrive to the sensor individually, stochastically, and asynchronously. While this physical process is abstracted away by the vast majority of computer vision algorithms, an emerging class of single-photon cameras provides means to precisely detect and catalog the arrival times of individual photons. Within the past five years, the growing availability of single-photon cameras has led to numerous never-seen-before imaging capabilities, from vision systems that can see in near-total darkness and around corners, to high-precision 3D cameras for automobiles and smartphones, to new biomedical imaging techniques for seeing below the skin. However, the single-photon cameras currently used by various laboratories around the world, including our own, have a critical limitation: they have only one pixel. Capturing an image with these cameras requires mechanical scanning across the field of view, a slow and tedious process which places profound limits on imaging applications. For example, despite their extremely fast temporal resolution, single- pixel, single-photon cameras cannot be used to image dynamic scenes, e.g., settings where the scene or camera moves or where the illumination itself is changing. Instead, these cameras are restricted to imaging static scenes in laboratory settings where scanning and image acquisition can take minutes to hours. Moreover, current single-photon cameras have restricted bandwidth, limiting the total number of photons that can be collected within a given period of time. Motivated by the recently announced commercial availability of 2D single-photon cameras, we propose to build a novel experimental testbed to push dynamic computer vision, 2D imaging, and 3D sensing to an entirely new level of speed, sensitivity, and accuracy. Specifically, this proposal requests 2D single-photon cameras of two distinct types: (1) a single-photon camera with 23 pixels that individually detect and stream out up to 7.8 million high-precision photon arrival times per second, and (2) a high-resolution 512x512 camera that outputs 100,000 binary frames of photon detections per second. The first type of camera is a 2D generalization of single-pixel, single-photon cameras, offering a 20x improvement in photon count rate, dramatically reduced scan times, and extreme imaging capabilities: low-light, high-speed videography; dynamic, single-photon 3D imaging; and passive localization and mapping in the dark. The second type of camera is best understood as a high-frame-rate, high-resolution, zero-noise binary video camera; while it does not output photon timing information, its single-photon sensitivity, high speed, and high spatial resolution enable new forms of real-time visual processing, and may lead to revolutionary new types of Earth-based astronomical telescopes in a collaborative research direction described in the proposal.