Grant: Discovery Grants Program - Individual
University of Toronto
$58,000.00
Transport-Aware Image Sensors
FY 2019-20
Summary
The main objective of our research program for the next five-year term will be to design, prototype and demonstrate a new class of computational image sensor integrated circuits whose key property is that they are transport-aware. Unlike conventional image sensors, which record all incident light, transport-aware imagers can be programmed to selectively detect only some of that light, depending on the actual 3D paths it followed through a visual scene. Such an imager acquires selected components of light transport (e.g., direct-only contributions, indirect-only contributions, specular-indirect contributions, etc.) by physically blocking all “undesirable” light paths so they cannot contribute to the image formed on the sensor.****** Transport-aware cameras promise to dominate many high-end industrial, scientific and commercial 2D and 3D imaging applications in robotics and computer vision such as: augmented and virtual reality, self-driving cars, 3D printers and scanners, video games, biomedical imaging, and materials analysis. Conventional cameras which are currently widely used for these applications have very limited programmability needed to apply transport-aware imaging techniques. Current full-functionality transport-aware cameras require a large mechanically deforming digital micromirror device (DMD) to implement programmable sensor masking, which introduces the following handicaps: (i) excessive form factor the size of a printer a key barrier to the largest market of portable consumer electronics; (ii) prohibitive distortion due to DMD-imposed large-lens curvature, (iii) low electro-mechanical mask update speed significantly limiting the range of applications, (iv) high power dissipation hindering mobile implementations, and (v) high cost in the range of thousands of dollars. We will develop a new class of computational image sensors that not only eliminate DMDs and their shortfalls, but, by electronic per-pixel masking, also offer previously unattainable versatility in coded-exposure imaging.****** The proposed image sensors combine spatial (intensity) and temporal (phase) exposure coding and will deliver previously unattainable performance levels to transport-aware cameras, a radically new class of cameras in their own right. Cameras based on such image sensors will offer an unrivaled view of the world around us in which refraction and scattering can be selectively attenuated or amplified and object surfaces can be reconstructed in 3D under challenging conditions well beyond of what is possible with the existing state of the art imaging technologies.
Program
Discovery Grants Program - Individual
Recipient
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
RGPIN-2017-06931
Name nom
Genov, Roman
Department département
Electrical & Computer Engineering
Organization id
31
Institution établissement
University of Toronto
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2019
Competition year année de concours
2017
Award amount
58000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1510
Committee name en
Electrical and Computer Engineering
Committee name fr
Génie électrique et informatique
Area of application code
513
Area of application group en
Manufacturing processes and products
Area of application group fr
Procédés et produits industriels
Area of application en
Electrical and electronic machinery and equipment (including computer hardware)
Area of application fr
Machinerie et équipement électriques et électroniques (y compris matériel informatique)
Research subject code
2506
Research subject group en
Electrical and electronic engineering
Research subject group fr
Génie électrique et électronique
Research subject en
Electron devices
Research subject fr
Dispositifs électroniques
Installment
inst:3 - 5
Partie
3
Nb partie
5
Application title
Transport-Aware Image Sensors
Keyword
NSERC
Application summary
The main objective of our research program for the next five-year term will be to design, prototype and demonstrate a new class of computational image sensor integrated circuits whose key property is that they are transport-aware. Unlike conventional image sensors, which record all incident light, transport-aware imagers can be programmed to selectively detect only some of that light, depending on the actual 3D paths it followed through a visual scene. Such an imager acquires selected components of light transport (e.g., direct-only contributions, indirect-only contributions, specular-indirect contributions, etc.) by physically blocking all “undesirable” light paths so they cannot contribute to the image formed on the sensor.****** Transport-aware cameras promise to dominate many high-end industrial, scientific and commercial 2D and 3D imaging applications in robotics and computer vision such as: augmented and virtual reality, self-driving cars, 3D printers and scanners, video games, biomedical imaging, and materials analysis. Conventional cameras which are currently widely used for these applications have very limited programmability needed to apply transport-aware imaging techniques. Current full-functionality transport-aware cameras require a large mechanically deforming digital micromirror device (DMD) to implement programmable sensor masking, which introduces the following handicaps: (i) excessive form factor the size of a printer a key barrier to the largest market of portable consumer electronics; (ii) prohibitive distortion due to DMD-imposed large-lens curvature, (iii) low electro-mechanical mask update speed significantly limiting the range of applications, (iv) high power dissipation hindering mobile implementations, and (v) high cost in the range of thousands of dollars. We will develop a new class of computational image sensors that not only eliminate DMDs and their shortfalls, but, by electronic per-pixel masking, also offer previously unattainable versatility in coded-exposure imaging.****** The proposed image sensors combine spatial (intensity) and temporal (phase) exposure coding and will deliver previously unattainable performance levels to transport-aware cameras, a radically new class of cameras in their own right. Cameras based on such image sensors will offer an unrivaled view of the world around us in which refraction and scattering can be selectively attenuated or amplified and object surfaces can be reconstructed in 3D under challenging conditions well beyond of what is possible with the existing state of the art imaging technologies.