Canada Spends Logo

Explore our Budget 2026 analysis

Grant: Discovery Grants Program - Individual

Université Laval

$48,000.00

Understanding the world behind the image

FY 2021-22

Summary
Practical computer vision applications have, for a long time, found themselves confined to the realm of "machine vision": the image processing technologies for automatic inspection and analysis used in assembly lines. In the past decade, and in combination with the advent of very large image datasets, immense compute power and powerful deep learning algorithms, we have witnessed an emergence of computer vision in massive-scale consumer applications. For example, computers have learned to combine virtual objects into real video feeds for augmented reality, to realistically recreate the appearance of a person in a video, to safely steer autonomous cars by detecting the position and orientation of obstacles, and to robustly estimate the depth of a scene from a single image---effectively understanding the world in 3D. Despite this progress, computer vision systems suffer from one major limitation: they have trouble understanding the real world as a whole. Most techniques estimate a single component of the world (the lighting conditions, scene depth, geometry of objects, orientation of surfaces, etc.) at a time, without considering the fact that all of these components are at play when the image is formed. Indeed, images are created through a series of complex interactions between light and the geometry and reflectance of surfaces and objects in the scene. While the human eye can easily understand the combinations of these effects through millions of years of adaptation (e.g., we easily interpret that shadows are created by a bright light source being occluded), the same cannot be said of digital cameras and of algorithms operating on their images (e.g., shadows could be misinterpreted as objects). This research program will introduce novel methods for automatically understanding the 3D, lighting, and reflectance properties of scenes in a holistic manner. To do so, we will tackle the following four objectives. We will: 1) introduce new algorithms for estimating spatially-varying illumination in scenes; 2) develop a novel approach to efficiently estimate the spatially-varying reflectance properties of surfaces and scenes using a portable multi-light capture apparatus; 3) incorporate physical models of image formation for jointly estimating lighting, reflectance, and geometry from a single image to push the state of the art in holistic scene understanding; and 4) capture new databases of real-world lighting and surface reflectance to faithfully digitize and recreate the world at scale. In addition to the applications above, our proposed activities will also impact other fields such as computer graphics, video gaming, motion pictures, virtual and augmented reality, and artificial intelligence. This research program at Université Laval will contribute to maintain and even expand Canada's relevance in these fields via its key technical contributions, and by training highly qualified personnel in its development.
Program
Discovery Grants Program - Individual
Location
Quebec, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data

As published

Application id
RGPIN-2020-04799
Name nom
Lalonde, Jean-François
Department département
Génie électrique et génie informatique - Génie électrique et génie informatique
Organization id
51
Institution établissement
Université Laval
Province en
Québec
Province fr
Québec
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2021
Competition year année de concours
2020
Award amount
48000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
Discovery grants programs
Group fr
Programmes de subventions à la découverte
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
801
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Computer software
Area of application fr
Logiciel
Research subject code
2603
Research subject group en
Robotics
Research subject group fr
Robotique
Research subject en
Computer vision
Research subject fr
Vision artificielle
Installment
inst:2
Partie
1
Nb partie
5
Application title
Understanding the world behind the image
Keyword
NSERC
Application summary
Practical computer vision applications have, for a long time, found themselves confined to the realm of "machine vision": the image processing technologies for automatic inspection and analysis used in assembly lines. In the past decade, and in combination with the advent of very large image datasets, immense compute power and powerful deep learning algorithms, we have witnessed an emergence of computer vision in massive-scale consumer applications. For example, computers have learned to combine virtual objects into real video feeds for augmented reality, to realistically recreate the appearance of a person in a video, to safely steer autonomous cars by detecting the position and orientation of obstacles, and to robustly estimate the depth of a scene from a single image---effectively understanding the world in 3D. Despite this progress, computer vision systems suffer from one major limitation: they have trouble understanding the real world as a whole. Most techniques estimate a single component of the world (the lighting conditions, scene depth, geometry of objects, orientation of surfaces, etc.) at a time, without considering the fact that all of these components are at play when the image is formed. Indeed, images are created through a series of complex interactions between light and the geometry and reflectance of surfaces and objects in the scene. While the human eye can easily understand the combinations of these effects through millions of years of adaptation (e.g., we easily interpret that shadows are created by a bright light source being occluded), the same cannot be said of digital cameras and of algorithms operating on their images (e.g., shadows could be misinterpreted as objects). This research program will introduce novel methods for automatically understanding the 3D, lighting, and reflectance properties of scenes in a holistic manner. To do so, we will tackle the following four objectives. We will: 1) introduce new algorithms for estimating spatially-varying illumination in scenes; 2) develop a novel approach to efficiently estimate the spatially-varying reflectance properties of surfaces and scenes using a portable multi-light capture apparatus; 3) incorporate physical models of image formation for jointly estimating lighting, reflectance, and geometry from a single image to push the state of the art in holistic scene understanding; and 4) capture new databases of real-world lighting and surface reflectance to faithfully digitize and recreate the world at scale. In addition to the applications above, our proposed activities will also impact other fields such as computer graphics, video gaming, motion pictures, virtual and augmented reality, and artificial intelligence. This research program at Université Laval will contribute to maintain and even expand Canada's relevance in these fields via its key technical contributions, and by training highly qualified personnel in its development.