Canada Spends Logo

Explore our Budget 2026 analysis

Grant: Discovery Grants Program - Individual

University of Toronto

$24,000.00

Event-based Vision for Robotic Scene Understanding

FY 2024-25

Summary
Robotic and autonomous systems that are capable of operating in diverse and unstructured environments hold the promise to enable a big variety of new products and impact broad areas of society. To fulfill that promise robots need to provide a broad scope of capabilities for better environment understanding. While recent years have seen remarkable results in that space, robust energy-efficient high-rate scene understanding on a broad variety of platforms in diverse application settings remains a major challenge for robotic perception. Addressing this challenge requires significant fundamental research on novel algorithms and sensing modalities. For practical applications, high computational cost and low-rate visual sensing can make operation in highly dynamic settings infeasible. Effects such as motion-blur or an insufficient update rate can prohibit the desired levels of robustness. Simply opting for a more elaborate computation and sensing setup often comes with a higher financial cost and higher energy consumption. Both limit the scope of applications reducing the potential for low-cost deployment and long-term autonomous operation. The proposed work aims to address the challenge by investigating fundamental algorithms for using event-based cameras in robotic applications. Event-based cameras are a visual sensing modality in which the sensor provides instantaneous information about brightness changes in pixels rather than full image frames at a fixed rate. This allows for much higher processing rates while at the same time reducing required energy consumption. To overcome the limitations imposed by the absence of full-frame data, this research program will investigate both, data-fusion with classical RGB cameras and operation within interactive perception pipelines. Compared to several decades of research in computer vision, event-based perception is still a young research topic with a plethora of open problems. In this work we want to address (i) fundamentals of event-based representation learning for perception and interaction, (ii) novel algorithms for event-based scene understanding such as for segmentation, object detection, and object pose-estimation in highly dynamic settings, and (iii) integration of these algorithms in robot learning and decision-making pipelines. This work program will create new theoretical foundations, algorithms, and benchmarks for using event-based cameras in robotics. Our goal is to contribute to the foundations for expanding robotic perception systems to a wider variety of platforms and devices. Additionally, to the merit of basic academic research, we will ensure transfer of technology and ideas through exchange with sensor manufacturers and industrial application partners. We will leverage Canada's strong competence in developing robotic technology while providing the innovation and personnel needed for bringing this technology to the market.
Program
Discovery Grants Program - Individual
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data

As published

Application id
RGPIN-2021-03720
Name nom
Gilitschenski, Igor
Department département
Computer Science
Organization id
31
Institution établissement
University of Toronto
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2021
Award amount
24000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
1207
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Information, computer and communication technologies
Area of application fr
Technologie de l'information, des ordinateurs et des réseaux de communications
Research subject code
2603
Research subject group en
Robotics
Research subject group fr
Robotique
Research subject en
Computer vision
Research subject fr
Vision artificielle
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Event-based Vision for Robotic Scene Understanding
Keywords
event-based cameras; robotic perception; representation learning; scene understanding; autonomous systems; interactive perception
Application summary
Robotic and autonomous systems that are capable of operating in diverse and unstructured environments hold the promise to enable a big variety of new products and impact broad areas of society. To fulfill that promise robots need to provide a broad scope of capabilities for better environment understanding. While recent years have seen remarkable results in that space, robust energy-efficient high-rate scene understanding on a broad variety of platforms in diverse application settings remains a major challenge for robotic perception. Addressing this challenge requires significant fundamental research on novel algorithms and sensing modalities. For practical applications, high computational cost and low-rate visual sensing can make operation in highly dynamic settings infeasible. Effects such as motion-blur or an insufficient update rate can prohibit the desired levels of robustness. Simply opting for a more elaborate computation and sensing setup often comes with a higher financial cost and higher energy consumption. Both limit the scope of applications reducing the potential for low-cost deployment and long-term autonomous operation. The proposed work aims to address the challenge by investigating fundamental algorithms for using event-based cameras in robotic applications. Event-based cameras are a visual sensing modality in which the sensor provides instantaneous information about brightness changes in pixels rather than full image frames at a fixed rate. This allows for much higher processing rates while at the same time reducing required energy consumption. To overcome the limitations imposed by the absence of full-frame data, this research program will investigate both, data-fusion with classical RGB cameras and operation within interactive perception pipelines. Compared to several decades of research in computer vision, event-based perception is still a young research topic with a plethora of open problems. In this work we want to address (i) fundamentals of event-based representation learning for perception and interaction, (ii) novel algorithms for event-based scene understanding such as for segmentation, object detection, and object pose-estimation in highly dynamic settings, and (iii) integration of these algorithms in robot learning and decision-making pipelines. This work program will create new theoretical foundations, algorithms, and benchmarks for using event-based cameras in robotics. Our goal is to contribute to the foundations for expanding robotic perception systems to a wider variety of platforms and devices. Additionally, to the merit of basic academic research, we will ensure transfer of technology and ideas through exchange with sensor manufacturers and industrial application partners. We will leverage Canada's strong competence in developing robotic technology while providing the innovation and personnel needed for bringing this technology to the market.