Grant: Discovery Grants Program - Individual
University of Toronto
$46,000.00
New Directions in Robotic Environmental Monitoring via Machine Learning
FY 2024-25
Summary
What is preventing robotic environmental monitoring of Canadian landscapes from becoming automated? The answer is twofold. First, there are challenges in hardware robustness, such as operation in adverse environmental conditions. Second, and most crucial, robots are oblivious to the type of data that environmental scientists really need. They do not know where to direct their attention. My overarching goal is to address the latter set of challenges with little supervision from and interaction with humans.
This research will develop perception, inference, and control algorithms to address challenging robotics problems in informed visual exploration. The main application domain is the automation of environmental monitoring by robots operating outdoors, building on my past work in this area. The main users of the technology are environmental scientists. The core algorithms developed will be directly applicable to a vast array of human-robot interaction scenarios, and will span three key directions:
(D1) Data-Efficient Learning of Robot Visual Attention with User Specifications: we will develop methods that efficiently learn what visual content in an image matters to the user. Predictive models of visual attention will be learned from a small number of labeled and a large number of unlabeled images. They will enable robots to perform informed visual exploration tailored to the user's preferences. We will also develop connections with reward learning and inverse reinforcement learning over images. This is a key enabling factor for human-robot interaction, which despite years of progress in low-dimensional settings, still lacks robust practical methods in high dimensions.
(D2) Online Visual Exploration Strategies for Robots via Uncertainty in Visual Attention: we will estimate the model (epistemic) uncertainty of visual attention models, going beyond mere learning of a single probabilistic model. This will enable online, information gathering behaviors from the robot. It will also enable active selection of only a few images that the user needs to label in (D1). Most importantly, however, it will pave the way for the development of uncertainty-aware visual exploration strategies in 3D.
(D3) Collaborative Human Multi-Robot Visual Exploration for Environmental Monitoring: we will design methods to enable a team of robots and a human to visually explore an unknown environment in tandem. This will allow robots of different capabilities to explore collaboratively, without constantly having to see where the human is, by making informed short-term and long-term predictions about his/her location, intent, and preferences. After exploring on their own, robots will need to relocate their human collaborator to redirect his/her attention towards any interesting observations.
This research is unique in jointly addressing these challenges with a human in the loop. The techniques proposed herein will be validated on ground and aerial robots operating outdoors.
Program
Discovery Grants Program - Individual
Recipient
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
RGPIN-2019-06919
Name nom
Shkurti, Florian
Department département
Mathematical and Computational Sciences
Organization id
31
Institution établissement
University of Toronto
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2019
Award amount
46000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
1002
Area of application group en
Northern development
Area of application group fr
Développement du nord
Area of application en
Environment
Area of application fr
Environnement
Research subject code
2603
Research subject group en
Robotics
Research subject group fr
Robotique
Research subject en
Computer vision
Research subject fr
Vision artificielle
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
New Directions in Robotic Environmental Monitoring via Machine Learning
Keywords
robotics; computer vision; visual attention; reward learning; visual exploration and search; 3d autonomous navigation; visual tracking under uncertainty; uncertainty in neural networks; human-robot collaboration; environmental monitoring
Application summary
What is preventing robotic environmental monitoring of Canadian landscapes from becoming automated? The answer is twofold. First, there are challenges in hardware robustness, such as operation in adverse environmental conditions. Second, and most crucial, robots are oblivious to the type of data that environmental scientists really need. They do not know where to direct their attention. My overarching goal is to address the latter set of challenges with little supervision from and interaction with humans.
This research will develop perception, inference, and control algorithms to address challenging robotics problems in informed visual exploration. The main application domain is the automation of environmental monitoring by robots operating outdoors, building on my past work in this area. The main users of the technology are environmental scientists. The core algorithms developed will be directly applicable to a vast array of human-robot interaction scenarios, and will span three key directions:
(D1) Data-Efficient Learning of Robot Visual Attention with User Specifications: we will develop methods that efficiently learn what visual content in an image matters to the user. Predictive models of visual attention will be learned from a small number of labeled and a large number of unlabeled images. They will enable robots to perform informed visual exploration tailored to the user's preferences. We will also develop connections with reward learning and inverse reinforcement learning over images. This is a key enabling factor for human-robot interaction, which despite years of progress in low-dimensional settings, still lacks robust practical methods in high dimensions.
(D2) Online Visual Exploration Strategies for Robots via Uncertainty in Visual Attention: we will estimate the model (epistemic) uncertainty of visual attention models, going beyond mere learning of a single probabilistic model. This will enable online, information gathering behaviors from the robot. It will also enable active selection of only a few images that the user needs to label in (D1). Most importantly, however, it will pave the way for the development of uncertainty-aware visual exploration strategies in 3D.
(D3) Collaborative Human Multi-Robot Visual Exploration for Environmental Monitoring: we will design methods to enable a team of robots and a human to visually explore an unknown environment in tandem. This will allow robots of different capabilities to explore collaboratively, without constantly having to see where the human is, by making informed short-term and long-term predictions about his/her location, intent, and preferences. After exploring on their own, robots will need to relocate their human collaborator to redirect his/her attention towards any interesting observations.
This research is unique in jointly addressing these challenges with a human in the loop. The techniques proposed herein will be validated on ground and aerial robots operating outdoors.