Grant: Discovery Grants Program - Individual
Université de Montréal
$48,000.00
Learning Actionable and Semantic Representations that Enable Autonomous Mobile Robots to Complete Complex Tasks
FY 2024-25
Summary
One of the core considerations in robotics is how we represent the world. Traditionally, approaches such as "Simultaneous Localization Mapping," or SLAM, have built purely geometric representations that are useful for basic tasks such as navigation, but are limited in their ability to reason about more complex semantic concepts. Large pre-trained neural network models have recently shown impressive reasoning capabilities. It is appealing to try to leverage this information inside a robot representation but it is not trivial to guarantee its correctness or reliability in a specific scenario. There are also aspects of a specific environment which are not well captured by these generalist models, such as where objects tend to be at certain times of day, or what particular objects are typically used for. The core objective of this research program is to combine "general" knowledge from pre-trained models that are trained at large scale, with "specific" knowledge that is obtained by a robot operating in an environment to build a representation that enables the robot to perform complex tasks reliably and with minimal supervision.
As the basis of our approach, we leverage the "3D scene graph", a hierarchical data structure that represents entities in the environment and their relationship, and is useful for robot planning. We propose a new type of scene graph representation that has several desirable properties: 1) it is useful for reasoning about the uncertainty of the concepts and relationships that it encodes, 2) it is persistent in that it models the evolution of entities over time, and 3) it considers the particular affordances of objects and the capabilities of the robot that is operating within the environment. Furthermore, we propose to use this representation to conditionally generate new synthetic environments that the robot can use to learn new skills and refine existing ones.
This research is needed for integration of mobile robots into complex environments such as homes. For robots to operate in such environments, they need to be able to perform complex tasks in a reliable manner. We desire to be able to specify such tasks in intuitive ways, such as through language or demonstration. And we expect that such an agent would gradually become more adept over time. All of these objectives require an agent to be performing reasoning at a sufficient level of abstraction.
Program
Discovery Grants Program - Individual
Recipient
Location
Quebec, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
RGPIN-2024-04956
Name nom
Paull, Liam
Department département
Département d'informatique et de recherche opérationnelle
Organization id
63
Institution établissement
Université de Montréal
Province en
Québec
Province fr
Québec
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2024
Award amount
48000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
1200
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Advancement of knowledge
Area of application fr
Avancement des connaissances
Research subject code
2600
Research subject group en
Robotics
Research subject group fr
Robotique
Research subject en
Robotics
Research subject fr
Robotique
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Learning Actionable and Semantic Representations that Enable Autonomous Mobile Robots to Complete Complex Tasks
Keywords
mobile robotics; robot perception; deep learning; robot learning; representation learning; simultaneous localization and mapping; robot localization; robot mapping; mobile manipulation
Application summary
One of the core considerations in robotics is how we represent the world. Traditionally, approaches such as "Simultaneous Localization Mapping," or SLAM, have built purely geometric representations that are useful for basic tasks such as navigation, but are limited in their ability to reason about more complex semantic concepts. Large pre-trained neural network models have recently shown impressive reasoning capabilities. It is appealing to try to leverage this information inside a robot representation but it is not trivial to guarantee its correctness or reliability in a specific scenario. There are also aspects of a specific environment which are not well captured by these generalist models, such as where objects tend to be at certain times of day, or what particular objects are typically used for. The core objective of this research program is to combine "general" knowledge from pre-trained models that are trained at large scale, with "specific" knowledge that is obtained by a robot operating in an environment to build a representation that enables the robot to perform complex tasks reliably and with minimal supervision.
As the basis of our approach, we leverage the "3D scene graph", a hierarchical data structure that represents entities in the environment and their relationship, and is useful for robot planning. We propose a new type of scene graph representation that has several desirable properties: 1) it is useful for reasoning about the uncertainty of the concepts and relationships that it encodes, 2) it is persistent in that it models the evolution of entities over time, and 3) it considers the particular affordances of objects and the capabilities of the robot that is operating within the environment. Furthermore, we propose to use this representation to conditionally generate new synthetic environments that the robot can use to learn new skills and refine existing ones.
This research is needed for integration of mobile robots into complex environments such as homes. For robots to operate in such environments, they need to be able to perform complex tasks in a reliable manner. We desire to be able to specify such tasks in intuitive ways, such as through language or demonstration. And we expect that such an agent would gradually become more adept over time. All of these objectives require an agent to be performing reasoning at a sufficient level of abstraction.