Canada Spends Logo

Explorez notre analyse du budget 2026

Grant: Discovery Grants Program - Individual

University of Alberta

$52,000.00

Learning robot object manipulation from human demonstrations using image, force, and contact geometry

EF 2024-25

Résumé
Humans learn from others by hands-on instruction. Engineering approaches robot manipulation by using detailed physical modeling, while Computer Science focuses on Deep Learning from data or simulation. A robot motion goal and trajectory in Euclidean coordinates is specific to a scene, while Deep Learning from video is sensitive to scene and object appearance. Our thesis is that by understanding how humans perform object manipulation and analyzing the geometry and physics, we will be able to modularize complex robotics problems into tractable parts and approach robotics with a combination of modeling and learning. 1. Geometric invariants for describing task alignments in camera coordinates. Over the past 20 years, starting with my PDF in Hager's group at Yale U, we have developed object and task descriptions using projective multi-view geometry. This enables general learning of robot tasks, not dependent on the particular training environment, camera, or robot. A practical challenge has been the human annotation of geometry in task demonstration videos. Here we propose to instead analyze video of human task demonstrations and automatically identify which visual geometric alignments define the task. A putative set of geometric constraints is extracted by computer vision, then geometric combinations formed in a graph Neural Net (NN), finally a maxEntropy Inverse Reinforcement Learning (IRL) is used to learn which geometric constraints matter in the value function. In the long term we aim for a better understanding of how particular functional surfaces affords task learning. A human can interchangeably use a screwdriver, pencil or fork as a lever to pry something. We ask whether robots can learn similar general abilities? 2. Learning from Demonstration (LfD) of motor primitives. We currently develop both dynamics systems and VAE LfD to capture and represent low dimensional manifolds from human demonstrations, e.g. a vertical rotation for jars, water taps; and different types of reaches - open table top, confined fridge. These motor primitives provide tractable, low dimensional action spaces. Challenges for LfD we will research include stable, reproducible learning, and imposing geometric properties on the latent coordinates, such as, uniform scaling global and consistent differential geometric properties. We look to incorporate contact manipulation in LfD, either through learning impedance or direct forces and torques to apply at execution. Overall our approach is to reduce DoF and complexity by modularizing the robotics problem and selecting salient coordinate frames. A typical end-to-end DL or RL system would seek to learn directly between the millions of pixels and 6-7 arm and 4-15 hand DoF. Our first step is to reduce the video into 10-40 geometric alignments, then learn the hand-eye coordination mapping of these to the low DoF parameters of our motor primitives.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Alberta, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data

Tel que publié

Application id
RGPIN-2023-04887
Name nom
Jagersand, Martin
Department département
Computing Science
Organization id
9
Institution établissement
University of Alberta
Province en
Alberta
Province fr
Alberta
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2023
Award amount
52000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
802
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Information systems and technology
Area of application fr
Systèmes d'information et technologie de l'information
Research subject code
2600
Research subject group en
Robotics
Research subject group fr
Robotique
Research subject en
Robotics
Research subject fr
Robotique
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Learning robot object manipulation from human demonstrations using image, force, and contact geometry
Keywords
robotics; computer vision; visual servoing; geometric invariants; learning hand eye coordination; hri - human- robot interaction; lfd - learning motor skills from (human) demonstration
Application summary
Humans learn from others by hands-on instruction. Engineering approaches robot manipulation by using detailed physical modeling, while Computer Science focuses on Deep Learning from data or simulation. A robot motion goal and trajectory in Euclidean coordinates is specific to a scene, while Deep Learning from video is sensitive to scene and object appearance. Our thesis is that by understanding how humans perform object manipulation and analyzing the geometry and physics, we will be able to modularize complex robotics problems into tractable parts and approach robotics with a combination of modeling and learning. 1. Geometric invariants for describing task alignments in camera coordinates. Over the past 20 years, starting with my PDF in Hager's group at Yale U, we have developed object and task descriptions using projective multi-view geometry. This enables general learning of robot tasks, not dependent on the particular training environment, camera, or robot. A practical challenge has been the human annotation of geometry in task demonstration videos. Here we propose to instead analyze video of human task demonstrations and automatically identify which visual geometric alignments define the task. A putative set of geometric constraints is extracted by computer vision, then geometric combinations formed in a graph Neural Net (NN), finally a maxEntropy Inverse Reinforcement Learning (IRL) is used to learn which geometric constraints matter in the value function. In the long term we aim for a better understanding of how particular functional surfaces affords task learning. A human can interchangeably use a screwdriver, pencil or fork as a lever to pry something. We ask whether robots can learn similar general abilities? 2. Learning from Demonstration (LfD) of motor primitives. We currently develop both dynamics systems and VAE LfD to capture and represent low dimensional manifolds from human demonstrations, e.g. a vertical rotation for jars, water taps; and different types of reaches - open table top, confined fridge. These motor primitives provide tractable, low dimensional action spaces. Challenges for LfD we will research include stable, reproducible learning, and imposing geometric properties on the latent coordinates, such as, uniform scaling global and consistent differential geometric properties. We look to incorporate contact manipulation in LfD, either through learning impedance or direct forces and torques to apply at execution. Overall our approach is to reduce DoF and complexity by modularizing the robotics problem and selecting salient coordinate frames. A typical end-to-end DL or RL system would seek to learn directly between the millions of pixels and 6-7 arm and 4-15 hand DoF. Our first step is to reduce the video into 10-40 geometric alignments, then learn the hand-eye coordination mapping of these to the low DoF parameters of our motor primitives.