Grant: Discovery Grants Program - Individual
University of Toronto
$34,000.00
Compositional Causal Model-based Reinforcement Learning
FY 2022-23
Summary
One of the most important, unsolved problems of artificial intelligence is to build agents with human-like creativity, curiosity, self-assessment, and commonsense reasoning. Recently, model--free reinforcement learning (MFRL) has shown impressive performance in video game playing and locomotion controls using deep neural networks. Despite their success, MFRL methods are fundamentally limited by their trial--and--error nature, which requires millions of training examples to learn a reliable policy. On the other hand, a model--based reinforcement learning (MBRL) agent is capable of deliberate reasoning to achieve its goal. Unlike model--free agents, the MBRL agent iteratively learns a model of the world and plan its action according to its world model. MBRL has a great appeal because the learned model allows the agent to predict its future and reason about the consequences of its own actions. One of the ultimate goals of reinforcement learning research is to have agents acting in multiple environments and generalize previous learning experience to new situations. The ability to transfer knowledge across tasks is considered a critical aspect of any intelligent agent. The main objectives of the proposed research are to introduce a general model-based reinforcement learning algorithm that brings together three key ideas--compositionality, causality, and intrinsic curiosity--have been separately influential in machine learning over the past several decades. The objectives in this 5-year project are as follows: 1. Establish baselines for comparisons: Train and evaluate state-of-the-art model-based reinforcement learning agents in the latest locomotion control physics simulators. 2. Derive a compositional forward dynamics model, where the internal representations are object-based. 3. Explore, evaluate different types of causal inference methods in the proposed compositional model, including linear independent component analysis, mutual information-based independence tests, variational inference. 4. Develop planning-based algorithms to overcome non-stationary intrinsic rewards in exploration. 5. Answer the hypothesis that causal representations lead to simplified learning on new down-stream tasks, to help end-users in interpreting data, and to generalize to novel test examples. I anticipate that this project will benefit both deep learning and reinforcement learning community in several ways, ranging from the establishment of a new approach to actively infer causal factors, to elucidating new knowledge of exploration algorithms, to providing benchmark and open-source implementations of state-of-the-art MFRL and MBRL agents to maximally facilitate future research in the field of machine learning.
Program
Discovery Grants Program - Individual
Recipient
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
RGPIN-2020-06904
Name nom
Ba, Jimmy
Department département
Computer Science (St. George Campus) - Computer Science (St. George Campus)
Organization id
31
Institution établissement
University of Toronto
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2022
Competition year année de concours
2020
Award amount
34000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
1200
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Advancement of knowledge
Area of application fr
Avancement des connaissances
Research subject code
2800
Research subject group en
Artificial intelligence
Research subject group fr
Intelligence artificielle
Research subject en
Artificial intelligence (computer vision, use2603)
Research subject fr
Intelligence artificielle (vision artificielle, utiliser 2603)
Installment
inst:3
Partie
1
Nb partie
5
Application title
Compositional Causal Model-based Reinforcement Learning
Application summary
One of the most important, unsolved problems of artificial intelligence is to build agents with human-like creativity, curiosity, self-assessment, and commonsense reasoning. Recently, model--free reinforcement learning (MFRL) has shown impressive performance in video game playing and locomotion controls using deep neural networks. Despite their success, MFRL methods are fundamentally limited by their trial--and--error nature, which requires millions of training examples to learn a reliable policy. On the other hand, a model--based reinforcement learning (MBRL) agent is capable of deliberate reasoning to achieve its goal. Unlike model--free agents, the MBRL agent iteratively learns a model of the world and plan its action according to its world model. MBRL has a great appeal because the learned model allows the agent to predict its future and reason about the consequences of its own actions. One of the ultimate goals of reinforcement learning research is to have agents acting in multiple environments and generalize previous learning experience to new situations. The ability to transfer knowledge across tasks is considered a critical aspect of any intelligent agent. The main objectives of the proposed research are to introduce a general model-based reinforcement learning algorithm that brings together three key ideas--compositionality, causality, and intrinsic curiosity--have been separately influential in machine learning over the past several decades. The objectives in this 5-year project are as follows: 1. Establish baselines for comparisons: Train and evaluate state-of-the-art model-based reinforcement learning agents in the latest locomotion control physics simulators. 2. Derive a compositional forward dynamics model, where the internal representations are object-based. 3. Explore, evaluate different types of causal inference methods in the proposed compositional model, including linear independent component analysis, mutual information-based independence tests, variational inference. 4. Develop planning-based algorithms to overcome non-stationary intrinsic rewards in exploration. 5. Answer the hypothesis that causal representations lead to simplified learning on new down-stream tasks, to help end-users in interpreting data, and to generalize to novel test examples. I anticipate that this project will benefit both deep learning and reinforcement learning community in several ways, ranging from the establishment of a new approach to actively infer causal factors, to elucidating new knowledge of exploration algorithms, to providing benchmark and open-source implementations of state-of-the-art MFRL and MBRL agents to maximally facilitate future research in the field of machine learning.