Grant: Discovery Grants Program - Individual
University of Alberta
$41,000.00
Integrated Architectures for State and Temporal Abstraction in Reinforcement Learning
FY 2023-24
Summary
In reinforcement learning (RL), an agent interacts with its environment such that the agent both receives an observation from the environment and takes an action based on that observation. The agent's goal is to maximize the sum of rewards, a numerical signal provided by the environment at every step, which is the fundamental unit of time in this formulation. Given the current state of the environment, the reward signal represents how good the agent's most recent action was. Because the agent needs to act at every step, actions are often modeled to be actions executed in a short period of time (e.g., muscle twitches). The observations the agent receives capture a slice of the true state of the world; they are often too high-dimensional and diverse to be enumerated (e.g., a transformation of the pixels of a camera image). Such simple problem formulation has led to several high-profile success stories such as super-human performance in games, autonomous control for navigating stratospheric balloons, and nuclear fusion. In truly difficult RL problems, algorithms need to tackle three main challenges: credit-assignment, exploration, and generalization. The agent's desired outcome is often due to a sequence of actions taken by the agent, not one; thus the agent needs to assign credit to past actions that were responsible for that outcome (e.g., which were the key actions that led the agent to win the game?). Naturally, the agent first needs to observe the desired outcome in order to be able to learn, and the agent needs to constantly deal with the trade-off between exploiting the best behaviour it learned so far and exploring alternative behaviours that might lead to better outcomes. Finally, in large environments, the agent might never observe the same setting twice, which means it should be able to generalize from past experience in order to succeed. The proposed program focuses on tackling credit assignment, exploration, and generalization simultaneously by having agents creating abstractions of both their observations and behaviours. We will do so by using neural networks for learning abstract representations of the agents' observations and options to model temporal abstraction. Options describe how actions should be taken over a period of time (e.g., picking up an object instead of a single muscle twitch). The proposed program is based on the idea that temporal abstractions and abstractions on observations (state abstractions) should be studied in tandem, with one serving as a scaffold to the other: Agents should use their learned state abstractions/representation of the world for credit assignment and to discover new ways of behaving (i.e., options). These discovered options will not only make credit assignment easier, they will also empower the agent to observe things it had not observed before, further improving its ability to explore and generating more diverse observations that will lead to even better representations of the world.
Program
Discovery Grants Program - Individual
Recipient
Location
Alberta, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
RGPIN-2023-03945
Name nom
Cholodovskis Machado, Marlos
Department département
Department of Computing Science
Organization id
9
Institution établissement
University of Alberta
Province en
Alberta
Province fr
Alberta
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2023
Competition year année de concours
2023
Award amount
41000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
802
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Information systems and technology
Area of application fr
Systèmes d'information et technologie de l'information
Research subject code
9999
Research subject group en
Not available
Research subject group fr
Not available
Research subject en
Non disponible
Research subject fr
Non disponible
Application title
Integrated Architectures for State and Temporal Abstraction in Reinforcement Learning
Keyword
Discovery; Eigenoptions; Option-keyboard; Options; Planning; Reinforcement learning; Representation learning; Successor representation; Temporal abstraction; Temporally-Extended Exploration
Application summary
In reinforcement learning (RL), an agent interacts with its environment such that the agent both receives an observation from the environment and takes an action based on that observation. The agent's goal is to maximize the sum of rewards, a numerical signal provided by the environment at every step, which is the fundamental unit of time in this formulation. Given the current state of the environment, the reward signal represents how good the agent's most recent action was. Because the agent needs to act at every step, actions are often modeled to be actions executed in a short period of time (e.g., muscle twitches). The observations the agent receives capture a slice of the true state of the world; they are often too high-dimensional and diverse to be enumerated (e.g., a transformation of the pixels of a camera image). Such simple problem formulation has led to several high-profile success stories such as super-human performance in games, autonomous control for navigating stratospheric balloons, and nuclear fusion. In truly difficult RL problems, algorithms need to tackle three main challenges: credit-assignment, exploration, and generalization. The agent's desired outcome is often due to a sequence of actions taken by the agent, not one; thus the agent needs to assign credit to past actions that were responsible for that outcome (e.g., which were the key actions that led the agent to win the game?). Naturally, the agent first needs to observe the desired outcome in order to be able to learn, and the agent needs to constantly deal with the trade-off between exploiting the best behaviour it learned so far and exploring alternative behaviours that might lead to better outcomes. Finally, in large environments, the agent might never observe the same setting twice, which means it should be able to generalize from past experience in order to succeed. The proposed program focuses on tackling credit assignment, exploration, and generalization simultaneously by having agents creating abstractions of both their observations and behaviours. We will do so by using neural networks for learning abstract representations of the agents' observations and options to model temporal abstraction. Options describe how actions should be taken over a period of time (e.g., picking up an object instead of a single muscle twitch). The proposed program is based on the idea that temporal abstractions and abstractions on observations (state abstractions) should be studied in tandem, with one serving as a scaffold to the other: Agents should use their learned state abstractions/representation of the world for credit assignment and to discover new ways of behaving (i.e., options). These discovered options will not only make credit assignment easier, they will also empower the agent to observe things it had not observed before, further improving its ability to explore and generating more diverse observations that will lead to even better representations of the world.