Grant: Discovery Grants Program - Individual
University of Alberta
$41,000.00
Low-level Algorithms for Continual and Meta Reinforcement Learning
FY 2024-25
Summary
The reinforcement learning (RL) approach to artificial intelligence has had many impressive successes, including superhuman performance in Backgammon, Atari games, Go, Chess, Poker, StarCraft, and simulated race-car driving. The RL approach is special in that it treats the entirety of an autonomous intelligent agent interacting with its environment. Even simple RL agents sense their environment, take action, and have a goal (maximizing reward). More advanced agents integrate perception and planning into their decision-making.
Despite the successes of RL, its low-level learning algorithms are limited in continual learning settings. The most impressive successes of RL use artificial neural networks, but these deep reinforcement learning (DRL) methods have great difficulty continuing to learn as they continue to interact with their environment. The standard strategy in DRL is to maintain a buffer of past experience and selectively replay samples from it. We propose instead to refine the low-level learning algorithms so that they themselves are capable of continual learning and the complexity of replay buffers is not needed. In initial work, we have shown loss of plasticity in standard supervised learning domains (ImageNet and MNIST) and developed simple variants of the backpropagation algorithm that maintain plasticity indefinitely.
An agent that continues to learn will gain extensive experience with the process of learning; it could potentially learn to learn better, a phenomena called meta-learning. A meta-learning agent may learn to generalize better from state to state by forming better state representations. The standard approach to meta-learning today involves transfer between tasks, but we propose to instead use a single task that is too large to be solved exactly (the under-parameterized, or big-world case), or to emulate this with a single nonstationary task. In these cases, we changes to the low-level algorithms may be sufficient to obtain meta-learning. In particular, we are interested in meta-gradient algorithms for step-size adaptation such as Incremental Delta-Bar-Delta (IDBD). We expect to show that IDBD is qualitatively different from optimizers commonly used in modern DRL, such as Adam and RMSprop, in ways that make it a better starting place for developing low-level algorithms for representation learning.
Continual learning and meta-learning are closely related. Both directly contribute to the efficiency and robustness of ongoing learning processes. At the lowest level, we believe these two abilities are different aspects of the same algorithmic mechanisms.
To the extent that this research is successful, it will contribute to a better understanding of the domain-independent algorithms and principles of intelligence. In the near term, the developed technology will improve the ability of our machine learning systems to learn efficiently in applications that involve change and nonstationarity, which is arguably almost all of them.
Program
Discovery Grants Program - Individual
Recipient
Location
Alberta, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
RGPIN-2023-04946
Name nom
Sutton, Richard
Department département
Computing Science - Computing Science
Organization id
9
Institution établissement
University of Alberta
Province en
Alberta
Province fr
Alberta
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2023
Award amount
41000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
1200
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Advancement of knowledge
Area of application fr
Avancement des connaissances
Research subject code
2800
Research subject group en
Artificial intelligence
Research subject group fr
Intelligence artificielle
Research subject en
Artificial intelligence (computer vision, use2603)
Research subject fr
Intelligence artificielle (vision artificielle, utiliser 2603)
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Low-level Algorithms for Continual and Meta Reinforcement Learning
Keywords
artificial intelligence; reinforcement learning; machine learning; continual learning; temporal-difference learning; meta-learning; tracking; time-series prediction; control; adaptive signal processing
Application summary
The reinforcement learning (RL) approach to artificial intelligence has had many impressive successes, including superhuman performance in Backgammon, Atari games, Go, Chess, Poker, StarCraft, and simulated race-car driving. The RL approach is special in that it treats the entirety of an autonomous intelligent agent interacting with its environment. Even simple RL agents sense their environment, take action, and have a goal (maximizing reward). More advanced agents integrate perception and planning into their decision-making.
Despite the successes of RL, its low-level learning algorithms are limited in continual learning settings. The most impressive successes of RL use artificial neural networks, but these deep reinforcement learning (DRL) methods have great difficulty continuing to learn as they continue to interact with their environment. The standard strategy in DRL is to maintain a buffer of past experience and selectively replay samples from it. We propose instead to refine the low-level learning algorithms so that they themselves are capable of continual learning and the complexity of replay buffers is not needed. In initial work, we have shown loss of plasticity in standard supervised learning domains (ImageNet and MNIST) and developed simple variants of the backpropagation algorithm that maintain plasticity indefinitely.
An agent that continues to learn will gain extensive experience with the process of learning; it could potentially learn to learn better, a phenomena called meta-learning. A meta-learning agent may learn to generalize better from state to state by forming better state representations. The standard approach to meta-learning today involves transfer between tasks, but we propose to instead use a single task that is too large to be solved exactly (the under-parameterized, or big-world case), or to emulate this with a single nonstationary task. In these cases, we changes to the low-level algorithms may be sufficient to obtain meta-learning. In particular, we are interested in meta-gradient algorithms for step-size adaptation such as Incremental Delta-Bar-Delta (IDBD). We expect to show that IDBD is qualitatively different from optimizers commonly used in modern DRL, such as Adam and RMSprop, in ways that make it a better starting place for developing low-level algorithms for representation learning.
Continual learning and meta-learning are closely related. Both directly contribute to the efficiency and robustness of ongoing learning processes. At the lowest level, we believe these two abilities are different aspects of the same algorithmic mechanisms.
To the extent that this research is successful, it will contribute to a better understanding of the domain-independent algorithms and principles of intelligence. In the near term, the developed technology will improve the ability of our machine learning systems to learn efficiently in applications that involve change and nonstationarity, which is arguably almost all of them.