Canada Spends Logo

Explorez notre analyse du budget 2026

Grant: Discovery Launch Supplement

Université de Montréal

$12,500.00

Building robust representations for deep reinforcement learning

EF 2024-25

Résumé
Reinforcement learning (RL) agents aim to learn behaviour policies, which are mappings from states to a distribution over a set of possible actions. For this purpose, most methods rely on learning value functions, which map states and actions to real values. Algorithms can use these learned values to obtain policies either directly (by choosing the action with the highest value from each state) or indirectly (by using them as a "critic" in actor-critic methods). The foundations of RL are often presented in terms of tabular systems, which are ones where state-action values can be stored in a table. Most problems of interest, however, have state-action spaces which are too large to be stored as such. Thus, researchers often operate on state features which can be more compactly represented, and have the added benefit of generalizability. Specifically, effective state features will enable unseen states to benefit from learning updates performed on other states which are similar, thereby increasing the coverage of the agent's learned values without much loss. Theoretical works assume state features are given, while most practical methods learn the features along with the value function. Usually, this is done with neural networks, and the combination of RL with neural networks is referred to as deep reinforcement learning (DRL). The state features, also called state representations, are learned implicitly since the learning objective of most RL agents is usually limited to value estimation. There are works which do explicitly learn state representations by learning representations that respect some notion of similarity between states, such as similarity of raw inputs, temporal occurrence, and long-term behaviors, and which are learned via differentiable losses. These representation losses are learned in conjunction with the standard RL loss on a pre-specified layer of the neural network. The resulting performance depends largely on network design and hyper-parameter optimization. The effectiveness of these representations is usually measured only through agent performance on standard benchmarks, which fails to provide proper insights into the true underlying properties of the learned representations. This proposal takes an experimental approach to understand learned representations by: investigating quantifiable measures of the quality of a learned representation; studying their sensitivity to hyperparameters, network designs, and exploration strategies; examining the evolution of the learned representation geometry; exploring their robustness to varying tasks and environments; and ultimately designing new representation learning algorithms and losses. By taking a careful yet directly empirical approach to studying learned representations, we will advance our community's understanding of how to more effectively use neural networks for tackling challenging real-world problems with RL.
Programme
Discovery Launch Supplement
Emplacement
Quebec, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data

Tel que publié

Application id
DGECR-2024-00219
Name nom
Castro, Pablo
Department département
Département d'informatique et de recherche opérationelle
Organization id
63
Institution établissement
Université de Montréal
Province en
Québec
Province fr
Québec
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2024
Award amount
12500
Program id
DGECR
Program name en
Discovery Launch Supplement
Program name fr
Supplément Tremplin vers la découverte
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
1207
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Information, computer and communication technologies
Area of application fr
Technologie de l'information, des ordinateurs et des réseaux de communications
Research subject code
2800
Research subject group en
Artificial intelligence
Research subject group fr
Intelligence artificielle
Research subject en
Artificial intelligence (computer vision, use2603)
Research subject fr
Intelligence artificielle (vision artificielle, utiliser 2603)
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Building robust representations for deep reinforcement learning
Keywords
reinforcement learning; representation learning; behavioural metrics; markov decision processes; bisimulation metrics; deep learning; neural networks
Application summary
Reinforcement learning (RL) agents aim to learn behaviour policies, which are mappings from states to a distribution over a set of possible actions. For this purpose, most methods rely on learning value functions, which map states and actions to real values. Algorithms can use these learned values to obtain policies either directly (by choosing the action with the highest value from each state) or indirectly (by using them as a "critic" in actor-critic methods). The foundations of RL are often presented in terms of tabular systems, which are ones where state-action values can be stored in a table. Most problems of interest, however, have state-action spaces which are too large to be stored as such. Thus, researchers often operate on state features which can be more compactly represented, and have the added benefit of generalizability. Specifically, effective state features will enable unseen states to benefit from learning updates performed on other states which are similar, thereby increasing the coverage of the agent's learned values without much loss. Theoretical works assume state features are given, while most practical methods learn the features along with the value function. Usually, this is done with neural networks, and the combination of RL with neural networks is referred to as deep reinforcement learning (DRL). The state features, also called state representations, are learned implicitly since the learning objective of most RL agents is usually limited to value estimation. There are works which do explicitly learn state representations by learning representations that respect some notion of similarity between states, such as similarity of raw inputs, temporal occurrence, and long-term behaviors, and which are learned via differentiable losses. These representation losses are learned in conjunction with the standard RL loss on a pre-specified layer of the neural network. The resulting performance depends largely on network design and hyper-parameter optimization. The effectiveness of these representations is usually measured only through agent performance on standard benchmarks, which fails to provide proper insights into the true underlying properties of the learned representations. This proposal takes an experimental approach to understand learned representations by: investigating quantifiable measures of the quality of a learned representation; studying their sensitivity to hyperparameters, network designs, and exploration strategies; examining the evolution of the learned representation geometry; exploring their robustness to varying tasks and environments; and ultimately designing new representation learning algorithms and losses. By taking a careful yet directly empirical approach to studying learned representations, we will advance our community's understanding of how to more effectively use neural networks for tackling challenging real-world problems with RL.