Canada Spends Logo

Explorez notre analyse du budget 2026

Grant: Discovery Grants Program - Individual

Carleton University

$31,000.00

Effective and Efficient Policy Optimization in Reinforcement Learning

EF 2024-25

Résumé
Reinforcement learning (RL), an important branch of machine learning and artificial intelligence, trains an intelligent agent to perform a specific task by letting it interact with a complex environment. It has a wide range of applications, such as competing in video/board games, automatic trading in financial markets, autonomous vehicles, and prescribing drugs in healthcare. Despite its tremendous success in recent years, training RL agents remains inefficient in practice. To gain a theoretical understanding of RL algorithms, especially in the modern deep learning regime, and to develop efficient algorithms for practical applications, this research proposal investigates three related learning scenarios in RL: online, offline and transfer settings. In the online setting, the RL agent improves itself by interacting with an environment, which can be costly and time-consuming if the interaction is not effectively utilized. We will study the convergence of various policy optimization methods including multiple actor-critic algorithms using recent advancements in the optimization community, and derive theorems based on different policy parametrizations. This would be essential to achieve better sample efficiency from a theoretical perspective and to develop powerful algorithms for real-world applications. In the offline setting, the agent conducts learning and inference using a fixed dataset without interacting with the environment. The quality of the dataset has a significant impact on offline learning. We will investigate the essential conditions for successful offline learning from a theoretical perspective. We will also explore the offline-to-online adaptation, in which the offline dataset can facilitate or speed up subsequent online training. In the transfer setting, the RL agent learns certain knowledge or skills from one task and applies them to a different but related task. Since knowledge can be in many different forms, it is non-trivial to formally describe what can or cannot be transferred between tasks. We will utilize recent advancements in the theory of transfer supervised learning to analyze transfer reinforcement learning and identify the key similarity measures suitable to understand task resemblance in the RL context. Finally, we will also investigate the effect of the replay buffer, which is ubiquitous in modern deep RL algorithms. By studying the effects of different sampling strategies, we can understand how the reinforcement signals can be propagated effectively, and design novel buffer management schemes beneficial for all aforementioned learning settings.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Ontario, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data

Tel que publié

Application id
RGPIN-2024-05357
Name nom
Wen, Junfeng
Department département
School of Computer Science
Organization id
23
Institution établissement
Carleton University
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2024
Award amount
31000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
1200
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Advancement of knowledge
Area of application fr
Avancement des connaissances
Research subject code
2805
Research subject group en
Artificial intelligence
Research subject group fr
Intelligence artificielle
Research subject en
Learning and inference theories
Research subject fr
Théories de l'apprentissage et l'inférence
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Effective and Efficient Policy Optimization in Reinforcement Learning
Keywords
machine learning; reinforcement learning; optimization; policy gradient; actor-critic; offline reinforcement learning; transfer reinforcement learning
Application summary
Reinforcement learning (RL), an important branch of machine learning and artificial intelligence, trains an intelligent agent to perform a specific task by letting it interact with a complex environment. It has a wide range of applications, such as competing in video/board games, automatic trading in financial markets, autonomous vehicles, and prescribing drugs in healthcare. Despite its tremendous success in recent years, training RL agents remains inefficient in practice. To gain a theoretical understanding of RL algorithms, especially in the modern deep learning regime, and to develop efficient algorithms for practical applications, this research proposal investigates three related learning scenarios in RL: online, offline and transfer settings. In the online setting, the RL agent improves itself by interacting with an environment, which can be costly and time-consuming if the interaction is not effectively utilized. We will study the convergence of various policy optimization methods including multiple actor-critic algorithms using recent advancements in the optimization community, and derive theorems based on different policy parametrizations. This would be essential to achieve better sample efficiency from a theoretical perspective and to develop powerful algorithms for real-world applications. In the offline setting, the agent conducts learning and inference using a fixed dataset without interacting with the environment. The quality of the dataset has a significant impact on offline learning. We will investigate the essential conditions for successful offline learning from a theoretical perspective. We will also explore the offline-to-online adaptation, in which the offline dataset can facilitate or speed up subsequent online training. In the transfer setting, the RL agent learns certain knowledge or skills from one task and applies them to a different but related task. Since knowledge can be in many different forms, it is non-trivial to formally describe what can or cannot be transferred between tasks. We will utilize recent advancements in the theory of transfer supervised learning to analyze transfer reinforcement learning and identify the key similarity measures suitable to understand task resemblance in the RL context. Finally, we will also investigate the effect of the replay buffer, which is ubiquitous in modern deep RL algorithms. By studying the effects of different sampling strategies, we can understand how the reinforcement signals can be propagated effectively, and design novel buffer management schemes beneficial for all aforementioned learning settings.