Canada Spends Logo

Explorez notre analyse du budget 2026

Grant: Alliance Grants

McGill University

$25,000.00

Reinforcement Learning with delayed reward feedback

EF 2024-25

Résumé
Reinforcement learning (RL) is a promising tool for developing agents which learn to act optimally by repeated interactions in unknown environments. The current RL practice has achieved several successes in the last decade; with perhaps the most famous being AlphaGo, which beat the human world champion in the game of Go. The current success of reinforcement learning is based on a combination of RL theory with recent advancements in deep learning. However, the current theory is restricted to idealized models and, therefore, the success of RL has been limited to idealized simulation-based environments, which satisfy the modeling assumptions made in the theory. However, these assumptions are not valid in real world applications such as recommender systems, personalized medicine, integration of renewable generation in smart grids, and many others. The objective of this proposal is relax a key modeling assumption in RL theory and to develop new theories and algorithms that are able to address learning with delayed feedback. The convergence guarantees of standard RL algorithms rely on stochastic approximation theory, which does not account for delay. To address this issue, we propose to use fundamentally new ideas from stability of time-delay systems to generalize stochastic approximation theory to handle feedback delays. We will then use this generalized theory to identify sufficient conditions for the convergence of RL algorithms with feedback delay and verify these conditions with extensive simulation experiments. The proposed project will lead to the training of one PhD student, develop a new collaboration between Canada and France, and lead to the advancement of knowledge in the field of reinforcement learning, thereby maintaining Canada's position as a leader in the field.
Programme
Alliance Grants
Bénéficiaire
Emplacement
Quebec, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data

Tel que publié

Application id
592356-2023
Name nom
Mahajan, Aditya A
Department département
Electrical and Computer Engineering
Organization id
61
Institution établissement
McGill University
Province en
Québec
Province fr
Québec
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2023
Award amount
25000
Program id
ALLRP
Program name en
Alliance Grants
Program name fr
Subventions Alliance
Group en
RESEARCH PARTNERSHIPS
Group fr
PARTENARIATS DE RECHERCHE
Committee code
96
Committee name en
RPP Internal Decision Cttee
Committee name fr
Cté de décision interne - PPR
Area of application code
1207
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Information, computer and communication technologies
Area of application fr
Technologie de l'information, des ordinateurs et des réseaux de communications
Research subject code
2510
Research subject group en
Electrical and electronic engineering
Research subject group fr
Génie électrique et électronique
Research subject en
Adaptive, learning and evolutionary systems
Research subject fr
Systèmes d'apprentissage adaptatifs et évolutifs
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Reinforcement Learning with delayed reward feedback
Keywords
delay differential equations; delayed feedback; reinforcement learning; stochastic approximation
Application summary
Reinforcement learning (RL) is a promising tool for developing agents which learn to act optimally by repeated interactions in unknown environments. The current RL practice has achieved several successes in the last decade; with perhaps the most famous being AlphaGo, which beat the human world champion in the game of Go. The current success of reinforcement learning is based on a combination of RL theory with recent advancements in deep learning. However, the current theory is restricted to idealized models and, therefore, the success of RL has been limited to idealized simulation-based environments, which satisfy the modeling assumptions made in the theory. However, these assumptions are not valid in real world applications such as recommender systems, personalized medicine, integration of renewable generation in smart grids, and many others. The objective of this proposal is relax a key modeling assumption in RL theory and to develop new theories and algorithms that are able to address learning with delayed feedback. The convergence guarantees of standard RL algorithms rely on stochastic approximation theory, which does not account for delay. To address this issue, we propose to use fundamentally new ideas from stability of time-delay systems to generalize stochastic approximation theory to handle feedback delays. We will then use this generalized theory to identify sufficient conditions for the convergence of RL algorithms with feedback delay and verify these conditions with extensive simulation experiments. The proposed project will lead to the training of one PhD student, develop a new collaboration between Canada and France, and lead to the advancement of knowledge in the field of reinforcement learning, thereby maintaining Canada's position as a leader in the field.