Grant: Discovery Grants Program - Individual
McGill University
$96,000.00
Reinforcement Learning for Artificial Intelligence
FY 2023-24
Summary
Artificial intelligence (AI) has made great progress in isolating different aspects of intelligence and proposing flexible representations and powerful algorithms that lead to competence in specific tasks. For example, AI agents are better than humans at playing games like Go, a feat once considered impossible. However, the sort of flexible, robust, and autonomous competence routinely exhibited by humans, or even animals, remains elusive. Our main research goal is to develop general AI methodology that relies, at its core, on reinforcement learning (RL). RL is an approach to learning from interaction with an environment. The key, well-known idea is that if an action results in a satisfactory state of affairs, or an improved state of affairs, then the tendency to produce that action is strengthened, i.e. reinforced. RL methods have developed at the confluence of control theory, neuroscience, animal learning, machine learning and operations research. They are very suitable for modeling problems faced by artificial autonomous agents that are learning to interact in real-time with complex, stochastic environments. Success stories include diverse domains, from game playing to plasma reactor control. From a theoretical point of view, these algorithms have traditionally been developed and analyzed using the framework of Markov Decision Processes (MDPs) and extensions such as Partially Observable Markov Decision Processes (POMDPs). However, using RL in complex, open-ended domains requires going beyond the current framework of MDPs/POMDPs, and re-thinking the theoretical underpinnings and algorithmic details of RL in the context of continual learning. This proposal aims to design algorithms that can automatically create representations and perform efficient exploration and credit assignment for continual reinforcement learning agents. Such agents are faced with environment that are non-stationary, or too large for an agent to even visit all relevant regions. Continual learning and adaptation to change are crucial to biological organisms. However, RL algorithms, especially those relying on deep neural networks, can overfit the initial data they see, and as a result, may not be able to continue learning over time. Moreover, theoretical results rely on recurrence properties of the agent-environment interaction, and on the ability of an agent to exhaustively and repeatedly visit all parts of its environment. We propose to revisit these assumption and build a more comprehensive theory and algorithmic toolkit of RL in non-stationary settings. For this purpose, we will formalize and study the notion of regret in this setting, and study its tradeoffs with computational and sample complexity. We will develop and evaluate knowledge representation methods which enable RL agents to explore in non-stationary environments and do efficient credit assignment, and we will test them in simulation domains, from large-scale games to simulating drug treatments.
Program
Discovery Grants Program - Individual
Recipient
Location
Quebec, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
RGPIN-2023-05773
Name nom
Precup, Doina
Department département
School of Computer Science
Organization id
61
Institution établissement
McGill University
Province en
Québec
Province fr
Québec
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2023
Competition year année de concours
2023
Award amount
96000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
1207
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Information, computer and communication technologies
Area of application fr
Technologie de l'information, des ordinateurs et des réseaux de communications
Research subject code
2700
Research subject group en
Information technology
Research subject group fr
Information technology
Research subject en
Technologie de l'information
Research subject fr
Technologie de l'information
Application title
Reinforcement Learning for Artificial Intelligence
Keyword
Deep Neural Networks; Markov Decision Processes; Partially Observable Markov Decision Processes; Planning; Reasoning under uncertainty; Reinforcement Learning; Temporal Abstraction
Application summary
Artificial intelligence (AI) has made great progress in isolating different aspects of intelligence and proposing flexible representations and powerful algorithms that lead to competence in specific tasks. For example, AI agents are better than humans at playing games like Go, a feat once considered impossible. However, the sort of flexible, robust, and autonomous competence routinely exhibited by humans, or even animals, remains elusive. Our main research goal is to develop general AI methodology that relies, at its core, on reinforcement learning (RL). RL is an approach to learning from interaction with an environment. The key, well-known idea is that if an action results in a satisfactory state of affairs, or an improved state of affairs, then the tendency to produce that action is strengthened, i.e. reinforced. RL methods have developed at the confluence of control theory, neuroscience, animal learning, machine learning and operations research. They are very suitable for modeling problems faced by artificial autonomous agents that are learning to interact in real-time with complex, stochastic environments. Success stories include diverse domains, from game playing to plasma reactor control. From a theoretical point of view, these algorithms have traditionally been developed and analyzed using the framework of Markov Decision Processes (MDPs) and extensions such as Partially Observable Markov Decision Processes (POMDPs). However, using RL in complex, open-ended domains requires going beyond the current framework of MDPs/POMDPs, and re-thinking the theoretical underpinnings and algorithmic details of RL in the context of continual learning. This proposal aims to design algorithms that can automatically create representations and perform efficient exploration and credit assignment for continual reinforcement learning agents. Such agents are faced with environment that are non-stationary, or too large for an agent to even visit all relevant regions. Continual learning and adaptation to change are crucial to biological organisms. However, RL algorithms, especially those relying on deep neural networks, can overfit the initial data they see, and as a result, may not be able to continue learning over time. Moreover, theoretical results rely on recurrence properties of the agent-environment interaction, and on the ability of an agent to exhaustively and repeatedly visit all parts of its environment. We propose to revisit these assumption and build a more comprehensive theory and algorithmic toolkit of RL in non-stationary settings. For this purpose, we will formalize and study the notion of regret in this setting, and study its tradeoffs with computational and sample complexity. We will develop and evaluate knowledge representation methods which enable RL agents to explore in non-stationary environments and do efficient credit assignment, and we will test them in simulation domains, from large-scale games to simulating drug treatments.