Grant: Discovery Grants Program - Individual
University of Waterloo
$55,000.00
Versatile Reinforcement Learning
EF 2024-25
Résumé
Exciting advances in AI have recently been achieved by large foundation models for vision, (e.g., CLIP, Stable diffusion), language (e.g., GPT-X, LLAMA-X) and decision making (e.g., GATO). Pre-trained foundation models have led to generalist agents that can be fine-tuned on specific tasks and often outperform specialist agents tailored and trained with data specific to a desired tasks. However, such foundation models are typically trained by supervised learning, which requires lots of labeled data and mimics expert demonstrations with all their imperfections. How can we train foundation models to go beyond imitation for tasks where the strategies used to generate the data are unsatisfactory? Furthermore, how do we ensure feasibility, safety and performance guarantees in industrial applications?
The long-term vision for my research program is the development of versatile Reinforcement Learning (RL) agents that generalize across tasks, outperform expert demonstrations used for training and offer performance guarantees. As demonstrated by AlphaGo, RL can significantly improve an agent initially trained by supervised learning. Similarly, ChatGPT leverages human feedback by reinforcement learning to improve GPT3.5 initially trained by supervised learning. Over the next five years, my research program will focus on three main thrusts: 1) RL foundation models that can be fine-tuned for many tasks to reduce the amount of data and computational search for optimal decision making, 2) adaptive offline RL to allow pretrained policies to adapt after deployment during their execution 3) constrained RL and inverse constrained RL to ensure feasibility, safety and performance guarantees are met in industrial applications.
There is potential for high impact since the resulting RL foundation models will provide generic and versatile solutions that reduce the amount of data needed, mitigate risks and improve decision making in a variety of applications including diagnostics and troubleshooting, recommender systems, material design, drug discovery, marketing, optimization, operations research, control, etc. Since these application domains cut across many sectors of the economy, trainees in my research program will become not only AI leaders, but will help to optimize decision making in finance, healthcare, transportation, manufacturing, etc. Furthermore, the pre-trained RL foundation models that will be developed in this research will become the starting point for fine tuning by anyone around the world, establishing Canada as a leader in AI.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Ontario, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data
Tel que publié
Application id
RGPIN-2024-04773
Name nom
Poupart, Pascal
Department département
Computer Science
Organization id
33
Institution établissement
University of Waterloo
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2024
Award amount
55000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
1207
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Information, computer and communication technologies
Area of application fr
Technologie de l'information, des ordinateurs et des réseaux de communications
Research subject code
2805
Research subject group en
Artificial intelligence
Research subject group fr
Intelligence artificielle
Research subject en
Learning and inference theories
Research subject fr
Théories de l'apprentissage et l'inférence
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Versatile Reinforcement Learning
Keywords
reinforcement learning; foundation models; offline learning; constraint learning; bayes adaptive
Application summary
Exciting advances in AI have recently been achieved by large foundation models for vision, (e.g., CLIP, Stable diffusion), language (e.g., GPT-X, LLAMA-X) and decision making (e.g., GATO). Pre-trained foundation models have led to generalist agents that can be fine-tuned on specific tasks and often outperform specialist agents tailored and trained with data specific to a desired tasks. However, such foundation models are typically trained by supervised learning, which requires lots of labeled data and mimics expert demonstrations with all their imperfections. How can we train foundation models to go beyond imitation for tasks where the strategies used to generate the data are unsatisfactory? Furthermore, how do we ensure feasibility, safety and performance guarantees in industrial applications?
The long-term vision for my research program is the development of versatile Reinforcement Learning (RL) agents that generalize across tasks, outperform expert demonstrations used for training and offer performance guarantees. As demonstrated by AlphaGo, RL can significantly improve an agent initially trained by supervised learning. Similarly, ChatGPT leverages human feedback by reinforcement learning to improve GPT3.5 initially trained by supervised learning. Over the next five years, my research program will focus on three main thrusts: 1) RL foundation models that can be fine-tuned for many tasks to reduce the amount of data and computational search for optimal decision making, 2) adaptive offline RL to allow pretrained policies to adapt after deployment during their execution 3) constrained RL and inverse constrained RL to ensure feasibility, safety and performance guarantees are met in industrial applications.
There is potential for high impact since the resulting RL foundation models will provide generic and versatile solutions that reduce the amount of data needed, mitigate risks and improve decision making in a variety of applications including diagnostics and troubleshooting, recommender systems, material design, drug discovery, marketing, optimization, operations research, control, etc. Since these application domains cut across many sectors of the economy, trainees in my research program will become not only AI leaders, but will help to optimize decision making in finance, healthcare, transportation, manufacturing, etc. Furthermore, the pre-trained RL foundation models that will be developed in this research will become the starting point for fine tuning by anyone around the world, establishing Canada as a leader in AI.