Canada Spends Logo

Explore our Budget 2026 analysis

Grant: Discovery Grants Program - Individual

University of Alberta

$55,000.00

Useful Even If Inaccurate: More Effective and Efficient Use of World Models in Reinforcement Learning

FY 2024-25

Summary
As summarized by the statistician George Box, "all models are wrong, but some are useful". Human decision-making and understanding of the world relies heavily on the construction of models. They are used to predict and plan courses of action, without the need to execute and observe the result of these actions. Box's aphorism recognizes that the world is too big and complex for us to know it completely. And yet, we still gain considerable advantage from exploiting inaccurate, but useful, models. The role of models in artificial intelligence is both similar and different. In reinforcement learning, models predominately consist of predictions of future states, observations, and rewards from taking courses of action in the world. They are used to allow an agent to predict and plan without needing excessive experience directly interacting with its environment. However, this is commonly done contrary to Box, more following the maxim of "make sure the model is nearly perfect, and it will be useful." Most successful applications of model-based reinforcement learning have been the result of models being sufficiently accurate as to be nearly indistinguishable from the agent's real environment. For early model-based algorithms with discrete enumerable state-spaces, it was possible to learn complete description of the decision process. In situations demanding function approximation and generalization, model-based algorithms have struggled. Recently we have seen new successful approaches, but these have coincided with improvements in model learning techniques along and increases in computation, so they often end up using more compute than the very simulation environment in which they operate. This trend is repeated in multi-agent reinforcement learning, where few works have even tried to plan without complete knowledge of the rules of the game. The aim of this proposal is to develop new techniques that can reduce the dependence on (near-) perfect models. First, it aims to develop new planning techniques to focus an agent's planning energy where it is most effective, thereby learning where models hold useful knowledge and where they do not. Second, it aims to develop new model learning techniques, that embrace the goal of being useful, rather than accurate. Success will open new avenues for ever-powerful models that can incorporate longer horizon planning, models learning to operate at multiple levels of abstraction, and models that can condition on the current real-world circumstance. In two other separate thrusts, this proposal aims to reduce the need for perfect representation in multiagent learning settings, as well as developing the formal mathematical foundations for agents that have incomplete or imperfect understanding of their task.  Overall, the proposal aims to open up model-based algorithms --- and their advantages --- to work in more messy, real-world situations where Box's maxim "all models are wrong, but some are useful" applies.
Program
Discovery Grants Program - Individual
Location
Alberta, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data

As published

Application id
RGPIN-2024-04975
Name nom
Bowling, Michael
Department département
Computing Science
Organization id
9
Institution établissement
University of Alberta
Province en
Alberta
Province fr
Alberta
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2024
Award amount
55000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
1200
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Advancement of knowledge
Area of application fr
Avancement des connaissances
Research subject code
2800
Research subject group en
Artificial intelligence
Research subject group fr
Intelligence artificielle
Research subject en
Artificial intelligence (computer vision, use2603)
Research subject fr
Intelligence artificielle (vision artificielle, utiliser 2603)
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Useful Even If Inaccurate: More Effective and Efficient Use of World Models in Reinforcement Learning
Keywords
reinforcement learning; world models; deep learning; representation learning; meta-learning; computational game theory; game playing
Application summary
As summarized by the statistician George Box, "all models are wrong, but some are useful". Human decision-making and understanding of the world relies heavily on the construction of models. They are used to predict and plan courses of action, without the need to execute and observe the result of these actions. Box's aphorism recognizes that the world is too big and complex for us to know it completely. And yet, we still gain considerable advantage from exploiting inaccurate, but useful, models. The role of models in artificial intelligence is both similar and different. In reinforcement learning, models predominately consist of predictions of future states, observations, and rewards from taking courses of action in the world. They are used to allow an agent to predict and plan without needing excessive experience directly interacting with its environment. However, this is commonly done contrary to Box, more following the maxim of "make sure the model is nearly perfect, and it will be useful." Most successful applications of model-based reinforcement learning have been the result of models being sufficiently accurate as to be nearly indistinguishable from the agent's real environment. For early model-based algorithms with discrete enumerable state-spaces, it was possible to learn complete description of the decision process. In situations demanding function approximation and generalization, model-based algorithms have struggled. Recently we have seen new successful approaches, but these have coincided with improvements in model learning techniques along and increases in computation, so they often end up using more compute than the very simulation environment in which they operate. This trend is repeated in multi-agent reinforcement learning, where few works have even tried to plan without complete knowledge of the rules of the game. The aim of this proposal is to develop new techniques that can reduce the dependence on (near-) perfect models. First, it aims to develop new planning techniques to focus an agent's planning energy where it is most effective, thereby learning where models hold useful knowledge and where they do not. Second, it aims to develop new model learning techniques, that embrace the goal of being useful, rather than accurate. Success will open new avenues for ever-powerful models that can incorporate longer horizon planning, models learning to operate at multiple levels of abstraction, and models that can condition on the current real-world circumstance. In two other separate thrusts, this proposal aims to reduce the need for perfect representation in multiagent learning settings, as well as developing the formal mathematical foundations for agents that have incomplete or imperfect understanding of their task.  Overall, the proposal aims to open up model-based algorithms --- and their advantages --- to work in more messy, real-world situations where Box's maxim "all models are wrong, but some are useful" applies.
University of Alberta - $55,000 Grant (2024-25) | Canada Spends