Grant: Discovery Grants Program - Individual
University of Waterloo
$32,000.00
Neural mechanisms of state-value interactions in reinforcement learning
FY 2014-15
Summary
In a complex and changing world, organisms face a daunting task: out of many possible actions, which one is best? Influential theoretical accounts of how to solve this problem propose segmenting the world into situations (""states""), actions, and rewards, and keeping score of how much reward results from taking certain actions in particular states. In this ""reinforcement learning"" framework, decision-making and learning from feedback are intimately connected: actions are chosen based on how well they turned out in the past. Key quantities in this process are the ""prediction error"" which reflects the difference between received and expected reward, and the ""state value"" which allows the agent to learn from actions which are not immediately rewarded, but are important in obtaining reward later. Despite the apparent simplicity of this reinforcement learning scheme, these theoretical quantities show striking similarities with signals recorded from the mammalian brain, suggesting that it captures fundamental aspects of information processing in biological systems. Experimental studies of reinforcement learning in the brain have overwhelmingly focused on value-related signals, such as state values and prediction errors. However, these signals depend on appropriate state representations. For instance, the colour of the walls in a restaurant is presumably irrelevant in deciding which menu item to choose, but the colour of a traffic light is critical in deciding whether to cross the road. How the brain comes up with state representations suitable for reinforcement learning is a crucial, yet poorly understood issue. This proposal focuses on the interaction between value and state signals in the rat brain. In particular, we will record value and state signals during behaviour to provide a novel perspective on the mechanistic underpinnings of decision-making in biological systems. As the concepts of state and value lie at the center of current theories of learning and decision-making, this work will impact a wide range of fields from neuroscience, psychology and cognitive science to economics, computer science and robotics.
Program
Discovery Grants Program - Individual
Recipient
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
402942-2012
Name nom
vanderMeer, Matthijs
Department département
Biology
Organization id
33
Institution établissement
University of Waterloo
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2014
Competition year année de concours
2012
Award amount
32000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
Discovery grants programs
Group fr
Programmes de subventions à la découverte
Committee code
1502
Committee name en
Biological Systems and Functions
Committee name fr
Systèmes et fonctions biologiques
Area of application code
1210
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Psychology
Area of application fr
Psychologie
Research subject code
6310
Research subject group en
Psychology
Research subject group fr
Psychologie
Research subject en
Behavioural neuroscience --reward, motivation
Research subject fr
Neurologie du comportement --récompense, motivation
Partie
3
Nb partie
3
Application title
Neural mechanisms of state-value interactions in reinforcement learning
Keyword
NSERC
Application summary
In a complex and changing world, organisms face a daunting task: out of many possible actions, which one is best? Influential theoretical accounts of how to solve this problem propose segmenting the world into situations (""states""), actions, and rewards, and keeping score of how much reward results from taking certain actions in particular states. In this ""reinforcement learning"" framework, decision-making and learning from feedback are intimately connected: actions are chosen based on how well they turned out in the past. Key quantities in this process are the ""prediction error"" which reflects the difference between received and expected reward, and the ""state value"" which allows the agent to learn from actions which are not immediately rewarded, but are important in obtaining reward later. Despite the apparent simplicity of this reinforcement learning scheme, these theoretical quantities show striking similarities with signals recorded from the mammalian brain, suggesting that it captures fundamental aspects of information processing in biological systems. Experimental studies of reinforcement learning in the brain have overwhelmingly focused on value-related signals, such as state values and prediction errors. However, these signals depend on appropriate state representations. For instance, the colour of the walls in a restaurant is presumably irrelevant in deciding which menu item to choose, but the colour of a traffic light is critical in deciding whether to cross the road. How the brain comes up with state representations suitable for reinforcement learning is a crucial, yet poorly understood issue. This proposal focuses on the interaction between value and state signals in the rat brain. In particular, we will record value and state signals during behaviour to provide a novel perspective on the mechanistic underpinnings of decision-making in biological systems. As the concepts of state and value lie at the center of current theories of learning and decision-making, this work will impact a wide range of fields from neuroscience, psychology and cognitive science to economics, computer science and robotics.