Grant: Discovery Grants Program - Individual
University of Alberta
$78,000.00
Sparse representations for reinforcement learning
EF 2023-24
Résumé
A key component of an artificial intelligence system is the ability to process and learn from a high-dimensional, high-volume sensory stream of information. For example, an agent controlling the pumps in an industrial plant continually receives sensory information about temperatures and energy consumption, to continually adjust the motor speed in real-time to optimize performance. To make such decision, the agents needs to be able to predict the long-term outcomes of their behaviour. For example, if the industrial agent can predict the long-term temperature of the motor, given the current state of the system, they can use these predictions to improve their decisions and ensure motors are not damaged. Such predictions, however, can be difficult to learn accurately from raw sensory information. Predictions are typically learned as functions of inputted sensory information. For example, the prediction of motor temperature in five minutes could be approximated as a polynomial function of the last ten recorded temperature and motor speeds. Polynomials, however, are only one possible functional form, and not necessarily the best one. Further, to obtain general learning agents, the functional forms should be effective across multiple settings or tasks. This is the goal of representation learning in reinforcement learning: identifying a general mapping from a sequence of raw sensory information to a set of features, that facilitates accurate predictions. The goal in my research is to understandboth theoretically and empiricallythe properties of effective representations for a reinforcement learning agent learning on a continual stream of sensory information. A part of this challenge is to identify simpler representations for which we can provide optimization guarantees, but that are nonetheless sufficiently powerful to facilitate learning. Continuing preliminary research, I will explore prototype-based (kernel) representations and a sparse supervised auto-encoder representation. We have already found that, within this class of simpler representations, we can find computational models that provide highly accurate predictions, but are more amenable to theoretical analysis. A core component of this research direction will be to investigate sparsity as a generally useful property of representations, and how we can encode that property into our representation learning algorithms. If successful, this research will have important scientific and societal benefits. This research will contribute to a core endeavour in artificial intelligence: understanding how to develop intelligent agents that can learn in complex environments. This understanding, in turn, will contribute to improving the robustness of automated decision-making systems, which are becoming ubiquitous in our world, including in industrial systems and factories, in self-driving vehicles and even in our homes.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Alberta, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data
Tel que publié
Application id
RGPIN-2018-05721
Name nom
White, Martha
Department département
Computing Science
Organization id
9
Institution établissement
University of Alberta
Province en
Alberta
Province fr
Alberta
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2023
Competition year année de concours
2018
Award amount
78000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
1200
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Advancement of knowledge
Area of application fr
Avancement des connaissances
Research subject code
9999
Research subject group en
Not available
Research subject group fr
Not available
Research subject en
Non disponible
Research subject fr
Non disponible
Application title
Sparse representations for reinforcement learning
Keyword
machine learning; reinforcement learning; representation learning
Application summary
A key component of an artificial intelligence system is the ability to process and learn from a high-dimensional, high-volume sensory stream of information. For example, an agent controlling the pumps in an industrial plant continually receives sensory information about temperatures and energy consumption, to continually adjust the motor speed in real-time to optimize performance. To make such decision, the agents needs to be able to predict the long-term outcomes of their behaviour. For example, if the industrial agent can predict the long-term temperature of the motor, given the current state of the system, they can use these predictions to improve their decisions and ensure motors are not damaged. Such predictions, however, can be difficult to learn accurately from raw sensory information. Predictions are typically learned as functions of inputted sensory information. For example, the prediction of motor temperature in five minutes could be approximated as a polynomial function of the last ten recorded temperature and motor speeds. Polynomials, however, are only one possible functional form, and not necessarily the best one. Further, to obtain general learning agents, the functional forms should be effective across multiple settings or tasks. This is the goal of representation learning in reinforcement learning: identifying a general mapping from a sequence of raw sensory information to a set of features, that facilitates accurate predictions. The goal in my research is to understandboth theoretically and empiricallythe properties of effective representations for a reinforcement learning agent learning on a continual stream of sensory information. A part of this challenge is to identify simpler representations for which we can provide optimization guarantees, but that are nonetheless sufficiently powerful to facilitate learning. Continuing preliminary research, I will explore prototype-based (kernel) representations and a sparse supervised auto-encoder representation. We have already found that, within this class of simpler representations, we can find computational models that provide highly accurate predictions, but are more amenable to theoretical analysis. A core component of this research direction will be to investigate sparsity as a generally useful property of representations, and how we can encode that property into our representation learning algorithms. If successful, this research will have important scientific and societal benefits. This research will contribute to a core endeavour in artificial intelligence: understanding how to develop intelligent agents that can learn in complex environments. This understanding, in turn, will contribute to improving the robustness of automated decision-making systems, which are becoming ubiquitous in our world, including in industrial systems and factories, in self-driving vehicles and even in our homes.