Grant: Discovery Grants Program - Individual
University of Alberta
$24,000.00
Scalable Reinforcement Learning Methods for Learning in Real-Time with Robots
EF 2023-24
Résumé
Reinforcement learning brings the promise of continually adaptive systems for numerous tasks that humans do well but are physically laborious such as housekeeping, warehouse fulfillment, and delivery services. Such tasks require a common-sense understanding from the agent's part of a dynamically changing physical environment, which is difficult to enumerate and include in a system through hand-engineering. The proposed program aims at developing real-time learning robotic systems that interact with the physical world and adapt in real-time. Some of the most promising approaches in reinforcement learning for robotics are based on learning from human-provided demonstration data and simulators. However, approaches reliant on human interventions are not scalable or sufficient for developing robotics systems that can adapt their performance in real-time under new or changing environments. Our proposed program complements the existing approaches by developing scalable and automatic mechanisms for continually learning robotic systems. All advanced deep reinforcement learning methods for control use expensive learning mechanisms such as those based on experience replay buffers. While such expensive learning mechanisms are more appropriate for training offline or over clouds, we propose a lightweight onboard learning system to adapt and react to changes quickly in real-time. Our proposed onboard learning system will be composed of computationally inexpensive and stable policy and representation learning algorithms. We consider the policy to be only the last semi-linear layer of the network, for which gradient updates can be made more stably without using replay buffers. In addition, the onboard system will perform representation learning only through random perturbation to a small portion of the hidden nodes. We investigate whether such a lightweight learning system in conjunction with a more expensive replay-based learning system performs better than replay-based learning alone. The proposed program also aims at developing efficient and stable policy and representation learning methods. We develop a theoretical framework that enriches our understanding of how to create new and efficient policy learning methods in a directed way. For representation learning, we extend an existing strategy for representation search called generate-and-test to reinforcement learning. We develop a general mechanism of generate-and-test where the utility of features is defined solely based on the loss function, allowing applicability to any loss function and neural architecture. Computationally inexpensive learning mechanisms are essential for making reinforcement learning systems more accessible and applicable to robotics. The lightweight onboard system of the proposed program will allow graduate students, entrepreneurs, and enthusiasts around the world to build continually learning robots more easily, relieving humans from numerous laborious tasks.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Alberta, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data
Tel que publié
Application id
RGPIN-2021-02690
Name nom
Mahmood, Ashique
Department département
Computing Science
Organization id
9
Institution établissement
University of Alberta
Province en
Alberta
Province fr
Alberta
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2023
Competition year année de concours
2021
Award amount
24000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
1205
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Mathematical sciences
Area of application fr
Sciences mathématiques
Research subject code
2605
Research subject group en
Robotics
Research subject group fr
Pattern analysis and machine intelligence
Research subject en
Robotique
Research subject fr
Analyse de formes et intelligence artificielle
Application title
Scalable Reinforcement Learning Methods for Learning in Real-Time with Robots
Keyword
real-time; reinforcement learning; robot learning
Application summary
Reinforcement learning brings the promise of continually adaptive systems for numerous tasks that humans do well but are physically laborious such as housekeeping, warehouse fulfillment, and delivery services. Such tasks require a common-sense understanding from the agent's part of a dynamically changing physical environment, which is difficult to enumerate and include in a system through hand-engineering. The proposed program aims at developing real-time learning robotic systems that interact with the physical world and adapt in real-time. Some of the most promising approaches in reinforcement learning for robotics are based on learning from human-provided demonstration data and simulators. However, approaches reliant on human interventions are not scalable or sufficient for developing robotics systems that can adapt their performance in real-time under new or changing environments. Our proposed program complements the existing approaches by developing scalable and automatic mechanisms for continually learning robotic systems. All advanced deep reinforcement learning methods for control use expensive learning mechanisms such as those based on experience replay buffers. While such expensive learning mechanisms are more appropriate for training offline or over clouds, we propose a lightweight onboard learning system to adapt and react to changes quickly in real-time. Our proposed onboard learning system will be composed of computationally inexpensive and stable policy and representation learning algorithms. We consider the policy to be only the last semi-linear layer of the network, for which gradient updates can be made more stably without using replay buffers. In addition, the onboard system will perform representation learning only through random perturbation to a small portion of the hidden nodes. We investigate whether such a lightweight learning system in conjunction with a more expensive replay-based learning system performs better than replay-based learning alone. The proposed program also aims at developing efficient and stable policy and representation learning methods. We develop a theoretical framework that enriches our understanding of how to create new and efficient policy learning methods in a directed way. For representation learning, we extend an existing strategy for representation search called generate-and-test to reinforcement learning. We develop a general mechanism of generate-and-test where the utility of features is defined solely based on the loss function, allowing applicability to any loss function and neural architecture. Computationally inexpensive learning mechanisms are essential for making reinforcement learning systems more accessible and applicable to robotics. The lightweight onboard system of the proposed program will allow graduate students, entrepreneurs, and enthusiasts around the world to build continually learning robots more easily, relieving humans from numerous laborious tasks.