Grant: Discovery Grants Program - Individual
University of Waterloo
$50,000.00
Safe Reinforcement Learning for Real-World Autonomous Driving in Urban Environments
FY 2023-24
Summary
Reinforcement learning (RL) can be defined as learning from interactions to achieve a goal. While RL is a promising method for control systems design, its application to real-world safety-critical tasks, such as motion planning and control of autonomous vehicles (AVs), is still challenging. This method involves interactions between an agent and its environment, where a wide range of actions should be explored, including many failed iterations, before achieving the desired goal. When RL is deployed on a physical system like AV, there is a possibility of unsafe actions (decisions) by the agent (AV's controller) that can damage the vehicle or hurt people. As a result, certifying safety is essential during learning and deploying RL for real-life AV applications. While other techniques have been proposed to achieve RL with safety features, action projection methods can provide the closest provably safe action. An elegant action projection approach can be obtained by coupling RL and Control Barrier Functions (CBFs). We can apply RL-CBF techniques to find proper RL actions for AVs, like changing their speeds or steering angles, while satisfying safety constraints, such as keeping safe distances from nearby vehicles. Although RL-CBF methods have great potential, most RL-CBF AV motion planners and controllers proposed previously have been designed with unrealistic assumptions, and limited studies exist for complex urban driving scenarios like unsignalized intersections and roundabouts. This research program will advance existing RL-CBF AV safety-critical controllers by considering more realistic models and constraints while addressing optimization infeasibilities and modeling uncertainties. The resulting advanced RL-CBF AV controllers will be evaluated and compared against one another and with enhanced versions of promising non-learning CBF-based controllers proposed in the past. For our comparative study and to determine the best RL-CBF AV controller for urban driving, we will first run high-fidelity simulations. For a more effective evaluation of the best identified RL-CBF AV controller, we will conduct hardware-in-the-loop tests and real-world AV experiments. As a result, we will develop a novel RL-CBF control synthesis framework for real-world AV safety-critical tasks applicable to highly complex, arbitrary, and unseen driving scenarios in urban environments. Our research will add critical competitive values to AVs built by Canadian car companies while making their products safer for real-life applications. It will also help address the upcoming shortage of skilled engineers for the fast-growing AV job market. The proposed research will offer unique opportunities to combine RL with elegant control-theoretic techniques to ensure safety that will be applicable to other areas like unmanned aerial vehicles and human-robot collaboration. This will contribute to putting Canada at the forefront of safety-critical control systems design innovations.
Program
Discovery Grants Program - Individual
Recipient
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
RGPIN-2023-03678
Name nom
Lashgarian Azad, Nasser
Department département
Systems Design Engineering
Organization id
33
Institution établissement
University of Waterloo
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2023
Competition year année de concours
2023
Award amount
50000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1512
Committee name en
Mechanical Engineering
Committee name fr
Génie mécanique
Area of application code
700
Area of application group en
Transportation systems and services
Area of application group fr
Services et systèmes de transport
Area of application en
Transportation systems and services
Area of application fr
Services et systèmes de transport
Research subject code
9999
Research subject group en
Not available
Research subject group fr
Not available
Research subject en
Non disponible
Research subject fr
Non disponible
Application title
Safe Reinforcement Learning for Real-World Autonomous Driving in Urban Environments
Keyword
Autonomous Vehicle Motion Planning and Control; Control Barrier Functions; Full-Scale Autonomous Golf Cart Experiments; Hardware-in-the-Loop Tests; High-Fidelity Simulations; Model Uncertainties; Optimization Infeasibilities; Reinforcement Learning; Safety-Critical Control; Urban Driving Scenarios
Application summary
Reinforcement learning (RL) can be defined as learning from interactions to achieve a goal. While RL is a promising method for control systems design, its application to real-world safety-critical tasks, such as motion planning and control of autonomous vehicles (AVs), is still challenging. This method involves interactions between an agent and its environment, where a wide range of actions should be explored, including many failed iterations, before achieving the desired goal. When RL is deployed on a physical system like AV, there is a possibility of unsafe actions (decisions) by the agent (AV's controller) that can damage the vehicle or hurt people. As a result, certifying safety is essential during learning and deploying RL for real-life AV applications. While other techniques have been proposed to achieve RL with safety features, action projection methods can provide the closest provably safe action. An elegant action projection approach can be obtained by coupling RL and Control Barrier Functions (CBFs). We can apply RL-CBF techniques to find proper RL actions for AVs, like changing their speeds or steering angles, while satisfying safety constraints, such as keeping safe distances from nearby vehicles. Although RL-CBF methods have great potential, most RL-CBF AV motion planners and controllers proposed previously have been designed with unrealistic assumptions, and limited studies exist for complex urban driving scenarios like unsignalized intersections and roundabouts. This research program will advance existing RL-CBF AV safety-critical controllers by considering more realistic models and constraints while addressing optimization infeasibilities and modeling uncertainties. The resulting advanced RL-CBF AV controllers will be evaluated and compared against one another and with enhanced versions of promising non-learning CBF-based controllers proposed in the past. For our comparative study and to determine the best RL-CBF AV controller for urban driving, we will first run high-fidelity simulations. For a more effective evaluation of the best identified RL-CBF AV controller, we will conduct hardware-in-the-loop tests and real-world AV experiments. As a result, we will develop a novel RL-CBF control synthesis framework for real-world AV safety-critical tasks applicable to highly complex, arbitrary, and unseen driving scenarios in urban environments. Our research will add critical competitive values to AVs built by Canadian car companies while making their products safer for real-life applications. It will also help address the upcoming shortage of skilled engineers for the fast-growing AV job market. The proposed research will offer unique opportunities to combine RL with elegant control-theoretic techniques to ensure safety that will be applicable to other areas like unmanned aerial vehicles and human-robot collaboration. This will contribute to putting Canada at the forefront of safety-critical control systems design innovations.