Canada Spends Logo

Explorez notre analyse du budget 2026

Grant: Discovery Grants Program - Individual

University of Toronto

$26,000.00

Algorithms for adaptive near-optimal control

EF 2010-11

Résumé
In almost every field of science or engineering there are processes we would like to control in an optimal way, e.g. maximizing speed or accuracy or fuel efficiency. But optimal control is computationally so demanding that it is out of reach except for simple tasks. The next best thing may be near-optimal control, where we compute a sequence of better and better controllers, moving ever closer to the optimal one. There are many algorithms for this purpose, but the most efficient and versatile is probably the method of generalized Hamilton-Jacobi-Bellman (GHJB) equations. Here I show that this method is in an important sense indirect, and can be improved by using a more direct form of supervised learning. The GHJB method is based on the fact that if we have a feedback controller, and we learn to compute the gradient grad-J of its cost-to-go function, then we can use that gradient to define a better controller. We can then use the new controller's grad-J to define a still-better controller, and so on. But GHJB works indirectly in the sense that it doesn't learn the best approximation to grad-J but instead learns a related function and from that infers a suboptimal estimate of grad-J. I show how it is possible to learn the gradient directly; e.g. we need signals that report grad-J(x) for different states x of the controlled process, and I show how to obtain them using a formula similar to the Euler-Lagrange equation. I compare this direct algorithm with GHJB on test problems from recent control papers, and I show that the direct method yields controllers that are more nearly optimal and simpler, requiring (on one complex task) 10 times fewer function evaluations and adjustable parameters. But much more testing is needed, and there is a great deal of work to be done improving and extending this approach.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Ontario, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data

Tel que publié

Application id
391349-2010
Name nom
Tweed, Douglas
Department département
Physiology
Organization id
31
Institution établissement
University of Toronto
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2010
Competition year année de concours
2010
Award amount
26000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
Discovery grants programs
Group fr
Programmes de subventions à la découverte
Committee code
1510
Committee name en
Electrical and Computer Engineering
Committee name fr
Génie électrique et informatique
Area of application code
1206
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Engineering
Area of application fr
Génie
Research subject code
2956
Research subject group en
Applied mathematics
Research subject group fr
Mathématiques appliquées
Research subject en
Optimisation and optimal control theory
Research subject fr
Optimisation et théories de commande optimale
Partie
1
Nb partie
5
Application title
Algorithms for adaptive near-optimal control
Keyword
NSERC
Application summary
In almost every field of science or engineering there are processes we would like to control in an optimal way, e.g. maximizing speed or accuracy or fuel efficiency. But optimal control is computationally so demanding that it is out of reach except for simple tasks. The next best thing may be near-optimal control, where we compute a sequence of better and better controllers, moving ever closer to the optimal one. There are many algorithms for this purpose, but the most efficient and versatile is probably the method of generalized Hamilton-Jacobi-Bellman (GHJB) equations. Here I show that this method is in an important sense indirect, and can be improved by using a more direct form of supervised learning. The GHJB method is based on the fact that if we have a feedback controller, and we learn to compute the gradient grad-J of its cost-to-go function, then we can use that gradient to define a better controller. We can then use the new controller's grad-J to define a still-better controller, and so on. But GHJB works indirectly in the sense that it doesn't learn the best approximation to grad-J but instead learns a related function and from that infers a suboptimal estimate of grad-J. I show how it is possible to learn the gradient directly; e.g. we need signals that report grad-J(x) for different states x of the controlled process, and I show how to obtain them using a formula similar to the Euler-Lagrange equation. I compare this direct algorithm with GHJB on test problems from recent control papers, and I show that the direct method yields controllers that are more nearly optimal and simpler, requiring (on one complex task) 10 times fewer function evaluations and adjustable parameters. But much more testing is needed, and there is a great deal of work to be done improving and extending this approach.