Grant: Discovery Grants Program - Individual
The University of British Columbia
$38,000.00
Solving Partially Observable Markov Decision Processes Through A Deep Primal-Dual Framework
FY 2024-25
Summary
The partially observable Markov decision process (POMDP) is a fundamental model for sequential decision making under partial information. In the model, the state of the system (or object) under control cannot be fully observed by the decision maker. The model captures many important problems involving dynamic decision-making with learning. It has wide applications in management science, computer science, economics, healthcare management, etc. However, POMDPs are notoriously difficult to solve because they blend a belief process, which is updated forward in time, with a value-generation process, which is often computed backward in time. This mixture has led to two solution perspectives: the primal (forward or Bayesian) perspective and the dual (backward or future-reward) perspective. The former overshadows the latter in the existing literature, even in the methods that take both perspectives. Only recently, a stream of work has elevated the dual perspective to become a stand-alone framework (e.g., Zhang 2010).
My long-term goal is to propel the research of POMDP through a novel primal-dual framework, which is rooted in a deep understanding and balanced view of the two perspectives. This framework takes advantage of the aforementioned development on the dual perspective, which disentangles the forward and backward processes and allows them to be integrated freely. It enables a powerful and flexible primal-dual framework that achieves the best of both worlds.
My short-term objectives in the next five years are to: (1) Develop a new, exact algorithm for a general-purpose (moderate size) POMDP, which can be used in many application areas and will further enrich and strengthen the dual perspective and the primal-dual framework; (2) Develop new, approximation algorithms for large-scale POMDPs that will fully utilize both (primal) belief vectors and (dual) future-reward vectors via a deep primal-dual framework; (3) Derive novel analytical and structural results for POMDPs in operations research & management science (e.g., the machine-maintenance problem), healthcare management (e.g., the cancer screening and treatment problem), economics (e.g., the dynamic pricing and demand learning problem), and other application areas.
The proposed research program covers a broad scope and lies at the forefront of an impactful research area. It is ideal for training highly qualified personnel with diverse research interests and skills. The outputs of this program will benefit the academia, practitioners, policy makers, and the general public.
Program
Discovery Grants Program - Individual
Recipient
Location
British Columbia, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
RGPIN-2024-04432
Name nom
Zhang, Hao
Department département
Sauder School of Business - Sauder School of Business
Organization id
2
Institution établissement
The University of British Columbia
Province en
British Columbia
Province fr
Colombie-Britannique
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2024
Award amount
38000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1509
Committee name en
Civil, Industrial and Systems Engineering
Committee name fr
Génie civil, industriel et des systèmes
Area of application code
520
Area of application group en
Manufacturing processes and products
Area of application group fr
Procédés et produits industriels
Area of application en
Production and operations management
Area of application fr
Production et gestion des opérations
Research subject code
1601
Research subject group en
Industrial engineering
Research subject group fr
Génie industriel
Research subject en
Operations research and management science
Research subject fr
Recherche opérationnelle et science de la gestion
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Solving Partially Observable Markov Decision Processes Through A Deep Primal-Dual Framework
Keywords
partially observable markov decision processes; reinforcement learning; optimal control; dynamic programming; machine maintenance; artificial intelligence
Application summary
The partially observable Markov decision process (POMDP) is a fundamental model for sequential decision making under partial information. In the model, the state of the system (or object) under control cannot be fully observed by the decision maker. The model captures many important problems involving dynamic decision-making with learning. It has wide applications in management science, computer science, economics, healthcare management, etc. However, POMDPs are notoriously difficult to solve because they blend a belief process, which is updated forward in time, with a value-generation process, which is often computed backward in time. This mixture has led to two solution perspectives: the primal (forward or Bayesian) perspective and the dual (backward or future-reward) perspective. The former overshadows the latter in the existing literature, even in the methods that take both perspectives. Only recently, a stream of work has elevated the dual perspective to become a stand-alone framework (e.g., Zhang 2010).
My long-term goal is to propel the research of POMDP through a novel primal-dual framework, which is rooted in a deep understanding and balanced view of the two perspectives. This framework takes advantage of the aforementioned development on the dual perspective, which disentangles the forward and backward processes and allows them to be integrated freely. It enables a powerful and flexible primal-dual framework that achieves the best of both worlds.
My short-term objectives in the next five years are to: (1) Develop a new, exact algorithm for a general-purpose (moderate size) POMDP, which can be used in many application areas and will further enrich and strengthen the dual perspective and the primal-dual framework; (2) Develop new, approximation algorithms for large-scale POMDPs that will fully utilize both (primal) belief vectors and (dual) future-reward vectors via a deep primal-dual framework; (3) Derive novel analytical and structural results for POMDPs in operations research & management science (e.g., the machine-maintenance problem), healthcare management (e.g., the cancer screening and treatment problem), economics (e.g., the dynamic pricing and demand learning problem), and other application areas.
The proposed research program covers a broad scope and lies at the forefront of an impactful research area. It is ideal for training highly qualified personnel with diverse research interests and skills. The outputs of this program will benefit the academia, practitioners, policy makers, and the general public.