Grant: Discovery Grants Program - Individual

My program of research aims to develop reinforcement learning (RL)/dynamic treatment regime (DTR) methodology that can be used to provide data-driven decision support for systems where sequential decision- making is important and there is a human in the loop. In support of this aim, the proposed work will develop: 1) RL methodology for settings where action spaces are large and discrete but data are limited and hence expert knowledge about how actions are related is needed; 2) RL methodology for settings where different actions influence different nested components of a factored state space; and, 3) Explainable AI (XAI) methodology for conveying the evidence generated by applying the developed methods to data sources for decision support. This work is needed because 1) existing RL for large discrete action spaces assume large data sources (e.g. access to environment simulators); 2) while there is substantial work on 'hierarchical RL,' its core methods address action hierarchy where actions have compositional structure, rather than a state-space hierarchy where different actions have different scope of effect; and, 3) XAI for RL is not well-developed in general, and statistical and visualization methodology specifically for presenting recommendations together with uncertainty is lacking. This will build on our work on learning in large action spaces, Bayesian methods and structure learning methods and applications, data analysis and benchmark development for multi-level RL problems, and XAI. It will also continue to build on earlier work on algorithms for conveying the uncertainty associated with the decisions based on the statistical methodology of prediction and tolerance intervals and on visualizing relationships in functions of binary variables, which lay the foundation for nuanced recommendations that are appropriate for decision-makers with more than one goal while providing statistical uncertainty information about the outcomes of decisions. By creating new decision-making methodology, the outputs will impact applications where decision support over large action spaces is needed. For example, in health care, sequential treatment decisions are often made jointly and simultaneously, whether because patients need to manage several conditions (e.g., treatments for heart disease and diabetes), or because of the way care is delivered (e.g., a physiotherapy treatment plan for the week). In wildfire management, a joint decision must be made about which locations to send resources to in anticipation of future fires. In both examples, the action space is very large, and data resources are poor relative to other domains like language or video games. Hence, the proposed research will contribute both to the future development of decision support techniques and to important applications where action spaces are complex, including in health care and public safety.