Grant: Discovery Grants Program - Individual
McGill University
$46,000.00
Designing sensitivity analyses for weakly identified or non-identified models
EF 2022-23
Résumé
Many modern statistical analyses require choices regarding parameters or assumptions that can not be well-identified or verified from the observed data. For example, any data analysis using Bayesian methods requires selecting prior distributions and hyperparameters for those prior distributions. Analyses of coarsened data, such as censored time-to-event data or data with missing values, may not be able to assume that the data are coarsened at random and thus must choose models which specify how the coarsening of the data depends on the unobserved values themselves. Causal analyses require assumptions regarding the relationships of observed variables to unobserved confounders and mediators. Random effect meta-analyses make assumptions about the distribution of the random effects and the underlying similarity between study populations, often for small numbers of studies. In all of the above cases, researchers conducting analyses often simply make one choice for decisions or choices of hyperparameters or inputs, which can be lead to disastrous consequences if those choices are poor. The peer review community has increasingly demanded sensitivity analyses from researchers in these situations. However, the sensitivity analyses are often conducted haphazardly, relying on “reasonable choices” for model inputs and assumptions that are not systematically related or justified. The purpose of the proposed grant is to create a principled computational approach to sensitivity analyses that focuses on the sensitivity of potential decisions made from statistical analyses to different assumptions and model inputs, rather than focusing on the sensitivity of the parameter estimates themselves. Designing principled sensitivity analyses requires overcoming several obstacles. First, any solution must be scalable to large numbers of parameters. Second, different statistical analyses will require users to be able to specify different kinds of decision thresholds. Third, there must be the development of publicly available software for users to be able to implement the sensitivity analyses. The basic idea behind Bayesian optimization approach is to use Bayeisan non-parametric modelling (typically using Gaussian processes) to model the output of an expensive to evaluate objective function. For example, it has been used to choose parameter settings (numbers of levels, numbers of hidden nodes, etc.) for neural network and deep Bayes learning models to optimize prediction. Bayesian optimization has also been used for estimating regret bounds for decision process problems. The proposed project will develop methods for using Bayesian optimization to identify optimal regions, rather than optimal values, that correspond to important changes in decision making due to different choices of optimal parameters.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Quebec, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data
Tel que publié
Application id
RGPIN-2018-06439
Name nom
Steele, Russell
Department département
Mathematics and Statistics - Mathematics and Statistics
Organization id
61
Institution établissement
McGill University
Province en
Québec
Province fr
Québec
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2022
Competition year année de concours
2018
Award amount
46000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1508
Committee name en
Mathematics and Statistics
Committee name fr
Mathématiques et statistique
Area of application code
1205
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Mathematical sciences
Area of application fr
Sciences mathématiques
Research subject code
3000
Research subject group en
Statistics and probability
Research subject group fr
Statistique et probabilité
Research subject en
Statistics and probability
Research subject fr
Statistique et probabilité
Installment
inst:5;6
Partie
2
Nb partie
6
Application title
Designing sensitivity analyses for weakly identified or non-identified models
Application summary
Many modern statistical analyses require choices regarding parameters or assumptions that can not be well-identified or verified from the observed data. For example, any data analysis using Bayesian methods requires selecting prior distributions and hyperparameters for those prior distributions. Analyses of coarsened data, such as censored time-to-event data or data with missing values, may not be able to assume that the data are coarsened at random and thus must choose models which specify how the coarsening of the data depends on the unobserved values themselves. Causal analyses require assumptions regarding the relationships of observed variables to unobserved confounders and mediators. Random effect meta-analyses make assumptions about the distribution of the random effects and the underlying similarity between study populations, often for small numbers of studies. In all of the above cases, researchers conducting analyses often simply make one choice for decisions or choices of hyperparameters or inputs, which can be lead to disastrous consequences if those choices are poor. The peer review community has increasingly demanded sensitivity analyses from researchers in these situations. However, the sensitivity analyses are often conducted haphazardly, relying on “reasonable choices” for model inputs and assumptions that are not systematically related or justified. The purpose of the proposed grant is to create a principled computational approach to sensitivity analyses that focuses on the sensitivity of potential decisions made from statistical analyses to different assumptions and model inputs, rather than focusing on the sensitivity of the parameter estimates themselves. Designing principled sensitivity analyses requires overcoming several obstacles. First, any solution must be scalable to large numbers of parameters. Second, different statistical analyses will require users to be able to specify different kinds of decision thresholds. Third, there must be the development of publicly available software for users to be able to implement the sensitivity analyses. The basic idea behind Bayesian optimization approach is to use Bayeisan non-parametric modelling (typically using Gaussian processes) to model the output of an expensive to evaluate objective function. For example, it has been used to choose parameter settings (numbers of levels, numbers of hidden nodes, etc.) for neural network and deep Bayes learning models to optimize prediction. Bayesian optimization has also been used for estimating regret bounds for decision process problems. The proposed project will develop methods for using Bayesian optimization to identify optimal regions, rather than optimal values, that correspond to important changes in decision making due to different choices of optimal parameters.