Grant: Discovery Grants Program - Individual
The University of British Columbia
$25,000.00
Adaptive Design for Fast Machine/Statistical Learning
FY 2023-24
Summary
The overarching goal of the research program is to extend Gaussian processes (GPs) to enable much more complex applications. First, new methodology will scale up GPs to enable large sample sizes and use adaptive sampling, for accurate statistical modelling of complex relationships arising cross a broad spectrum of scientific and engineering disciplines. Second, efficiencies in adaptive search methods using GPs will allow automatic tuning of computationally intensive machine/statistical learners. GPs have had profound impact on science and engineering, where they are used directly as machine/statistical learners. Complex computer codes of physical systems can be too slow for optimization, calibration of unknowns, sensitivity analysis, etc. GPs trained on limited computer model runs are used for these purposes as computationally fast surrogates for the particular scientific objective. It is well known, however, that the computational time to train a GP increases as the cube of the sample size. Thus, GPs are less attractive for sample sizes of a few thousand or more. Existing methods, mainly based on localized modelling or special fixed experimental designs, will be assessed to determine the domain of problems where they are effective. It is clear in advance, however, that new methods will be required for complex applications: those with moderate to high-dimensional input, nonlinear relationships, and/or high-order interaction effects. Only by adapting the experiment - taking further observations where the target function has special features - can a dense sampling of the input space be obtained where it matters. Divide and conquer methods are especially promising. How to divide high-dimensional space, how to choose sub-regions for data augmentation, and guidance on the number of new runs per iteration will be critical research questions here. GPs are also used indirectly in support of other machine-learning (ML) methods such as deep learning neural networks. Neural networks for image classification, for example, have "tuning" parameters that have to be set by the user, to determine the basic network architecture or regularization, for instance. Users tune these so-called hyperparameters by trying different values and attempting to minimize validation error in various ways. To obtain the validation error requires training the ML method, which is itself computationally very intensive. Hence, systematic methods known as Bayesian optimization train a GP to model the relationship between the hyperparameter settings and validation error, and hence adaptively optimize the error. The research program will continue work in my lab on "automatic ML", to minimize the number of tries of the expensive underlying ML algorithm. Advances here will likely have impact on other computationally challenging optimization problems where the objective is produced by an expensive algorithm.
Program
Discovery Grants Program - Individual
Recipient
Location
British Columbia, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
RGPIN-2019-05019
Name nom
Welch, William
Department département
Statistics - Statistics
Organization id
2
Institution établissement
The University of British Columbia
Province en
British Columbia
Province fr
Colombie-Britannique
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2023
Competition year année de concours
2019
Award amount
25000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1508
Committee name en
Mathematics and Statistics
Committee name fr
Mathématiques et statistique
Area of application code
1206
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Engineering
Area of application fr
Génie
Research subject code
3000
Research subject group en
Statistics and probability
Research subject group fr
Statistics and probability
Research subject en
Statistique et probabilité
Research subject fr
Statistique et probabilité
Application title
Adaptive Design for Fast Machine/Statistical Learning
Keyword
Bayesian Optimization; Computational Complexity; Divide and Conquer; Gaussian Processes; Global Optimization; Hyperparameter Tuning; Sequential Design
Application summary
The overarching goal of the research program is to extend Gaussian processes (GPs) to enable much more complex applications. First, new methodology will scale up GPs to enable large sample sizes and use adaptive sampling, for accurate statistical modelling of complex relationships arising cross a broad spectrum of scientific and engineering disciplines. Second, efficiencies in adaptive search methods using GPs will allow automatic tuning of computationally intensive machine/statistical learners. GPs have had profound impact on science and engineering, where they are used directly as machine/statistical learners. Complex computer codes of physical systems can be too slow for optimization, calibration of unknowns, sensitivity analysis, etc. GPs trained on limited computer model runs are used for these purposes as computationally fast surrogates for the particular scientific objective. It is well known, however, that the computational time to train a GP increases as the cube of the sample size. Thus, GPs are less attractive for sample sizes of a few thousand or more. Existing methods, mainly based on localized modelling or special fixed experimental designs, will be assessed to determine the domain of problems where they are effective. It is clear in advance, however, that new methods will be required for complex applications: those with moderate to high-dimensional input, nonlinear relationships, and/or high-order interaction effects. Only by adapting the experiment - taking further observations where the target function has special features - can a dense sampling of the input space be obtained where it matters. Divide and conquer methods are especially promising. How to divide high-dimensional space, how to choose sub-regions for data augmentation, and guidance on the number of new runs per iteration will be critical research questions here. GPs are also used indirectly in support of other machine-learning (ML) methods such as deep learning neural networks. Neural networks for image classification, for example, have "tuning" parameters that have to be set by the user, to determine the basic network architecture or regularization, for instance. Users tune these so-called hyperparameters by trying different values and attempting to minimize validation error in various ways. To obtain the validation error requires training the ML method, which is itself computationally very intensive. Hence, systematic methods known as Bayesian optimization train a GP to model the relationship between the hyperparameter settings and validation error, and hence adaptively optimize the error. The research program will continue work in my lab on "automatic ML", to minimize the number of tries of the expensive underlying ML algorithm. Advances here will likely have impact on other computationally challenging optimization problems where the objective is produced by an expensive algorithm.