Canada Spends Logo

Explore our Budget 2026 analysis

Grant: Discovery Grants Program - Individual

McGill University

$27,000.00

Principled approaches to deep learning: generalization under distribution shift and predictive uncertainty

FY 2022-23

Summary
Deep Learning (DL) addresses problems which were previously not possible using traditional Machine Learning (ML) methods. However, unlike ML, which has a solid theoretical foundation, DL has so far relied engineering practices. In particular, while very successful in computer applications, DL has so far been limited in its applications to real world problems, for example autonomous vehicles. In order to continue to broaden the applications, deep learning requires a theoretical foundation, in the form of error estimates, which provide control over the accuracy of models on new inputs. Thirty years ago, Machine Learning (ML) was in a situation similar to the one currently faced by DL: the methods were far ahead of the theory. In a short time, ML theory was able to solve theoretical problems, allowing ML models to be safely deployed in a wide range of applications. The proposed program will do for DL what was done thirty years ago for ML. Error estimation can be addressed in two settings. The first is generalization error bounds, which apply before the inputs are seen by the model. The second is predictive uncertainty, which applies after the inputs are seen by the model (but before a decision is made). For example, suppose we have a decision problem where we can only act if the probability of error is less than 1%, but our model error is 5%, on average. Predictive uncertainty can tell us, on a case by case basis, on which inputs the model accuracy is high enough to act. For a second example, we train computer vision models using databases of images, but we want to deploy them on real world images, which are statistically different from the training set. We propose to extend data transformation techniques, and methods for measuring dataset differences, in order to better estimate the model error on real world images. This proposal will address the problem of applying deep learning in a broader setting by building a mathematical theory for DL out-of-distribution (OOD) generalization. It will 1.Formulate suitable definitions and assumptions which allow us to state the deep learning out-of- distribution generalization problem in mathematical terms. 2.Prove a theorem estimating (with high probability) the OOD generalization gap, in terms of relevant problem inputs. 3.Determine relevant problem inputs through mathematical modelling. Determining the relevant hypotheses requires applying the scientific method, which, in this case corresponds to computer experiments designed to probe the generalization behaviour of deep neural networks. Stating the definitions and assumptions precisely involves mathematical modelling. Proving the theorem described above involves mathematical analysis.
Program
Discovery Grants Program - Individual
Location
Quebec, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data

As published

Application id
RGPIN-2022-03609
Name nom
Oberman, Adam
Department département
Mathematics and Statistics - Mathematics and Statistics
Organization id
61
Institution établissement
McGill University
Province en
Québec
Province fr
Québec
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2022
Competition year année de concours
2022
Award amount
27000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1508
Committee name en
Mathematics and Statistics
Committee name fr
Mathématiques et statistique
Area of application code
1200
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Advancement of knowledge
Area of application fr
Avancement des connaissances
Research subject code
2600
Research subject group en
Robotics
Research subject group fr
Robotique
Research subject en
Robotics
Research subject fr
Robotique
Installment
inst:1
Partie
1
Nb partie
5
Application title
Principled approaches to deep learning: generalization under distribution shift and predictive uncertainty
Application summary
Deep Learning (DL) addresses problems which were previously not possible using traditional Machine Learning (ML) methods. However, unlike ML, which has a solid theoretical foundation, DL has so far relied engineering practices. In particular, while very successful in computer applications, DL has so far been limited in its applications to real world problems, for example autonomous vehicles. In order to continue to broaden the applications, deep learning requires a theoretical foundation, in the form of error estimates, which provide control over the accuracy of models on new inputs. Thirty years ago, Machine Learning (ML) was in a situation similar to the one currently faced by DL: the methods were far ahead of the theory. In a short time, ML theory was able to solve theoretical problems, allowing ML models to be safely deployed in a wide range of applications. The proposed program will do for DL what was done thirty years ago for ML. Error estimation can be addressed in two settings. The first is generalization error bounds, which apply before the inputs are seen by the model. The second is predictive uncertainty, which applies after the inputs are seen by the model (but before a decision is made). For example, suppose we have a decision problem where we can only act if the probability of error is less than 1%, but our model error is 5%, on average. Predictive uncertainty can tell us, on a case by case basis, on which inputs the model accuracy is high enough to act. For a second example, we train computer vision models using databases of images, but we want to deploy them on real world images, which are statistically different from the training set. We propose to extend data transformation techniques, and methods for measuring dataset differences, in order to better estimate the model error on real world images. This proposal will address the problem of applying deep learning in a broader setting by building a mathematical theory for DL out-of-distribution (OOD) generalization. It will 1.Formulate suitable definitions and assumptions which allow us to state the deep learning out-of- distribution generalization problem in mathematical terms. 2.Prove a theorem estimating (with high probability) the OOD generalization gap, in terms of relevant problem inputs. 3.Determine relevant problem inputs through mathematical modelling. Determining the relevant hypotheses requires applying the scientific method, which, in this case corresponds to computer experiments designed to probe the generalization behaviour of deep neural networks. Stating the definitions and assumptions precisely involves mathematical modelling. Proving the theorem described above involves mathematical analysis.