Canada Spends Logo

Explorez notre analyse du budget 2026

Grant: Discovery Grants Program - Individual

University of Toronto

$110,000.00

A Fresh Look at our Understanding of Machine Learning

EF 2024-25

Résumé
Machine Learning and other "Artificial Intelligence" technologies are rapidly being deployed across industry, science, and government. Much of this progress is due to the application of deep neural networks. Despite empirical success stories, our theoretical understanding of neural networks is still very limited, even though neural networks have been studied for decades. One reason is that modern neural networks are much larger and deeper. Another is that the way we train neural networks on data has evolved. While we are still relatively in the dark, a number of empirical phenomena can serve as beacons. One such phenomenon is interpolation, where neural networks can be trained to perform perfectly on training data, even if the training data in corrupted by noise. Remarkably, neural network classifiers do not seem to suffer from overfitting in this regime. Devising an explanation for this phenomenon is a major open problem. Another phenomenon relates to the role of data in generalization performance. Why does the standard learning algorithm, stochastic gradient descent, learn an accurate classifier on real data, when the same algorithm, running on the same architecture, overfits badly on corrupted data? What property of the data explains this? And can we predict it? As machine learning expands into sensitive application areas such as healthcare, transportation, and policy making, it is imperative that we develop better understanding. The central goal of my research program is to bridge the gap between empirical and theoretical performance, building on the progress made within the statistical learning community, while scrutinizing those aspects of the foundation that may divide theory and practice. In addition to studying this gap theoretically, this research program aims to use empirical methods to understand the limitations of existing theory and also inspire and evaluate new theory.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Ontario, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data

Tel que publié

Application id
RGPIN-2020-06641
Name nom
Roy, Daniel
Department département
Computer and Mathematical Sciences (Scarborough) - Computer and Mathematical Sciences (Scarborough)
Organization id
31
Institution établissement
University of Toronto
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2020
Award amount
110000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
1200
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Advancement of knowledge
Area of application fr
Avancement des connaissances
Research subject code
2805
Research subject group en
Artificial intelligence
Research subject group fr
Intelligence artificielle
Research subject en
Learning and inference theories
Research subject fr
Théories de l'apprentissage et l'inférence
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
A Fresh Look at our Understanding of Machine Learning
Keywords
machine learning; statistical learning theory; pac-bayes
Application summary
Machine Learning and other "Artificial Intelligence" technologies are rapidly being deployed across industry, science, and government. Much of this progress is due to the application of deep neural networks. Despite empirical success stories, our theoretical understanding of neural networks is still very limited, even though neural networks have been studied for decades. One reason is that modern neural networks are much larger and deeper. Another is that the way we train neural networks on data has evolved. While we are still relatively in the dark, a number of empirical phenomena can serve as beacons. One such phenomenon is interpolation, where neural networks can be trained to perform perfectly on training data, even if the training data in corrupted by noise. Remarkably, neural network classifiers do not seem to suffer from overfitting in this regime. Devising an explanation for this phenomenon is a major open problem. Another phenomenon relates to the role of data in generalization performance. Why does the standard learning algorithm, stochastic gradient descent, learn an accurate classifier on real data, when the same algorithm, running on the same architecture, overfits badly on corrupted data? What property of the data explains this? And can we predict it? As machine learning expands into sensitive application areas such as healthcare, transportation, and policy making, it is imperative that we develop better understanding. The central goal of my research program is to bridge the gap between empirical and theoretical performance, building on the progress made within the statistical learning community, while scrutinizing those aspects of the foundation that may divide theory and practice. In addition to studying this gap theoretically, this research program aims to use empirical methods to understand the limitations of existing theory and also inspire and evaluate new theory.