Canada Spends Logo

Explorez notre analyse du budget 2026

Grant: Discovery Grants Program - Individual

University of Toronto

$34,000.00

Inductive Biases in Deep Learning

EF 2021-22

Résumé
This NSERC discovery grant proposal focuses on studying the underlying assumptions and properties of modern deep learning systems, that persist across machine learning tasks, benchmark datasets, state-of-the-art architectures, and hyperparameters, in order to find solutions to the limitations that currently exist. Deep neural networks have millions of parameters, learned on a set of training data, consisting of inputs to the system and desired outputs. Often during training, the network learns to memorize the output of the training data almost perfectly. Such memorization might be anticipated to cause the network to perform poorly on unseen data. Yet, in practice, this is often not the case. There exist implicit underlying assumptions made by the learning algorithm, causing it to prefer models that perform well over others that don't. In machine learning, these predispositions are often referred to as inductive biases. It might be further expected that the parameters found through training would vary wildly, from instance to instance, dataset to dataset, and architecture to architecture, having no underlying invariant structure. Yet, my ongoing work shows that patterns can be found in the parameters of the model, as well as in other statistics of the network. These patterns are clues for the inductive biases, steering the training of the network towards choosing models that perform well. The proposed research has six objectives, aiming to explore these inductive biases, and to leverage them towards overcoming fundamental limitations of deep learning: 1) Increasing sample-size efficiency; 2) Boosting robustness to adversaries; 3) Achieving faster first-order optimization; 4) Achieving faster second-order optimization; 5) Enhancing robustness to shifts in distribution; and 6) Illuminating the opaque box. Current exploration of inductive biases is based on studying simplistic mathematical models. Phenomena observed in such settings could not be expected to persist in state-of-the-art architectures trained on real data, such as the ones being used in countless applications in the natural sciences and engineering. As such, the long-term objective of this research is to use rigorous empirical measurements in realistic settings to find novel inductive biases, and to encapsulate the underlying assumptions and properties into a unifying theory of deep learning. The expected outcome of the proposed research is to engineer better solutions to eliminate the existing limitations of current deep learning systems that are impacting Canadian's everyday lives through applications such as language translation, digital assistants, self-driving cars, healthcare, and fraud detection.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Ontario, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data

Tel que publié

Application id
RGPIN-2021-02527
Name nom
Papyan, Vardan
Department département
Mathematics
Organization id
31
Institution établissement
University of Toronto
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2021
Competition year année de concours
2021
Award amount
34000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
Discovery grants programs
Group fr
Programmes de subventions à la découverte
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
1207
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Information, computer and communication technologies
Area of application fr
Technologie de l'information, des ordinateurs et des réseaux de communications
Research subject code
2805
Research subject group en
Artificial intelligence
Research subject group fr
Intelligence artificielle
Research subject en
Learning and inference theories
Research subject fr
Théories de l'apprentissage et l'inférence
Installment
inst:1
Partie
1
Nb partie
5
Application title
Inductive Biases in Deep Learning
Keyword
NSERC
Application summary
This NSERC discovery grant proposal focuses on studying the underlying assumptions and properties of modern deep learning systems, that persist across machine learning tasks, benchmark datasets, state-of-the-art architectures, and hyperparameters, in order to find solutions to the limitations that currently exist. Deep neural networks have millions of parameters, learned on a set of training data, consisting of inputs to the system and desired outputs. Often during training, the network learns to memorize the output of the training data almost perfectly. Such memorization might be anticipated to cause the network to perform poorly on unseen data. Yet, in practice, this is often not the case. There exist implicit underlying assumptions made by the learning algorithm, causing it to prefer models that perform well over others that don't. In machine learning, these predispositions are often referred to as inductive biases. It might be further expected that the parameters found through training would vary wildly, from instance to instance, dataset to dataset, and architecture to architecture, having no underlying invariant structure. Yet, my ongoing work shows that patterns can be found in the parameters of the model, as well as in other statistics of the network. These patterns are clues for the inductive biases, steering the training of the network towards choosing models that perform well. The proposed research has six objectives, aiming to explore these inductive biases, and to leverage them towards overcoming fundamental limitations of deep learning: 1) Increasing sample-size efficiency; 2) Boosting robustness to adversaries; 3) Achieving faster first-order optimization; 4) Achieving faster second-order optimization; 5) Enhancing robustness to shifts in distribution; and 6) Illuminating the opaque box. Current exploration of inductive biases is based on studying simplistic mathematical models. Phenomena observed in such settings could not be expected to persist in state-of-the-art architectures trained on real data, such as the ones being used in countless applications in the natural sciences and engineering. As such, the long-term objective of this research is to use rigorous empirical measurements in realistic settings to find novel inductive biases, and to encapsulate the underlying assumptions and properties into a unifying theory of deep learning. The expected outcome of the proposed research is to engineer better solutions to eliminate the existing limitations of current deep learning systems that are impacting Canadian's everyday lives through applications such as language translation, digital assistants, self-driving cars, healthcare, and fraud detection.