Grant: Discovery Grants Program - Individual
McGill University
$29,000.00
Mathematical Principles for Neural Network Design
EF 2022-23
Résumé
Even as neural networks have shown their power in solving complex problems, their behavior remains poorly understood. Neural networks are now used for many purposes, from autonomous driving to medical image analysis. In each of these situations, a different set of properties is needed, such as good generalization to novel data or robustness to noise. Currently, there is no principled way to design neural networks to possess the particular set of characteristics needed for the task at hand. This lack of theoretical grounding means that instead of optimal algorithms designed from rigorous understanding, successes are incremental and arise from trial-and-error experimentation, while failures can be catastrophic and come as a surprise. The long-term goal of this research program is to gain a formal mathematical understanding of how design choices in neural networks affect their performance, and to use these theoretical results to derive actionable insights for practitioners. Our research has the following short-term objectives. Objective 1: Quantifying inductive biases The structure of a neural network and its optimization process determine the sets of functions that the network can express and learn, essentially encoding an inductive bias towards certain functions and away from others. In this objective, we will derive theoretical results on how the design of a neural network influences this inductive bias. Objective 2: Matching algorithms to data This objective will consider how the inductive biases of different neural networks can be leveraged to improve deep learning algorithms. We will derive methods for identifying the inductive biases that are needed to solve a particular task, and will design learning methods that give a high degree of control over which functions a neural network will learn. Objective 3: Improving security We will use our mathematical understanding of inductive biases to improve the security of deep learning algorithms. We will show when it is possible to extract information about the parameters of a neural network from the function it computes, as well as how to guard against this. Our work will protect the privacy of neural networks and the data used to train them, and will prevent adversarial attacks. Overall, this research will provide much-needed tools for the principled design of neural networks, allowing for significant increases in performance and reliability from algorithms that are increasingly essential across society. This will have an impact on fields from robotics to energy. Understanding the mathematical principles behind deep learning innovation will also help maintain Canada's preeminent position in AI. Our work will train HQP to be leading innovators in the intersection of deep learning theory and engineering, a synergistic combination of skills that is much in demand within both academia and industry.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Quebec, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data
Tel que publié
Application id
RGPIN-2021-03864
Name nom
Rolnick, David
Department département
Computer Science, School of - Computer Science, School of
Organization id
61
Institution établissement
McGill University
Province en
Québec
Province fr
Québec
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2022
Competition year année de concours
2021
Award amount
29000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
1207
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Information, computer and communication technologies
Area of application fr
Technologie de l'information, des ordinateurs et des réseaux de communications
Research subject code
2800
Research subject group en
Artificial intelligence
Research subject group fr
Intelligence artificielle
Research subject en
Artificial intelligence (computer vision, use2603)
Research subject fr
Intelligence artificielle (vision artificielle, utiliser 2603)
Installment
inst:2
Partie
1
Nb partie
5
Application title
Mathematical Principles for Neural Network Design
Application summary
Even as neural networks have shown their power in solving complex problems, their behavior remains poorly understood. Neural networks are now used for many purposes, from autonomous driving to medical image analysis. In each of these situations, a different set of properties is needed, such as good generalization to novel data or robustness to noise. Currently, there is no principled way to design neural networks to possess the particular set of characteristics needed for the task at hand. This lack of theoretical grounding means that instead of optimal algorithms designed from rigorous understanding, successes are incremental and arise from trial-and-error experimentation, while failures can be catastrophic and come as a surprise. The long-term goal of this research program is to gain a formal mathematical understanding of how design choices in neural networks affect their performance, and to use these theoretical results to derive actionable insights for practitioners. Our research has the following short-term objectives. Objective 1: Quantifying inductive biases The structure of a neural network and its optimization process determine the sets of functions that the network can express and learn, essentially encoding an inductive bias towards certain functions and away from others. In this objective, we will derive theoretical results on how the design of a neural network influences this inductive bias. Objective 2: Matching algorithms to data This objective will consider how the inductive biases of different neural networks can be leveraged to improve deep learning algorithms. We will derive methods for identifying the inductive biases that are needed to solve a particular task, and will design learning methods that give a high degree of control over which functions a neural network will learn. Objective 3: Improving security We will use our mathematical understanding of inductive biases to improve the security of deep learning algorithms. We will show when it is possible to extract information about the parameters of a neural network from the function it computes, as well as how to guard against this. Our work will protect the privacy of neural networks and the data used to train them, and will prevent adversarial attacks. Overall, this research will provide much-needed tools for the principled design of neural networks, allowing for significant increases in performance and reliability from algorithms that are increasingly essential across society. This will have an impact on fields from robotics to energy. Understanding the mathematical principles behind deep learning innovation will also help maintain Canada's preeminent position in AI. Our work will train HQP to be leading innovators in the intersection of deep learning theory and engineering, a synergistic combination of skills that is much in demand within both academia and industry.