Canada Spends Logo

Explorez notre analyse du budget 2026

Grant: Discovery Grants Program - Individual

University of Toronto

$62,000.00

Evaluating and Improving Deep Neural Networks

EF 2022-23

Résumé
Deep neural networks have recently pushed forward the state-of-the-art in applications as diverse as image understanding, language understanding, genomics, computational chemistry, game playing, and robotics. Deep generative models in particular have seen rapid progress in the past few years, with networks able to produce plausible images or speech signals. As diverse as these application areas are, the challenges and frustrations facing the researchers and practitioners have much in common. Neural networks can take weeks to train on expensive Graphics Processing Unit (GPU) hardware. The algorithms have many more knobs which need to be tweaked, compared with the previous generation of machine learning algorithms. In the case of deep generative models, it can be hard to determine if the networks are learning to model the data or simply memorizing their training examples.Solutions to any of these issues would have enormous impact across a range of application areas, both in industry and in scientific research. I propose to tame the complexity of neural networks using the techniques of structured probabilistic modeling. Over the next five years, I will focus on two main threads: improving the optimization of neural networks, and evaluating deep generative models. Training neural networks can take weeks, even with modern GPU hardware. Previously, I introduced a method for deriving efficient second-order optimization algorithms from probabilistic models of the curvature of a cost function. This led to large speedups in training many types of neural nets. I plan to extend this technique to state-of-the-art architectures for image, video, and text understanding, and to scale up the technique to training on a cluster rather than an individual processor. The end result will be a general-purpose neural net training algorithm which is efficient and scalable and requires little hand-tweaking.The main obstacle to evaluating generative models is the intractability of computing the probability assigned to a configuration, coupled with the difficulty of determining how accurate one's estimates are. (Papers published on the topic tend to include caveats that the reported probabilities may be extremely inaccurate.) The difficulty of evaluation is considered one of the main factors holding back scientific progress on generative modeling. I aim to develop techniques for obtaining confidence intervals for the probabilities which are both tight and accurate, so that we can have confidence in our evaluation of generative models. This will enable rigorous empirical study of generative models, which in turn will allow us to improve them.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Ontario, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data

Tel que publié

Application id
RGPIN-2017-06050
Name nom
Grosse, Roger
Department département
Computer Science (St. George Campus) - Computer Science (St. George Campus)
Organization id
31
Institution établissement
University of Toronto
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2022
Competition year année de concours
2017
Award amount
62000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
801
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Computer software
Area of application fr
Logiciel
Research subject code
2800
Research subject group en
Artificial intelligence
Research subject group fr
Intelligence artificielle
Research subject en
Artificial intelligence (computer vision, use2603)
Research subject fr
Intelligence artificielle (vision artificielle, utiliser 2603)
Installment
inst:6;7
Partie
2
Nb partie
7
Application title
Evaluating and Improving Deep Neural Networks
Application summary
Deep neural networks have recently pushed forward the state-of-the-art in applications as diverse as image understanding, language understanding, genomics, computational chemistry, game playing, and robotics. Deep generative models in particular have seen rapid progress in the past few years, with networks able to produce plausible images or speech signals. As diverse as these application areas are, the challenges and frustrations facing the researchers and practitioners have much in common. Neural networks can take weeks to train on expensive Graphics Processing Unit (GPU) hardware. The algorithms have many more knobs which need to be tweaked, compared with the previous generation of machine learning algorithms. In the case of deep generative models, it can be hard to determine if the networks are learning to model the data or simply memorizing their training examples.Solutions to any of these issues would have enormous impact across a range of application areas, both in industry and in scientific research. I propose to tame the complexity of neural networks using the techniques of structured probabilistic modeling. Over the next five years, I will focus on two main threads: improving the optimization of neural networks, and evaluating deep generative models. Training neural networks can take weeks, even with modern GPU hardware. Previously, I introduced a method for deriving efficient second-order optimization algorithms from probabilistic models of the curvature of a cost function. This led to large speedups in training many types of neural nets. I plan to extend this technique to state-of-the-art architectures for image, video, and text understanding, and to scale up the technique to training on a cluster rather than an individual processor. The end result will be a general-purpose neural net training algorithm which is efficient and scalable and requires little hand-tweaking.The main obstacle to evaluating generative models is the intractability of computing the probability assigned to a configuration, coupled with the difficulty of determining how accurate one's estimates are. (Papers published on the topic tend to include caveats that the reported probabilities may be extremely inaccurate.) The difficulty of evaluation is considered one of the main factors holding back scientific progress on generative modeling. I aim to develop techniques for obtaining confidence intervals for the probabilities which are both tight and accurate, so that we can have confidence in our evaluation of generative models. This will enable rigorous empirical study of generative models, which in turn will allow us to improve them.