Grant: Alliance Grants
Université de Montréal
$45,000.00
Towards Improved Understanding and Efficient Utilization of Depthwise Computation in Modern Neural Networks
FY 2024-25
Summary
Deep Neural Networks have shown remarkable performance in a wide array of tasks including image recognition, speech recognition, and text generation. They consist of a number of sequential modules typically jointly optimized for a specific task. The functional behavior of the individual modules that emerges in such systems is complex and difficult to understand or analyze, while at the same time yielding remarkable results. On the other hand, the computational and memory efficiency of deep networks is a large bottleneck to their widespread usability. In this project we propose to tackle these problems in a holistic manner. In the first step, we propose to develop and enhance analytical tools for comparing and analyzing intermediate representations of deep networks. Borrowing methods from kernel non-parametrics, our aim is to provide analytical tools to determine how representations evolve in depth, and what information is extracted at each layer. Ultimately we would like to use these tools to characterize an implicit objective function that emerges in depth. These aspects will be studied in both traditional feed-forward networks and modern networks with skip connections, as well as Transformer-based models utilizing self-attention layers. In the second step we will use these techniques to enhance local learning methods as part of principled local objective functions, targeted to allow increasing parallelism and saving memory when training large deep neural networks. Concurrently, we will investigate continuous-depth deep neural networks. Our work will utilize a variety of modern deep neural networks with a focus on the emerging Transformer architecture. Finally, we will apply the developed techniques in a broad set of application areas including natural language processing, computer vision, drug discovery, and recommender systems. Development of this will require interdisciplinary collective expertise in theoretical tools, model-parallel learning, and continuous depth networks. Over this three-year project the PI and co-investigators will train three PhD students and one MSc student across a wide range of industrially relevant skills, including deep learning and distributed learning.
Program
Alliance Grants
Recipient
Location
Quebec, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
577088-2022
Name nom
Wolf, Guy G
Department département
Arts et des sciences, faculté des
Organization id
63
Institution établissement
Université de Montréal
Province en
Québec
Province fr
Québec
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2022
Award amount
45000
Program id
ALLRP
Program name en
Alliance Grants
Program name fr
Subventions Alliance
Group en
RESEARCH PARTNERSHIPS
Group fr
PARTENARIATS DE RECHERCHE
Committee code
96
Committee name en
RPP Internal Decision Cttee
Committee name fr
Cté de décision interne - PPR
Area of application code
1200
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Advancement of knowledge
Area of application fr
Avancement des connaissances
Research subject code
2800
Research subject group en
Artificial intelligence
Research subject group fr
Intelligence artificielle
Research subject en
Artificial intelligence (computer vision, use2603)
Research subject fr
Intelligence artificielle (vision artificielle, utiliser 2603)
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Towards Improved Understanding and Efficient Utilization of Depthwise Computation in Modern Neural Networks
Keywords
computer vision; deep learning; graph neural networks; representation learning; self attention
Application summary
Deep Neural Networks have shown remarkable performance in a wide array of tasks including image recognition, speech recognition, and text generation. They consist of a number of sequential modules typically jointly optimized for a specific task. The functional behavior of the individual modules that emerges in such systems is complex and difficult to understand or analyze, while at the same time yielding remarkable results. On the other hand, the computational and memory efficiency of deep networks is a large bottleneck to their widespread usability. In this project we propose to tackle these problems in a holistic manner. In the first step, we propose to develop and enhance analytical tools for comparing and analyzing intermediate representations of deep networks. Borrowing methods from kernel non-parametrics, our aim is to provide analytical tools to determine how representations evolve in depth, and what information is extracted at each layer. Ultimately we would like to use these tools to characterize an implicit objective function that emerges in depth. These aspects will be studied in both traditional feed-forward networks and modern networks with skip connections, as well as Transformer-based models utilizing self-attention layers. In the second step we will use these techniques to enhance local learning methods as part of principled local objective functions, targeted to allow increasing parallelism and saving memory when training large deep neural networks. Concurrently, we will investigate continuous-depth deep neural networks. Our work will utilize a variety of modern deep neural networks with a focus on the emerging Transformer architecture. Finally, we will apply the developed techniques in a broad set of application areas including natural language processing, computer vision, drug discovery, and recommender systems. Development of this will require interdisciplinary collective expertise in theoretical tools, model-parallel learning, and continuous depth networks. Over this three-year project the PI and co-investigators will train three PhD students and one MSc student across a wide range of industrially relevant skills, including deep learning and distributed learning.