Canada Spends Logo

Explorez notre analyse du budget 2026

Grant: Discovery Grants Program - Individual

University of Calgary

$25,000.00

Training Unstructured Sparse Neural Networks

EF 2023-24

Résumé
The Increasing Cost of Training Deep Neural Networks Deep Neural Networks (DNNs) are behind the intelligence found in contemporary technology, enabling us to search our photos, ask questions of smart assistants or use machine translation to understand a foreign language. DNNs have a fundamental problem however: they are very expensive - in cost, energy usage, and time, both during training (learning a task) and inference (application). The state-of-the-art model for Natural Language Processing (NLP), GPT-3, has 175 billion parameters and is estimated to cost more than $4.6M USD to train. The current trend is that training DNNs will only get ever more expensive as the growth in the cost of training state-of-the-art DNN models has surpassed even the exponential growth in transistor technology that governs our computational capacity, i.e. Moore's Law. Proposed Program of Research Enable Sparse DNN Training DNNs are primarily expensive due to the empirically established, but poorly understood, requirement to over-parameterize DNNs models during training for good generalization (performance on unseen data). We know these models are over-parameterized because, after training, 80-95% of the learned weights (parameters) of DNN models can be removed (pruned) without significant loss of generalization [16]. Unstructured pruning, that is removing unnecessary individual weights from a DNN, can be highly effective at reducing the size and efficiency of pre-trained DNNs used in applications. However, attempting to train unstructured sparse DNN from random initialization - just as dense (standard) DNNs are trained - rarely achieves similar generalization as dense training empirically. For this reason, unstructured sparsity has not played a significant role in decreasing the cost of training DNNs as of yet. Developing effective approaches for efficient training would make DNN training cheaper, faster, more repeatable, and importantly more accessible to both researchers and new applications. Efficient Deep Learning and Society: Adversarial Robustness and Bias of Efficient DNNs Already DNNs used in industrial application are drastically different from those proposed in most academic papers, in using efficient DNNs methods (e.g. quantization, pruning, and distillation) and efficient DNN architectures - and yet, there is little work exploring the differences between the well-studied academic models, and industry-applied efficient DNNs beyond simply generalization performance and efficiency. Understanding the differences between efficient DNNs and the typical DNNs seen in academic research is becoming imperative given the increasing reliance of our society on this technology. With an ever larger potential to affect our society, identifying and resolving any issues with robustness and bias specific to efficient DNNs is an increasingly important area of research, and one relevant to a variety of potential industrial partners using DNNs in real-world applications.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Alberta, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data

Tel que publié

Application id
RGPIN-2022-03120
Name nom
Ioannou, Yani
Department département
Department of Electrical and Software Engineering
Organization id
11
Institution établissement
University of Calgary
Province en
Alberta
Province fr
Alberta
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2023
Competition year année de concours
2022
Award amount
25000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
802
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Information systems and technology
Area of application fr
Systèmes d'information et technologie de l'information
Research subject code
2603
Research subject group en
Robotics
Research subject group fr
Computer vision
Research subject en
Robotique
Research subject fr
Vision artificielle
Application title
Training Unstructured Sparse Neural Networks
Keyword
Computer Vision; Deep Learning; Efficient Deep Learning; Machine Learning; Sparse Neural Networks
Application summary
The Increasing Cost of Training Deep Neural Networks Deep Neural Networks (DNNs) are behind the intelligence found in contemporary technology, enabling us to search our photos, ask questions of smart assistants or use machine translation to understand a foreign language. DNNs have a fundamental problem however: they are very expensive - in cost, energy usage, and time, both during training (learning a task) and inference (application). The state-of-the-art model for Natural Language Processing (NLP), GPT-3, has 175 billion parameters and is estimated to cost more than $4.6M USD to train. The current trend is that training DNNs will only get ever more expensive as the growth in the cost of training state-of-the-art DNN models has surpassed even the exponential growth in transistor technology that governs our computational capacity, i.e. Moore's Law. Proposed Program of Research Enable Sparse DNN Training DNNs are primarily expensive due to the empirically established, but poorly understood, requirement to over-parameterize DNNs models during training for good generalization (performance on unseen data). We know these models are over-parameterized because, after training, 80-95% of the learned weights (parameters) of DNN models can be removed (pruned) without significant loss of generalization [16]. Unstructured pruning, that is removing unnecessary individual weights from a DNN, can be highly effective at reducing the size and efficiency of pre-trained DNNs used in applications. However, attempting to train unstructured sparse DNN from random initialization - just as dense (standard) DNNs are trained - rarely achieves similar generalization as dense training empirically. For this reason, unstructured sparsity has not played a significant role in decreasing the cost of training DNNs as of yet. Developing effective approaches for efficient training would make DNN training cheaper, faster, more repeatable, and importantly more accessible to both researchers and new applications. Efficient Deep Learning and Society: Adversarial Robustness and Bias of Efficient DNNs Already DNNs used in industrial application are drastically different from those proposed in most academic papers, in using efficient DNNs methods (e.g. quantization, pruning, and distillation) and efficient DNN architectures - and yet, there is little work exploring the differences between the well-studied academic models, and industry-applied efficient DNNs beyond simply generalization performance and efficiency. Understanding the differences between efficient DNNs and the typical DNNs seen in academic research is becoming imperative given the increasing reliance of our society on this technology. With an ever larger potential to affect our society, identifying and resolving any issues with robustness and bias specific to efficient DNNs is an increasingly important area of research, and one relevant to a variety of potential industrial partners using DNNs in real-world applications.