Canada Spends Logo

Explorez notre analyse du budget 2026

Grant: Discovery Grants Program - Individual

University of Waterloo

$55,000.00

Information Theoretic Coding for Deep Neural Networks: Frameworks, Theory, and Algorithms

EF 2024-25

Résumé
Deep neural networks (DNNs) are increasingly becoming ubiquitous in many applications including computer vision, speech recognition, and natural language processing. With more data collected in our digital society, DNNs will continue to be a major area of growth in engineering, and change how we live, work, and interact with each other and intelligent machines. Before DNNs can be widely deployed in many parts of our ubiquitous communications networks, however, several key challenges of DNNs have to be addressed. For example, take a look at DNNs for image classification. The first challenge lies in what type of raw data will be fed into DNNs. Indeed, in the context of ubiquitous communications networks which include the whole pipeline of data acquisition, data encoding (i.e., compression), data transmission, and data processing/utilization, the raw data fed into each DNN is not "raw"; instead, it is generally encoded/compressed in a lossy manner. How does lossy coding impact a DNN? According to the conventional wisdom, existing lossy codecs designed for human perception generally degrade the classification accuracy of the DNN. In one of our recent works, however, we showed experimentally the opposite---if one can choose intelligently which compressed version of the raw data is fed into the DNN, the classification accuracy of the DNN can actually be improved significantly while reducing dramatically the number of bits for transmission and storage. The question is, of course, how to encode raw data intelligently for DNNs. The second challenge is the vulnerability of DNNs to adversarial examples, maliciously modified inputs with imperceptible perturbation that lead DNNs to produce incorrect outputs. The existence and easy construction of adversarial examples pose significant security risks to DNNs, especially in safety critical applications. The third challenge lies in the huge number of model parameters in DNNs, which can be as high as a few billions. It is the huge number of model parameters that makes DNNs both computationally intensive and memory intensive, hindering the wide deployment of DNNs in resource limited devices. It also makes it difficult and costly (in terms of bandwidth) to transmit and update model parameters in distributed learning. Based on our early success, in this research program, we will investigate the challenges mentioned above systematically by introducing information theoretic ideas such as soft decision quantization into the domain of DNN, proposing new coding frameworks for DNNs, developing their respective theories, and designing new effective algorithms for new forms of compression for both DNNs and human, protecting DNNs against adversarial attacks, or jointly compressing and training DNN models. Our research results will significantly advance the fields of information theory, image coding, deep learning, and computer vision, and have great impacts on the related industries in Canada and beyond.
Programme
Discovery Grants Program - Individual
Emplacement
Ontario, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data

Tel que publié

Application id
RGPIN-2022-03526
Name nom
Yang, En-hui
Department département
Electrical and Computer Engineering
Organization id
33
Institution établissement
University of Waterloo
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2022
Award amount
55000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1510
Committee name en
Electrical and Computer Engineering
Committee name fr
Génie électrique et informatique
Area of application code
1207
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Information, computer and communication technologies
Area of application fr
Technologie de l'information, des ordinateurs et des réseaux de communications
Research subject code
2507
Research subject group en
Electrical and electronic engineering
Research subject group fr
Génie électrique et électronique
Research subject en
Communications systems
Research subject fr
Systèmes de télécommunications
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Information Theoretic Coding for Deep Neural Networks: Frameworks, Theory, and Algorithms
Keywords
information theory; deep learning; deep neural networks; lossy source coding; lossless coding; soft decision quantization; image coding; watermarking; adversarial attack detection; computer vision
Application summary
Deep neural networks (DNNs) are increasingly becoming ubiquitous in many applications including computer vision, speech recognition, and natural language processing. With more data collected in our digital society, DNNs will continue to be a major area of growth in engineering, and change how we live, work, and interact with each other and intelligent machines. Before DNNs can be widely deployed in many parts of our ubiquitous communications networks, however, several key challenges of DNNs have to be addressed. For example, take a look at DNNs for image classification. The first challenge lies in what type of raw data will be fed into DNNs. Indeed, in the context of ubiquitous communications networks which include the whole pipeline of data acquisition, data encoding (i.e., compression), data transmission, and data processing/utilization, the raw data fed into each DNN is not "raw"; instead, it is generally encoded/compressed in a lossy manner. How does lossy coding impact a DNN? According to the conventional wisdom, existing lossy codecs designed for human perception generally degrade the classification accuracy of the DNN. In one of our recent works, however, we showed experimentally the opposite---if one can choose intelligently which compressed version of the raw data is fed into the DNN, the classification accuracy of the DNN can actually be improved significantly while reducing dramatically the number of bits for transmission and storage. The question is, of course, how to encode raw data intelligently for DNNs. The second challenge is the vulnerability of DNNs to adversarial examples, maliciously modified inputs with imperceptible perturbation that lead DNNs to produce incorrect outputs. The existence and easy construction of adversarial examples pose significant security risks to DNNs, especially in safety critical applications. The third challenge lies in the huge number of model parameters in DNNs, which can be as high as a few billions. It is the huge number of model parameters that makes DNNs both computationally intensive and memory intensive, hindering the wide deployment of DNNs in resource limited devices. It also makes it difficult and costly (in terms of bandwidth) to transmit and update model parameters in distributed learning. Based on our early success, in this research program, we will investigate the challenges mentioned above systematically by introducing information theoretic ideas such as soft decision quantization into the domain of DNN, proposing new coding frameworks for DNNs, developing their respective theories, and designing new effective algorithms for new forms of compression for both DNNs and human, protecting DNNs against adversarial attacks, or jointly compressing and training DNN models. Our research results will significantly advance the fields of information theory, image coding, deep learning, and computer vision, and have great impacts on the related industries in Canada and beyond.