Canada Spends Logo

Explore our Budget 2026 analysis

Grant: Discovery Grants Program - Individual

University of Waterloo

$55,000.00

Information Theoretic Coding for Deep Neural Networks: Frameworks, Theory, and Algorithms

FY 2023-24

Summary
Deep neural networks (DNNs) are increasingly becoming ubiquitous in many applications including computer vision, speech recognition, and natural language processing. With more data collected in our digital society, DNNs will continue to be a major area of growth in engineering, and change how we live, work, and interact with each other and intelligent machines. Before DNNs can be widely deployed in many parts of our ubiquitous communications networks, however, several key challenges of DNNs have to be addressed. For example, take a look at DNNs for image classification. The first challenge lies in what type of raw data will be fed into DNNs. Indeed, in the context of ubiquitous communications networks which include the whole pipeline of data acquisition, data encoding (i.e., compression), data transmission, and data processing/utilization, the raw data fed into each DNN is not "raw"; instead, it is generally encoded/compressed in a lossy manner. How does lossy coding impact a DNN? According to the conventional wisdom, existing lossy codecs designed for human perception generally degrade the classification accuracy of the DNN. In one of our recent works, however, we showed experimentally the opposite---if one can choose intelligently which compressed version of the raw data is fed into the DNN, the classification accuracy of the DNN can actually be improved significantly while reducing dramatically the number of bits for transmission and storage. The question is, of course, how to encode raw data intelligently for DNNs. The second challenge is the vulnerability of DNNs to adversarial examples, maliciously modified inputs with imperceptible perturbation that lead DNNs to produce incorrect outputs. The existence and easy construction of adversarial examples pose significant security risks to DNNs, especially in safety critical applications. The third challenge lies in the huge number of model parameters in DNNs, which can be as high as a few billions. It is the huge number of model parameters that makes DNNs both computationally intensive and memory intensive, hindering the wide deployment of DNNs in resource limited devices. It also makes it difficult and costly (in terms of bandwidth) to transmit and update model parameters in distributed learning. Based on our early success, in this research program, we will investigate the challenges mentioned above systematically by introducing information theoretic ideas such as soft decision quantization into the domain of DNN, proposing new coding frameworks for DNNs, developing their respective theories, and designing new effective algorithms for new forms of compression for both DNNs and human, protecting DNNs against adversarial attacks, or jointly compressing and training DNN models. Our research results will significantly advance the fields of information theory, image coding, deep learning, and computer vision, and have great impacts on the related industries in Canada and beyond.
Program
Discovery Grants Program - Individual
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data

As published

Application id
RGPIN-2022-03526
Name nom
Yang, En-hui
Department département
Electrical and Computer Engineering
Organization id
33
Institution établissement
University of Waterloo
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2023
Competition year année de concours
2022
Award amount
55000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1510
Committee name en
Electrical and Computer Engineering
Committee name fr
Génie électrique et informatique
Area of application code
1207
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Information, computer and communication technologies
Area of application fr
Technologie de l'information, des ordinateurs et des réseaux de communications
Research subject code
2718
Research subject group en
Information technology
Research subject group fr
Communication and information theory
Research subject en
Technologie de l'information
Research subject fr
Théorie des télécommunications et de l'information
Application title
Information Theoretic Coding for Deep Neural Networks: Frameworks, Theory, and Algorithms
Keyword
Adversarial Attack Detection; Computer Vision; Deep Learning; Deep Neural Networks; Image Coding; Information Theory; Lossless Coding; Lossy Source Coding; Soft Decision Quantization; Watermarking
Application summary
Deep neural networks (DNNs) are increasingly becoming ubiquitous in many applications including computer vision, speech recognition, and natural language processing. With more data collected in our digital society, DNNs will continue to be a major area of growth in engineering, and change how we live, work, and interact with each other and intelligent machines. Before DNNs can be widely deployed in many parts of our ubiquitous communications networks, however, several key challenges of DNNs have to be addressed. For example, take a look at DNNs for image classification. The first challenge lies in what type of raw data will be fed into DNNs. Indeed, in the context of ubiquitous communications networks which include the whole pipeline of data acquisition, data encoding (i.e., compression), data transmission, and data processing/utilization, the raw data fed into each DNN is not "raw"; instead, it is generally encoded/compressed in a lossy manner. How does lossy coding impact a DNN? According to the conventional wisdom, existing lossy codecs designed for human perception generally degrade the classification accuracy of the DNN. In one of our recent works, however, we showed experimentally the opposite---if one can choose intelligently which compressed version of the raw data is fed into the DNN, the classification accuracy of the DNN can actually be improved significantly while reducing dramatically the number of bits for transmission and storage. The question is, of course, how to encode raw data intelligently for DNNs. The second challenge is the vulnerability of DNNs to adversarial examples, maliciously modified inputs with imperceptible perturbation that lead DNNs to produce incorrect outputs. The existence and easy construction of adversarial examples pose significant security risks to DNNs, especially in safety critical applications. The third challenge lies in the huge number of model parameters in DNNs, which can be as high as a few billions. It is the huge number of model parameters that makes DNNs both computationally intensive and memory intensive, hindering the wide deployment of DNNs in resource limited devices. It also makes it difficult and costly (in terms of bandwidth) to transmit and update model parameters in distributed learning. Based on our early success, in this research program, we will investigate the challenges mentioned above systematically by introducing information theoretic ideas such as soft decision quantization into the domain of DNN, proposing new coding frameworks for DNNs, developing their respective theories, and designing new effective algorithms for new forms of compression for both DNNs and human, protecting DNNs against adversarial attacks, or jointly compressing and training DNN models. Our research results will significantly advance the fields of information theory, image coding, deep learning, and computer vision, and have great impacts on the related industries in Canada and beyond.