Grant: Discovery Grants Program - Individual
Dalhousie University
$29,000.00
Fine-grained Interpretation of Deep Neural Network Models of NLP
EF 2022-23
Résumé
Our ability to interpret Deep neural networks (DNNs) is an essential step towards explainable AI, to ensure fairness of models, and in achieving models that learn better generalization. In this proposal, we work towards increasing the transparency of DNN models by providing fine-grained interpretation of their representation. Fine-grained interpretation targets how certain knowledge is encoded in the network. It answers questions such as, what is the role of neurons in the network? What task and language knowledge is learned in neurons?, Knowing the role of neurons and their importance in the network paves the way towards transparent, fair, explainable and efficient models. In this proposal, we will make multiple contributions to fine-grained interpretation of DNNs. More specifically, we will propose datasets and algorithms for interpretation. We will work towards causal interpretation that explains the prediction of a model. Moreover, we will develop applications that are enabled by neuron analysis. The specific projects are summarized below: Concept Dataset: Current work in DNN interpretation probes whether human-defined linguistic concepts such as noun, verb, etc. are learned in a model. However, they do not analyze what other latent concepts a model has learned about the task and the language. This project aims at preparing a model-centric concept dataset. We will identify group of words in high dimensional space using their contextualized representation and will manually annotate them under certain criteria. The resulting dataset is a novel multi-facet data and will enable model-centered interpretation. Interpretation Algorithms: Neurons are multivariate in nature and models exhibit redundancy due to training and architectural choices. How to identify neurons that redundantly learn a concept? How to find a group of neurons that work together to represent a concept? Common approaches to neuron analysis largely ignore these questions or require careful evaluation to prove their efficacy. This project aims at improving methods of neuron analysis. We will explore various multivariate feature selection methods such as (sparse) group lasso to analyze neurons. Causal Interpretation: The importance of a concept for a class or for a specific prediction is essential for explainable AI. This project targets concept-based causal interpretation of prediction. We will use gradient and perturbation-based attribution algorithms to identify salient neurons for a prediction and will connect it with neuron analysis to generate concept-based explanations of predictions. Applications: In this project, we will explore applications of neuron analysis, more specifically controlling model's behavior and domain adaptation. The idea is to identify neurons with respect to concepts of interest e.g. gender or domain and develop methods to manipulate those neurons at test time in order to control the behavior of the model in a desirable way.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Nova Scotia, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data
Tel que publié
Application id
RGPIN-2022-03943
Name nom
Sajjad, Hassan
Department département
Computer Science, Faculty of - Computer Science, Faculty of
Organization id
79
Institution établissement
Dalhousie University
Province en
Nova Scotia
Province fr
Nouvelle-Écosse
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2022
Competition year année de concours
2022
Award amount
29000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
801
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Computer software
Area of application fr
Logiciel
Research subject code
2700
Research subject group en
Information technology
Research subject group fr
Technologie de l'information
Research subject en
Information technology
Research subject fr
Technologie de l'information
Installment
inst:1
Partie
1
Nb partie
5
Application title
Fine-grained Interpretation of Deep Neural Network Models of NLP
Application summary
Our ability to interpret Deep neural networks (DNNs) is an essential step towards explainable AI, to ensure fairness of models, and in achieving models that learn better generalization. In this proposal, we work towards increasing the transparency of DNN models by providing fine-grained interpretation of their representation. Fine-grained interpretation targets how certain knowledge is encoded in the network. It answers questions such as, what is the role of neurons in the network? What task and language knowledge is learned in neurons?, Knowing the role of neurons and their importance in the network paves the way towards transparent, fair, explainable and efficient models. In this proposal, we will make multiple contributions to fine-grained interpretation of DNNs. More specifically, we will propose datasets and algorithms for interpretation. We will work towards causal interpretation that explains the prediction of a model. Moreover, we will develop applications that are enabled by neuron analysis. The specific projects are summarized below: Concept Dataset: Current work in DNN interpretation probes whether human-defined linguistic concepts such as noun, verb, etc. are learned in a model. However, they do not analyze what other latent concepts a model has learned about the task and the language. This project aims at preparing a model-centric concept dataset. We will identify group of words in high dimensional space using their contextualized representation and will manually annotate them under certain criteria. The resulting dataset is a novel multi-facet data and will enable model-centered interpretation. Interpretation Algorithms: Neurons are multivariate in nature and models exhibit redundancy due to training and architectural choices. How to identify neurons that redundantly learn a concept? How to find a group of neurons that work together to represent a concept? Common approaches to neuron analysis largely ignore these questions or require careful evaluation to prove their efficacy. This project aims at improving methods of neuron analysis. We will explore various multivariate feature selection methods such as (sparse) group lasso to analyze neurons. Causal Interpretation: The importance of a concept for a class or for a specific prediction is essential for explainable AI. This project targets concept-based causal interpretation of prediction. We will use gradient and perturbation-based attribution algorithms to identify salient neurons for a prediction and will connect it with neuron analysis to generate concept-based explanations of predictions. Applications: In this project, we will explore applications of neuron analysis, more specifically controlling model's behavior and domain adaptation. The idea is to identify neurons with respect to concepts of interest e.g. gender or domain and develop methods to manipulate those neurons at test time in order to control the behavior of the model in a desirable way.