Canada Spends Logo

Explorez notre analyse du budget 2026

Grant: Discovery Grants Program - Individual

University of Waterloo

$41,000.00

Building Semiparametric Models to Decouple Knowledge from Computation

EF 2023-24

Résumé
Recently, natural language processing models have achieved significant advances in various applications like question answering, fact verification, machine translation, etc. The progress of these models is mainly attributed to the pre-trained large language models like BERT, T5, GPT-3, etc. These models are designed in a fully-parametric fashion to encode all of the desired knowledge in the weights of a large neural network. Such implicit encoding of knowledge makes it hard to interpret whether the model is grounded on certain verifiable evidence. On the other hand, these fully-parametric models need a massive amount of parameters to encode an infinite amount of real-world knowledge. An alternative approach is to modularize the NLP models into two modules, where the memory module is used to store explicit facts and knowledge in a non-parametric form, and the neural computation module can ground on the retrieved memory to make predictions. By decoupling knowledge and computation, the semiparametric models can become more interpretable by demonstrating the evidence it retrieves from the memory. On the other hand, the model's trainable parameters can be reduced significantly to lower computation costs. In this long-term program, I plan to explore several directions related to semiparametric models. Firstly, I will develop semiparametric models that can robustly adapt to memory from different domains while maintaining consistent performance. More specifically, we hope the model can adapt to knowledge in different domains like science, medicine, news, etc without expensive re-training. Secondly, I plan to develop models that can better adapt to time changes in world facts. By updating the memory regularly with up-to-date knowledge, we hope the model can quickly capture the changes and make timely predictions. Thirdly, I plan to explore the possibility of semiparametric models to address broader AI tasks in other areas like biomedical, finance, vision, etc. Finally, I would like to investigate how to build more responsible AI systems with semiparametric models. Specifically, by making AI models more grounded on knowledge retrieval, we believe the model can make become fair across different demographic groups. Through this multi-year program, I aim to deepen our understanding of semiparametric models to understand their limits. Ultimately, we hope to build next-generation foundation models that can address a wide range of AI tasks with better trustworthiness and fairness. In turn, this will necessitate new highly qualified personnel trained in NLP at virtually every company that deals with text data roles -- roles my students will be prepared to fill.
Programme
Discovery Grants Program - Individual
Emplacement
Ontario, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data

Tel que publié

Application id
RGPIN-2023-03295
Name nom
Chen, Wenhu
Department département
Computer Science
Organization id
33
Institution établissement
University of Waterloo
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2023
Competition year année de concours
2023
Award amount
41000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
801
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Computer software
Area of application fr
Logiciel
Research subject code
2803
Research subject group en
Artificial intelligence
Research subject group fr
Knowledge representation
Research subject en
Intelligence artificielle
Research subject fr
Répresentation des connaissances
Application title
Building Semiparametric Models to Decouple Knowledge from Computation
Keyword
Deep Learning; Information Retrieval; Knowledge Representation and Reasoning; Multimodal AI; Natural Language Processing
Application summary
Recently, natural language processing models have achieved significant advances in various applications like question answering, fact verification, machine translation, etc. The progress of these models is mainly attributed to the pre-trained large language models like BERT, T5, GPT-3, etc. These models are designed in a fully-parametric fashion to encode all of the desired knowledge in the weights of a large neural network. Such implicit encoding of knowledge makes it hard to interpret whether the model is grounded on certain verifiable evidence. On the other hand, these fully-parametric models need a massive amount of parameters to encode an infinite amount of real-world knowledge. An alternative approach is to modularize the NLP models into two modules, where the memory module is used to store explicit facts and knowledge in a non-parametric form, and the neural computation module can ground on the retrieved memory to make predictions. By decoupling knowledge and computation, the semiparametric models can become more interpretable by demonstrating the evidence it retrieves from the memory. On the other hand, the model's trainable parameters can be reduced significantly to lower computation costs. In this long-term program, I plan to explore several directions related to semiparametric models. Firstly, I will develop semiparametric models that can robustly adapt to memory from different domains while maintaining consistent performance. More specifically, we hope the model can adapt to knowledge in different domains like science, medicine, news, etc without expensive re-training. Secondly, I plan to develop models that can better adapt to time changes in world facts. By updating the memory regularly with up-to-date knowledge, we hope the model can quickly capture the changes and make timely predictions. Thirdly, I plan to explore the possibility of semiparametric models to address broader AI tasks in other areas like biomedical, finance, vision, etc. Finally, I would like to investigate how to build more responsible AI systems with semiparametric models. Specifically, by making AI models more grounded on knowledge retrieval, we believe the model can make become fair across different demographic groups. Through this multi-year program, I aim to deepen our understanding of semiparametric models to understand their limits. Ultimately, we hope to build next-generation foundation models that can address a wide range of AI tasks with better trustworthiness and fairness. In turn, this will necessitate new highly qualified personnel trained in NLP at virtually every company that deals with text data roles -- roles my students will be prepared to fill.