Grant: Discovery Grants Program - Individual
University of Waterloo
$64,000.00
Pretrained Transformers for Effective and Efficient Information Access: BERT and Beyond
EF 2021-22
Résumé
Users expect search systems that are fast (i.e., efficient) and return good results (i.e., effective), but these features are often in tension: effective in-depth content analysis can be slow, and fast systems often sacrifice quality. Building on a quarter of a century of experience developing techniques and building systems that connect users to relevant information, this proposal will lead to groundbreaking techniques for search and question answering (QA) that are effective as well as efficient. These capabilities will be demonstrated in high-impact applications on scientific texts such as the literature on coronaviruses or hydrology. Overall, my research will advance the scientific frontiers of both natural language processing (NLP) and information retrieval (IR). This proposal formulates search and question answering as ranking problems, and adopts an approach based on deep learning using a class of neural network architectures known as transformers. I will pursue the overall research vision from two complementary perspectives: (A) From the perspective of effectiveness, my research group will develop retrieval-specific self-supervision techniques and gain a better understanding of why transformers work as the basis for building improved ranking models. (B) From the perspective of efficiency, this research will build learned representations for ranking that are amenable to simple vector comparisons and develop model distillation techniques for accelerated inference. These separate threads will come together (C) in a principled framework for reasoning about effectiveness/efficiency tradeoffs, (D) demonstrated in applications that provide search and QA capabilities to literature in three scientific domains: biomedicine, hydrology, and artificial intelligence. Deployed prototypes will serve as a testbed for research and provide useful tools for stakeholders. Efforts along these lines have already begun: shortly after the start of the global COVID-19 pandemic, I led the development of Covidex (covidex.ai), an online and publicly accessible search engine for a collection of scientific articles related to coronaviruses. Such a system could be valuable, for example, to public health officials assessing the efficacy of different interventions and clinicians conducting meta-analyses. This work will achieve impact in four ways: (1) scientific innovations in the form of breakthroughs in deep learning methods that tackle search and QA, (2) computational artifacts in the form of open-source code, data, and models that will help foster adoption of the innovations arising from this research, (3) real-world applications for searching scientific literature in three domains, and (4) high-quality training opportunities. These efforts will contribute to the Canadian economy by enriching the ecosystem of ideas and talent around artificial intelligence and data science, complementing investments in these fields at the federal and provincial levels.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Ontario, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data
Tel que publié
Application id
RGPIN-2021-02490
Name nom
Lin, Jimmy
Department département
School of Computer Science
Organization id
33
Institution établissement
University of Waterloo
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2021
Competition year année de concours
2021
Award amount
64000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
Discovery grants programs
Group fr
Programmes de subventions à la découverte
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
802
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Information systems and technology
Area of application fr
Systèmes d'information et technologie de l'information
Research subject code
2801
Research subject group en
Artificial intelligence
Research subject group fr
Intelligence artificielle
Research subject en
Natural language and speech understanding
Research subject fr
Langage naturel et reconnaissance de la parole
Installment
inst:1
Partie
1
Nb partie
5
Application title
Pretrained Transformers for Effective and Efficient Information Access: BERT and Beyond
Keyword
NSERC
Application summary
Users expect search systems that are fast (i.e., efficient) and return good results (i.e., effective), but these features are often in tension: effective in-depth content analysis can be slow, and fast systems often sacrifice quality. Building on a quarter of a century of experience developing techniques and building systems that connect users to relevant information, this proposal will lead to groundbreaking techniques for search and question answering (QA) that are effective as well as efficient. These capabilities will be demonstrated in high-impact applications on scientific texts such as the literature on coronaviruses or hydrology. Overall, my research will advance the scientific frontiers of both natural language processing (NLP) and information retrieval (IR). This proposal formulates search and question answering as ranking problems, and adopts an approach based on deep learning using a class of neural network architectures known as transformers. I will pursue the overall research vision from two complementary perspectives: (A) From the perspective of effectiveness, my research group will develop retrieval-specific self-supervision techniques and gain a better understanding of why transformers work as the basis for building improved ranking models. (B) From the perspective of efficiency, this research will build learned representations for ranking that are amenable to simple vector comparisons and develop model distillation techniques for accelerated inference. These separate threads will come together (C) in a principled framework for reasoning about effectiveness/efficiency tradeoffs, (D) demonstrated in applications that provide search and QA capabilities to literature in three scientific domains: biomedicine, hydrology, and artificial intelligence. Deployed prototypes will serve as a testbed for research and provide useful tools for stakeholders. Efforts along these lines have already begun: shortly after the start of the global COVID-19 pandemic, I led the development of Covidex (covidex.ai), an online and publicly accessible search engine for a collection of scientific articles related to coronaviruses. Such a system could be valuable, for example, to public health officials assessing the efficacy of different interventions and clinicians conducting meta-analyses. This work will achieve impact in four ways: (1) scientific innovations in the form of breakthroughs in deep learning methods that tackle search and QA, (2) computational artifacts in the form of open-source code, data, and models that will help foster adoption of the innovations arising from this research, (3) real-world applications for searching scientific literature in three domains, and (4) high-quality training opportunities. These efforts will contribute to the Canadian economy by enriching the ecosystem of ideas and talent around artificial intelligence and data science, complementing investments in these fields at the federal and provincial levels.