Canada Spends Logo

Explore our Budget 2026 analysis

Grant: Discovery Grants Program - Individual

Université de Montréal

$46,000.00

BUMP: Better Understanding of Model's Performance for more efficient Natural Language Processing

FY 2024-25

Summary
Fine-tuning a pre-trained language model on a target task is both an elegant and very strong solution. Yet, current models fail in a number of practical Natural Language Processing (NLP) deployments, and this even if they are impressively efficient at solving popular benchmarks. A growing number of studies have thus been questioning lately the NLP practices in benchmarking models. Benchmarks are indeed problematic in a number of ways: they contain biases (ex: gender bias), are typically small (annotating data is costly), thus very often in-domain, meaning they fail to contain a good coverage of representative phenomena useful to train models on. Also, models often succeed on given benchmarks, but for the wrong reasons. State-of-the-art models are indeed particularly prone to exploiting existing biases in benchmarks, learning « heuristics » that perform well on a specific test set, but bypass the learning of the linguistic expertise needed to solve the task. Last, we typically evaluate models with a single accuracy-like score, which is rather limited. This proposal is intended to develop the expertise needed to build a sane NLP benchmarking ecosystem. This encompasses good ways of evaluating models (meaning fine-grained evaluations), detecting and curing biases in models (thus increasing model's robustness) and developing new benchmarks that are as much as possible exempt of problems, and evolve as the same pace as technology does. The key element in this proposal is to devise meaningful ways of valuing a data point -- being either a training or testing example - so that we get better understanding of a model capabilities (diagnosing), and a better control on its training. We will investigate a number of NLP tasks, both canonical ones (such as named-entity recognition or part-of-speech tagging) as well as end-applications (such as machine translation, or any classification task). For each task, we will analyze popular benchmarks, their limitations, as well as sota model's failures. We will study ways of increasing model's robustness and will develop sound benchmarks to evaluate the technology. Such an expertise will contribute to improve NLP technology by delivering informative evaluations, assisting in developing new benchmarks, potentially reducing the number of annotations needed, and devising better training mechanisms.
Program
Discovery Grants Program - Individual
Location
Quebec, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data

As published

Application id
RGPIN-2023-04904
Name nom
Langlais, Philippe
Department département
DIRO
Organization id
63
Institution établissement
Université de Montréal
Province en
Québec
Province fr
Québec
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2023
Award amount
46000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
800
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Information and communication services
Area of application fr
Services d'information et de télécommunications
Research subject code
2800
Research subject group en
Artificial intelligence
Research subject group fr
Intelligence artificielle
Research subject en
Artificial intelligence (computer vision, use2603)
Research subject fr
Intelligence artificielle (vision artificielle, utiliser 2603)
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
BUMP: Better Understanding of Model's Performance for more efficient Natural Language Processing
Keywords
artificial intelligence; natural language processing; benchmarking; biases in natural language processing models; behavioural interpretation of models; model adaptation; pre-trained language models; question answering; machine translation; named-entity recognition
Application summary
Fine-tuning a pre-trained language model on a target task is both an elegant and very strong solution. Yet, current models fail in a number of practical Natural Language Processing (NLP) deployments, and this even if they are impressively efficient at solving popular benchmarks. A growing number of studies have thus been questioning lately the NLP practices in benchmarking models. Benchmarks are indeed problematic in a number of ways: they contain biases (ex: gender bias), are typically small (annotating data is costly), thus very often in-domain, meaning they fail to contain a good coverage of representative phenomena useful to train models on. Also, models often succeed on given benchmarks, but for the wrong reasons. State-of-the-art models are indeed particularly prone to exploiting existing biases in benchmarks, learning « heuristics » that perform well on a specific test set, but bypass the learning of the linguistic expertise needed to solve the task. Last, we typically evaluate models with a single accuracy-like score, which is rather limited. This proposal is intended to develop the expertise needed to build a sane NLP benchmarking ecosystem. This encompasses good ways of evaluating models (meaning fine-grained evaluations), detecting and curing biases in models (thus increasing model's robustness) and developing new benchmarks that are as much as possible exempt of problems, and evolve as the same pace as technology does. The key element in this proposal is to devise meaningful ways of valuing a data point -- being either a training or testing example - so that we get better understanding of a model capabilities (diagnosing), and a better control on its training. We will investigate a number of NLP tasks, both canonical ones (such as named-entity recognition or part-of-speech tagging) as well as end-applications (such as machine translation, or any classification task). For each task, we will analyze popular benchmarks, their limitations, as well as sota model's failures. We will study ways of increasing model's robustness and will develop sound benchmarks to evaluate the technology. Such an expertise will contribute to improve NLP technology by delivering informative evaluations, assisting in developing new benchmarks, potentially reducing the number of annotations needed, and devising better training mechanisms.