Canada Spends Logo

Explore our Budget 2026 analysis

Grant: Master's Award: Canada Graduate Research Scholarships - Master's program

McMaster University

$27,000.00

Semi-automating critical appraisal of biomedical texts with domain specific pre-trained transformer models

FY 2025-26

Summary
Background : Systematic reviews (SRs) are a crucial part of the health evidence pyramid, as they inform clinical care guidelines. Before application to guidelines, SRs must be critically appraised for their internal validity - most commonly with the AMSTAR-2 tool. Human assessment takes one hour, so semi-automation of this task would greatly improve the efficiency of this task. This is the first study to our knowledge that is working on automating AMSTAR-2 assessment. Objectives: To semi-automate critical appraisal with the AMSTAR-2 tool for SRs with domain specific transformers and to explain the classification decision made by these transformers with SHAP and integrated gradients. Methods: We will search for umbrella reviews (reviews of systematic reviews) published after the AMSTAR-2 tool was released. Articles will be uploaded into Covidence for screening in duplicate. The AMSTAR-2 assessments will be extracted, along with the article PDF. This will form the gold standard dataset that we will train our ML models on. Models will be trained on GPUs from the Digital Research Alliance of Canada. We will use deep learning language models with random forest classifiers as our previous work has demonstrated this approach yielded the best performance. The primary classification metric will be MCC. Significance: Our previous work with automating RoB 2.0 assessment showed smaller language models like BERT achieved similar performance to that of LLMs. Given that LLM's do not perform significantly better, the increased computational cost may not be worth the burden. By validating smaller transformer models, we will democratize this AI tool in making it available to those without intensive resources and mitigate the environmental implications associated with training LLMs.
Program
Master's Award: Canada Graduate Research Scholarships - Master's program
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
CIHR Funding Decisions Database

As published

Id
573038
Name
Xie Daniel
Region
Ontario
Country
Canada
Orgtype
Institution Paid; Research Institution
Prcname2
Special Cases - Awards Programs
Projecttitle
Semi-automating critical appraisal of biomedical texts with domain specific pre-trained transformer models
Programtype2
Studentship Programs
Orgnameinp2
McMaster University
Cihramount2
$27,000
Cihrequipment2
$0
Programname2
Master's Award: Canada Graduate Research Scholarships - Master's program
Keyworddelim
Ai Explainability; Artificial Intelligence; Binary Classification; Large Language Models; Machine Learning; Risk Of Bias
Abstract
Background : Systematic reviews (SRs) are a crucial part of the health evidence pyramid, as they inform clinical care guidelines. Before application to guidelines, SRs must be critically appraised for their internal validity - most commonly with the AMSTAR-2 tool. Human assessment takes one hour, so semi-automation of this task would greatly improve the efficiency of this task. This is the first study to our knowledge that is working on automating AMSTAR-2 assessment. Objectives: To semi-automate critical appraisal with the AMSTAR-2 tool for SRs with domain specific transformers and to explain the classification decision made by these transformers with SHAP and integrated gradients. Methods: We will search for umbrella reviews (reviews of systematic reviews) published after the AMSTAR-2 tool was released. Articles will be uploaded into Covidence for screening in duplicate. The AMSTAR-2 assessments will be extracted, along with the article PDF. This will form the gold standard dataset that we will train our ML models on. Models will be trained on GPUs from the Digital Research Alliance of Canada. We will use deep learning language models with random forest classifiers as our previous work has demonstrated this approach yielded the best performance. The primary classification metric will be MCC. Significance: Our previous work with automating RoB 2.0 assessment showed smaller language models like BERT achieved similar performance to that of LLMs. Given that LLM's do not perform significantly better, the increased computational cost may not be worth the burden. By validating smaller transformer models, we will democratize this AI tool in making it available to those without intensive resources and mitigate the environmental implications associated with training LLMs.
Competitiondate
202512
Approvedterm2
1 yr 0 mth
Pinamesdelim
Xie, Daniel
Approvedterm
1 an 0 mois
Cihrcontribution2
~ $27,000~ $0~