Canada Spends Logo

Explore our Budget 2026 analysis

Grant: Discovery Grants Program - Individual

University of Calgary

$29,000.00

Using case difficulty to improve predictive performance evaluation

FY 2024-25

Summary
In recent years, machine learning (ML) has led to a number of ground-breaking innovations in artificial intelligence (AI). In particular, predictive models have been at the forefront of this AI revolution (e.g., object detection in photos). Despite this seemingly substantial and rapid progress, some aspects of ML have not changed much for decades. One such area of stagnation is how all cases are treated equally in predictive performance evaluation. Most performance metrics are solely based on how many cases are predicted correctly or incorrectly and completely ignore which cases. As a result, the concept of case difficulty (i.e., how difficult is it to predict a given case correctly?) is not considered, often resulting in misleading performance evaluation. Although different cases tend to exhibit varying degrees of difficulty in many prediction problems, correctly/incorrectly predicting a difficult case is weighted just the same as correctly/incorrectly predicting an easy case. A related issue is whether all available test cases need to be used for predictive performance evaluation. Is it possible to accurately assess performance by only using a small number of test cases selected based on case difficulty? Inspired by how computerized tests such as the Graduate Record Examination (GRE) select and use only a subset of a pool of questions, this approach can prevent performance evaluation from being dominated by too easy or too difficult cases and mitigate insufficient data issues in many problem domains. My long-term research program focuses on developing novel ML methodologies that crosscut multiple problem domains. Over the next 5 years, my research will focus on developing novel predictive performance metrics and evaluation methods. The objectives are to: 1.Develop new predictive performance metrics based on case difficulty. These performance metrics will ensure correct prediction of difficult cases is rewarded, whereas incorrect prediction of easy cases is penalized. 2.Develop new case difficulty-based predictive performance evaluation methods that require fewer test cases than traditional methods. Test cases will be presented sequentially and the next test case will be selected based on case difficulty and whether the previous test cases were predicted correctly. The primary deliverable of this research will be an open-source Python package that implements our novel predictive performance metrics and evaluation methods. This package will enable ML researchers and engineers to incorporate case difficulty into their predictive modeling and precisely evaluate their models. In our increasingly AI-driven society, predictive models are more prevalent than ever. This research is important because it will lead to comprehensive insights into how these predictive models that affect our lives truly perform. Canadians will benefit from resulting trustworthy predictive models, further strengthening Canada's international leadership in ML research.
Program
Discovery Grants Program - Individual
Location
Alberta, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data

As published

Application id
RGPIN-2021-02588
Name nom
Lee, Joon
Department département
Community Health Sciences - Community Health Sciences
Organization id
11
Institution établissement
University of Calgary
Province en
Alberta
Province fr
Alberta
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2021
Award amount
29000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
802
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Information systems and technology
Area of application fr
Systèmes d'information et technologie de l'information
Research subject code
2804
Research subject group en
Artificial intelligence
Research subject group fr
Intelligence artificielle
Research subject en
Expert systems
Research subject fr
Systèmes experts
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Using case difficulty to improve predictive performance evaluation
Keywords
supervised machine learning; predictive performance evaluation; case difficulty; predictive modeling; computerized adaptive testing; predictive performance metric; item response theory
Application summary
In recent years, machine learning (ML) has led to a number of ground-breaking innovations in artificial intelligence (AI). In particular, predictive models have been at the forefront of this AI revolution (e.g., object detection in photos). Despite this seemingly substantial and rapid progress, some aspects of ML have not changed much for decades. One such area of stagnation is how all cases are treated equally in predictive performance evaluation. Most performance metrics are solely based on how many cases are predicted correctly or incorrectly and completely ignore which cases. As a result, the concept of case difficulty (i.e., how difficult is it to predict a given case correctly?) is not considered, often resulting in misleading performance evaluation. Although different cases tend to exhibit varying degrees of difficulty in many prediction problems, correctly/incorrectly predicting a difficult case is weighted just the same as correctly/incorrectly predicting an easy case. A related issue is whether all available test cases need to be used for predictive performance evaluation. Is it possible to accurately assess performance by only using a small number of test cases selected based on case difficulty? Inspired by how computerized tests such as the Graduate Record Examination (GRE) select and use only a subset of a pool of questions, this approach can prevent performance evaluation from being dominated by too easy or too difficult cases and mitigate insufficient data issues in many problem domains. My long-term research program focuses on developing novel ML methodologies that crosscut multiple problem domains. Over the next 5 years, my research will focus on developing novel predictive performance metrics and evaluation methods. The objectives are to: 1.Develop new predictive performance metrics based on case difficulty. These performance metrics will ensure correct prediction of difficult cases is rewarded, whereas incorrect prediction of easy cases is penalized. 2.Develop new case difficulty-based predictive performance evaluation methods that require fewer test cases than traditional methods. Test cases will be presented sequentially and the next test case will be selected based on case difficulty and whether the previous test cases were predicted correctly. The primary deliverable of this research will be an open-source Python package that implements our novel predictive performance metrics and evaluation methods. This package will enable ML researchers and engineers to incorporate case difficulty into their predictive modeling and precisely evaluate their models. In our increasingly AI-driven society, predictive models are more prevalent than ever. This research is important because it will lead to comprehensive insights into how these predictive models that affect our lives truly perform. Canadians will benefit from resulting trustworthy predictive models, further strengthening Canada's international leadership in ML research.