Canada Spends Logo

Explore our Budget 2026 analysis

Grant: Discovery Grants Program - Individual

University of Alberta

$53,000.00

Testing, Debugging and Repairing Machine Learning Software at the System Level

FY 2023-24

Summary
Machine learning (ML) has become the driving force of innovation in many application domains. However, current state-of-the-art ML software still suffers from quality issues. Different from traditional software, ML software adopts the data-driven programming paradigm. Instead of manually programming the decision logic (e.g., in form of source code), the decision logic of the ML software is automatically learned via training and encodes in a model (e.g., neural network). An ML model is often difficult to interpret and understand, calling for novel quality assurance (QA) methods. In practice, an ML model is often not used standalone but integrated as a component into a larger system, including both traditional software and ML models. While some recent progress is made on testing and analysis ML models, systematic research of QA for ML software at the system level is still largely untouched so far. Quality assurance at the ML system level is challenging and requires considering the components of ML models, traditional software, and their interactions. In particular, 1) The ML model behaviors are often difficult to understand. What is the role of an ML model and how its incorrect behaviors impact the whole system? 2) How to effectively detect the defects in the huge testing space of an ML software system? 3) With the error triggering tests, how to debug and identify the defect modules? 4) For system incorrect behaviors caused by ML models, how to repair them to improve system quality? This research program aims to address these challenges and propose novel methods of testing, debugging, and repairing for ML-driven software at the system level, providing key quality assurance supports to establish trustworthy intelligent software. 1) First, I plan to perform a large-scale empirical study to systematically investigate roles and defect impacts of ML models in state-of-the-art ML systems. 2) Then, I will propose an effective testing framework to detect the potential defects of ML at the system level. 3) With the found defects, I plan to design automated debugging techniques to localize the potentially incorrect modules. 4) Regarding the system defects introduced by ML models, I will further propose automated repairing methods to enhance the ML system quality. Large-scale experiments on open source and industrial ML software systems will be conducted to evaluate the advantage, practical value and limitation of proposed techniques. This outcome of this research will originally provide an initial set of key methods to detect, debug, and repair ML software at the system level, which can greatly accelerate the ML system development process with better quality assurance support, potentially impacting many industrial domains. This program will train nine highly qualified personnel (HQP) and provide them with the equity, diversity and inclusivity (EDI) platform to participate and contribute to the state-of-the-art intelligent software engineering research.
Program
Discovery Grants Program - Individual
Location
Alberta, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data

As published

Application id
RGPIN-2021-02549
Name nom
Ma, Lei
Department département
Electrical and Computer Engineering
Organization id
9
Institution établissement
University of Alberta
Province en
Alberta
Province fr
Alberta
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2023
Competition year année de concours
2021
Award amount
53000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
801
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Computer software
Area of application fr
Logiciel
Research subject code
2705
Research subject group en
Information technology
Research subject group fr
Software and development
Research subject en
Technologie de l'information
Research subject fr
Logiciels et développement
Application title
Testing, Debugging and Repairing Machine Learning Software at the System Level
Keyword
Intelligent Software Engineering; Intelligent Software Quality Assurance; Intelligent Software Testing, Debugging; Software Analytics; Software Debugging; Software Quality Assurance; Software Repairing; Software Testing
Application summary
Machine learning (ML) has become the driving force of innovation in many application domains. However, current state-of-the-art ML software still suffers from quality issues. Different from traditional software, ML software adopts the data-driven programming paradigm. Instead of manually programming the decision logic (e.g., in form of source code), the decision logic of the ML software is automatically learned via training and encodes in a model (e.g., neural network). An ML model is often difficult to interpret and understand, calling for novel quality assurance (QA) methods. In practice, an ML model is often not used standalone but integrated as a component into a larger system, including both traditional software and ML models. While some recent progress is made on testing and analysis ML models, systematic research of QA for ML software at the system level is still largely untouched so far. Quality assurance at the ML system level is challenging and requires considering the components of ML models, traditional software, and their interactions. In particular, 1) The ML model behaviors are often difficult to understand. What is the role of an ML model and how its incorrect behaviors impact the whole system? 2) How to effectively detect the defects in the huge testing space of an ML software system? 3) With the error triggering tests, how to debug and identify the defect modules? 4) For system incorrect behaviors caused by ML models, how to repair them to improve system quality? This research program aims to address these challenges and propose novel methods of testing, debugging, and repairing for ML-driven software at the system level, providing key quality assurance supports to establish trustworthy intelligent software. 1) First, I plan to perform a large-scale empirical study to systematically investigate roles and defect impacts of ML models in state-of-the-art ML systems. 2) Then, I will propose an effective testing framework to detect the potential defects of ML at the system level. 3) With the found defects, I plan to design automated debugging techniques to localize the potentially incorrect modules. 4) Regarding the system defects introduced by ML models, I will further propose automated repairing methods to enhance the ML system quality. Large-scale experiments on open source and industrial ML software systems will be conducted to evaluate the advantage, practical value and limitation of proposed techniques. This outcome of this research will originally provide an initial set of key methods to detect, debug, and repair ML software at the system level, which can greatly accelerate the ML system development process with better quality assurance support, potentially impacting many industrial domains. This program will train nine highly qualified personnel (HQP) and provide them with the equity, diversity and inclusivity (EDI) platform to participate and contribute to the state-of-the-art intelligent software engineering research.
University of Alberta - $53,000 Grant (2023-24) | Canada Spends