Canada Spends Logo

Explore our Budget 2026 analysis

Grant: Alliance Grants

École Polytechnique de Montréal

$65,132.00

A Comprehensive Framework for the Automatic Evaluation of the Quality of ML-based Software Systems

FY 2021-22

Summary
Nowadays, Machine Learning Software Systems (MLSS)s have become a part of our daily life (e.g., recommendation systems, speech recognition, face detection). An increasing demand is observed in various companies to employ ML for solving problems in their business. The heart of MLSS is an ML model. These models are implemented as software and like any other software, quality assurance is necessary. The quality assessment of MLSSs is regarded as a challenging task and is currently a hot research topic in the literature. According to the growing deployment of MLSSs, there is a strong need for ensuring their serving quality. False or poor decisions of such systems can lead to malfunction of other systems, significant financial losses, or even threat to human life.In this project, for a comprehensive quality assessment of ML models in MLSSs, the role of the ML model as a part of the system at different stages of its life cycle will be considered. Various aspects of the quality of ML models from performance and robustness (like prediction accuracy, data bias, and variance), to scalability, hardware/software demand, complexity, user acceptance, and explainability will be evaluated. A multi-objective framework will be implemented to take into account all properties of ML model quality in MLSSs. Finally, we will aggregate our proposed solutions in a practical toolset to automatically evaluate, validate, and track the quality of ML models throughout their life cycle. The toolset will be integrated into the state-of-the-art tools for continuous integration and delivery of software systems. This toolset will provide MoovAI as well as other Quebec and Candian companies using ML, with a competitive edge in the booming ML and AI market. The dependability of their MLSS will be a great asset in winning new markets. The usage of high-quality ML models and reliable MLSS will increase trust in ML/AI technologies across the Quebec and Canadian industry.
Program
Alliance Grants
Location
Quebec, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data

As published

Application id
561420-2020
Name nom
Khomh, Foutse
Department département
Génie Informatique et Génie Logiciel
Organization id
64
Institution établissement
École Polytechnique de Montréal
Province en
Québec
Province fr
Québec
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2021
Competition year année de concours
2020
Award amount
65132
Program id
ALLRP
Program name en
Alliance Grants
Program name fr
Subventions Alliance
Group en
Research partnerships programs
Group fr
Programmes de partenariat
Committee code
96
Committee name en
RPP Internal Decision Cttee
Committee name fr
Cté de décision interne - PPR
Area of application code
802
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Information systems and technology
Area of application fr
Systèmes d'information et technologie de l'information
Research subject code
2706
Research subject group en
Information technology
Research subject group fr
Technologie de l'information
Research subject en
Software engineering
Research subject fr
Génie logiciel
Installment
inst:1
Partie
1
Nb partie
4
Application title
A Comprehensive Framework for the Automatic Evaluation of the Quality of ML-based Software Systems
Keyword
NSERC
Application summary
Nowadays, Machine Learning Software Systems (MLSS)s have become a part of our daily life (e.g., recommendation systems, speech recognition, face detection). An increasing demand is observed in various companies to employ ML for solving problems in their business. The heart of MLSS is an ML model. These models are implemented as software and like any other software, quality assurance is necessary. The quality assessment of MLSSs is regarded as a challenging task and is currently a hot research topic in the literature. According to the growing deployment of MLSSs, there is a strong need for ensuring their serving quality. False or poor decisions of such systems can lead to malfunction of other systems, significant financial losses, or even threat to human life.In this project, for a comprehensive quality assessment of ML models in MLSSs, the role of the ML model as a part of the system at different stages of its life cycle will be considered. Various aspects of the quality of ML models from performance and robustness (like prediction accuracy, data bias, and variance), to scalability, hardware/software demand, complexity, user acceptance, and explainability will be evaluated. A multi-objective framework will be implemented to take into account all properties of ML model quality in MLSSs. Finally, we will aggregate our proposed solutions in a practical toolset to automatically evaluate, validate, and track the quality of ML models throughout their life cycle. The toolset will be integrated into the state-of-the-art tools for continuous integration and delivery of software systems. This toolset will provide MoovAI as well as other Quebec and Candian companies using ML, with a competitive edge in the booming ML and AI market. The dependability of their MLSS will be a great asset in winning new markets. The usage of high-quality ML models and reliable MLSS will increase trust in ML/AI technologies across the Quebec and Canadian industry.
École Polytechnique de Montréal - $65,132 Grant (2021… | Canada Spends