Canada Spends Logo

Explorez notre analyse du budget 2026

Grant: Alliance Grants

École Polytechnique de Montréal

$65,132.00

A Comprehensive Framework for the Automatic Evaluation of the Quality of ML-based Software Systems

EF 2021-22

Résumé
Nowadays, Machine Learning Software Systems (MLSS)s have become a part of our daily life (e.g., recommendation systems, speech recognition, face detection). An increasing demand is observed in various companies to employ ML for solving problems in their business. The heart of MLSS is an ML model. These models are implemented as software and like any other software, quality assurance is necessary. The quality assessment of MLSSs is regarded as a challenging task and is currently a hot research topic in the literature. According to the growing deployment of MLSSs, there is a strong need for ensuring their serving quality. False or poor decisions of such systems can lead to malfunction of other systems, significant financial losses, or even threat to human life.In this project, for a comprehensive quality assessment of ML models in MLSSs, the role of the ML model as a part of the system at different stages of its life cycle will be considered. Various aspects of the quality of ML models from performance and robustness (like prediction accuracy, data bias, and variance), to scalability, hardware/software demand, complexity, user acceptance, and explainability will be evaluated. A multi-objective framework will be implemented to take into account all properties of ML model quality in MLSSs. Finally, we will aggregate our proposed solutions in a practical toolset to automatically evaluate, validate, and track the quality of ML models throughout their life cycle. The toolset will be integrated into the state-of-the-art tools for continuous integration and delivery of software systems. This toolset will provide MoovAI as well as other Quebec and Candian companies using ML, with a competitive edge in the booming ML and AI market. The dependability of their MLSS will be a great asset in winning new markets. The usage of high-quality ML models and reliable MLSS will increase trust in ML/AI technologies across the Quebec and Canadian industry.
Programme
Alliance Grants
Emplacement
Quebec, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data

Tel que publié

Application id
561420-2020
Name nom
Khomh, Foutse
Department département
Génie Informatique et Génie Logiciel
Organization id
64
Institution établissement
École Polytechnique de Montréal
Province en
Québec
Province fr
Québec
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2021
Competition year année de concours
2020
Award amount
65132
Program id
ALLRP
Program name en
Alliance Grants
Program name fr
Subventions Alliance
Group en
Research partnerships programs
Group fr
Programmes de partenariat
Committee code
96
Committee name en
RPP Internal Decision Cttee
Committee name fr
Cté de décision interne - PPR
Area of application code
802
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Information systems and technology
Area of application fr
Systèmes d'information et technologie de l'information
Research subject code
2706
Research subject group en
Information technology
Research subject group fr
Technologie de l'information
Research subject en
Software engineering
Research subject fr
Génie logiciel
Installment
inst:1
Partie
1
Nb partie
4
Application title
A Comprehensive Framework for the Automatic Evaluation of the Quality of ML-based Software Systems
Keyword
NSERC
Application summary
Nowadays, Machine Learning Software Systems (MLSS)s have become a part of our daily life (e.g., recommendation systems, speech recognition, face detection). An increasing demand is observed in various companies to employ ML for solving problems in their business. The heart of MLSS is an ML model. These models are implemented as software and like any other software, quality assurance is necessary. The quality assessment of MLSSs is regarded as a challenging task and is currently a hot research topic in the literature. According to the growing deployment of MLSSs, there is a strong need for ensuring their serving quality. False or poor decisions of such systems can lead to malfunction of other systems, significant financial losses, or even threat to human life.In this project, for a comprehensive quality assessment of ML models in MLSSs, the role of the ML model as a part of the system at different stages of its life cycle will be considered. Various aspects of the quality of ML models from performance and robustness (like prediction accuracy, data bias, and variance), to scalability, hardware/software demand, complexity, user acceptance, and explainability will be evaluated. A multi-objective framework will be implemented to take into account all properties of ML model quality in MLSSs. Finally, we will aggregate our proposed solutions in a practical toolset to automatically evaluate, validate, and track the quality of ML models throughout their life cycle. The toolset will be integrated into the state-of-the-art tools for continuous integration and delivery of software systems. This toolset will provide MoovAI as well as other Quebec and Candian companies using ML, with a competitive edge in the booming ML and AI market. The dependability of their MLSS will be a great asset in winning new markets. The usage of high-quality ML models and reliable MLSS will increase trust in ML/AI technologies across the Quebec and Canadian industry.