Canada Spends Logo

Explorez notre analyse du budget 2026

Grant: Discovery Grants Program - Individual

University of Ottawa

$24,000.00

A visualization framework for machine learning evaluation

EF 2009-10

Résumé
In the past 25 years, machine learning, first, and then, data mining, made constant progress to the point where the technology is now being used in all walks of life such as in automatic fraud detection, recommender systems, equipment monitoring and so on. In recent years, however, very few actual improvements were reported in the literature, particularly in the area of classification. This could either mean that classification is at its pinnacle and does not need to be investigated any further; or it could mean that our approach to evaluating the outcome of our classification algorithms is too simplistic and does not consider all the relevant aspects of our systems. Given the reticence of many practitioner in other fields to adopt our techniques which are not, they feel, tested to a level of adequacy that makes them worth their while, the latter is more likely than the former. The purpose of this proposed research is, thus, to address the issue of classifier evaluation by proposing to build a software tool that will allow classifiers to be evaluated visually so as to allow the user to gather a great deal of relevant information on the performance of one or several systems, in a way that is both sound and humanely understandable. In particular, we propose a new framework that approaches the problem of classifier evaluation as a problem of visualization of high-dimensional data. In so doing, we intend to borrow tools previously designed for that field (e.g., sophisticated nonlinear projections) and expand on these tools to allow for complex evaluation procedures such as the incorporation of statistical guarantees and the consideration of threshold-sensitive classifiers to be included. In addition to providing tools for comparing classifiers on various domains, we intend to develop tools that will allow us to organize domains into equivalence classes within which various types of classifiers are known to behave predictably. We also intend to explore the role of artificial data generation for machine learning evaluation and provide tools to generate such domains. Like recent research on ROC Analysis, our work is intended to make the evaluation process more meaningful, but unlike that research, it is intended to be easily usable and interpretable.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Ontario, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data

Tel que publié

Application id
228118-2009
Name nom
Japkowicz, Nathalie
Department département
Information Technology and Engineering, School of
Organization id
28
Institution établissement
University of Ottawa
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2009
Competition year année de concours
2009
Award amount
24000
Program id
RGPIN
Program nane en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
Discovery grants programs
Group fr
Programmes de subventions à la découverte
Committee code
331
Committee name en
Computing and Information Sciences - B
Committee name fr
Informatique - B
Area of application code
802
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Information systems and technology
Area of application fr
Systèmes d'information et technologie de l'information
Research subject code
2805
Research subject group en
Artificial intelligence
Research subject group fr
Intelligence artificielle
Research subject en
Learning and inference theories
Research subject fr
Théories de l'apprentissage et l'inférence
Partie
1
Nb partie
5
Application title
A visualization framework for machine learning evaluation
Keyword
NA
Application summary
In the past 25 years, machine learning, first, and then, data mining, made constant progress to the point where the technology is now being used in all walks of life such as in automatic fraud detection, recommender systems, equipment monitoring and so on. In recent years, however, very few actual improvements were reported in the literature, particularly in the area of classification. This could either mean that classification is at its pinnacle and does not need to be investigated any further; or it could mean that our approach to evaluating the outcome of our classification algorithms is too simplistic and does not consider all the relevant aspects of our systems. Given the reticence of many practitioner in other fields to adopt our techniques which are not, they feel, tested to a level of adequacy that makes them worth their while, the latter is more likely than the former. The purpose of this proposed research is, thus, to address the issue of classifier evaluation by proposing to build a software tool that will allow classifiers to be evaluated visually so as to allow the user to gather a great deal of relevant information on the performance of one or several systems, in a way that is both sound and humanely understandable. In particular, we propose a new framework that approaches the problem of classifier evaluation as a problem of visualization of high-dimensional data. In so doing, we intend to borrow tools previously designed for that field (e.g., sophisticated nonlinear projections) and expand on these tools to allow for complex evaluation procedures such as the incorporation of statistical guarantees and the consideration of threshold-sensitive classifiers to be included. In addition to providing tools for comparing classifiers on various domains, we intend to develop tools that will allow us to organize domains into equivalence classes within which various types of classifiers are known to behave predictably. We also intend to explore the role of artificial data generation for machine learning evaluation and provide tools to generate such domains. Like recent research on ROC Analysis, our work is intended to make the evaluation process more meaningful, but unlike that research, it is intended to be easily usable and interpretable.