Grant: Discovery Grants Program - Individual
Carleton University
$31,000.00
Category partition black-box software testing: theory, tool support and experiments
EF 2020-21
Résumé
Category partition (CP) is a black-box software testing technique based on equivalence class partitioning and boundary value analysis. It applies to many contexts: e.g., C unit testing, Java unit/integration testing, system testing of use cases. It has also been combined with other techniques such as state-based testing from extended finite state machines.
CP begins by identifying the parameters and environment variables of a functionality under test. Environment variables are factors in the environment of execution of the program under test that may impact its behaviour. These parameters and environment variables are then envisioned into categories, which are characteristics that are deemed important from a testing viewpoint. Each characteristic leads to the definition of choices (i.e., equivalent classes) splitting the domain of values (implicitly) defined by the characteristic. Constraints can then specify that some choices from two different categories should always be used together, can never be used together, or can only be used together under certain condition. The choices are then combined to form test frames according to a selection criterion, while satisfying constraints. A simple adequacy criterion requires that each choice appears at least once in the set of test frames. Each test frame is then associated with actual input values (for parameters and environment variables), according to the choice specifications, to produce test cases.
When using CP, one needs to take many decisions that can impact the result (e.g., cost, fault detection) of the testing campaign. Obviously the selection of choices has an impact: e.g., poorly defined categories and choices will lead to faults that slip through the testing activity; Choosing an adequacy criterion matters because criteria are not all equally demanding (i.e., cost) or effective (i.e., fault detection); The technology to produce a set of test frames adequate for a criterion, while accounting for constraints, will have an impact; The procedure to identify input values for test frames may have an impact.
Unfortunately, very little is known about the actual, precise impact of those decisions on the result of applying CP.
The purpose of this project is to fill this gap. Tool support will be created to facilitate the use of CP and alternative decisions, on different case studies. Real and synthetic faults will be used to evaluate the impact of the many alternatives on the effectiveness of the testing technique at finding faults.
The results will be of high interest to the research community: it is the first time the many alternative applications of CP will be studied. Results will also be highly relevant to industry: tool support, experimental evidence leading to better decision making. Results will be highly interesting for teaching purposes: tailoring the teaching of the technique, availability of the tool, including an online version.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Ontario, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data
Tel que publié
Application id
RGPIN-2016-06214
Name nom
Labiche, Yvan
Department département
Systems and Computer Engineering
Organization id
23
Institution établissement
Carleton University
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2020
Competition year année de concours
2016
Award amount
31000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
Discovery grants programs
Group fr
Programmes de subventions à la découverte
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
801
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Computer software
Area of application fr
Logiciel
Research subject code
2706
Research subject group en
Information technology
Research subject group fr
Technologie de l'information
Research subject en
Software engineering
Research subject fr
Génie logiciel
Installment
inst:5-6
Partie
2
Nb partie
6
Application title
Category partition black-box software testing: theory, tool support and experiments
Keyword
NSERC
Application summary
Category partition (CP) is a black-box software testing technique based on equivalence class partitioning and boundary value analysis. It applies to many contexts: e.g., C unit testing, Java unit/integration testing, system testing of use cases. It has also been combined with other techniques such as state-based testing from extended finite state machines.
CP begins by identifying the parameters and environment variables of a functionality under test. Environment variables are factors in the environment of execution of the program under test that may impact its behaviour. These parameters and environment variables are then envisioned into categories, which are characteristics that are deemed important from a testing viewpoint. Each characteristic leads to the definition of choices (i.e., equivalent classes) splitting the domain of values (implicitly) defined by the characteristic. Constraints can then specify that some choices from two different categories should always be used together, can never be used together, or can only be used together under certain condition. The choices are then combined to form test frames according to a selection criterion, while satisfying constraints. A simple adequacy criterion requires that each choice appears at least once in the set of test frames. Each test frame is then associated with actual input values (for parameters and environment variables), according to the choice specifications, to produce test cases.
When using CP, one needs to take many decisions that can impact the result (e.g., cost, fault detection) of the testing campaign. Obviously the selection of choices has an impact: e.g., poorly defined categories and choices will lead to faults that slip through the testing activity; Choosing an adequacy criterion matters because criteria are not all equally demanding (i.e., cost) or effective (i.e., fault detection); The technology to produce a set of test frames adequate for a criterion, while accounting for constraints, will have an impact; The procedure to identify input values for test frames may have an impact.
Unfortunately, very little is known about the actual, precise impact of those decisions on the result of applying CP.
The purpose of this project is to fill this gap. Tool support will be created to facilitate the use of CP and alternative decisions, on different case studies. Real and synthetic faults will be used to evaluate the impact of the many alternatives on the effectiveness of the testing technique at finding faults.
The results will be of high interest to the research community: it is the first time the many alternative applications of CP will be studied. Results will also be highly relevant to industry: tool support, experimental evidence leading to better decision making. Results will be highly interesting for teaching purposes: tailoring the teaching of the technique, availability of the tool, including an online version.