Canada Spends Logo

Explorez notre analyse du budget 2026

Grant: Discovery Grants Program - Individual

Carleton University

$36,000.00

Automating the Use of and Experimentally Studying Category-Partition Black-Box Software Testing

EF 2024-25

Résumé
Category Partition (CP) is a standard functional software testing technique. CP is said to be functional because its primary source of information is the, often plain-language, specification of the software under test. When using CP, the engineer has to make a number of important decisions that will have an impact on the result of the testing campaign, both in terms of cost (e.g., the number of tests to execute) and effectiveness (e.g., at finding faults). Similar to many decisions engineers make in software engineering, the impact of these decisions on cost and effectiveness cannot be determined by analytical means and we have to resort to experiments. Unfortunately, there is very scarce empirical data engineers can rely on to make wise choices when applying CP; more often than not, the engineer chooses without a sound rationale. The first main objective of this research is therefore to conduct extensive controlled experiments to study the impact on cost-effectiveness of alternative choices when using CP, to create a body of knowledge that will help engineers better use CP. Note that this requires that we automate CP as much as possible; so a related objective is to build this tool support for CP, which we refer to as Melba. Another important impediment to the use of CP is that it does not include the identification of a test oracle, which is used to identify what to expect from a test execution and compare it to what is observed. Rather, it is up to the user to manually identify what behaviour is expected from a correct implementation of the system under test when executing test cases. The second main objective of this research is therefore to automated the oracle. Completely automating the oracle is hard to imagine without additional significant effort from the engineers, such as requiring that contracts be inserted into the source code (the so-called pre and post conditions for instance). Asking the engineer to specify what behaviour to expect for every single test case is also out of the question to us, especially when CP produces many tests for a non-trivial system under test (on the order of one hundred for instance). Our research instead relies on the specification of outputs of the system under test, similarly to what CP asks users to do with inputs. Executing the tests created with CP will reveal which output characterizations are actually used and machine learning will help the engineer identify whether the learned input-output relations make sense, thereby verifying the system under test. These specified/learned input-output relationships are metamorphic relations to be used in metamorphic testing. A third objective will then to add tool support to Melba for the proposed procedure and to experimentally study it.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Ontario, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data

Tel que publié

Application id
RGPIN-2023-05849
Name nom
Labiche, Yvan
Department département
Systems and Computer Engineering
Organization id
23
Institution établissement
Carleton University
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2023
Award amount
36000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
801
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Computer software
Area of application fr
Logiciel
Research subject code
2700
Research subject group en
Information technology
Research subject group fr
Technologie de l'information
Research subject en
Information technology
Research subject fr
Technologie de l'information
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Automating the Use of and Experimentally Studying Category-Partition Black-Box Software Testing
Keywords
software verification and validation; software testing; functional testing; empirical software engineering; oracle problem; automation; tool support
Application summary
Category Partition (CP) is a standard functional software testing technique. CP is said to be functional because its primary source of information is the, often plain-language, specification of the software under test. When using CP, the engineer has to make a number of important decisions that will have an impact on the result of the testing campaign, both in terms of cost (e.g., the number of tests to execute) and effectiveness (e.g., at finding faults). Similar to many decisions engineers make in software engineering, the impact of these decisions on cost and effectiveness cannot be determined by analytical means and we have to resort to experiments. Unfortunately, there is very scarce empirical data engineers can rely on to make wise choices when applying CP; more often than not, the engineer chooses without a sound rationale. The first main objective of this research is therefore to conduct extensive controlled experiments to study the impact on cost-effectiveness of alternative choices when using CP, to create a body of knowledge that will help engineers better use CP. Note that this requires that we automate CP as much as possible; so a related objective is to build this tool support for CP, which we refer to as Melba. Another important impediment to the use of CP is that it does not include the identification of a test oracle, which is used to identify what to expect from a test execution and compare it to what is observed. Rather, it is up to the user to manually identify what behaviour is expected from a correct implementation of the system under test when executing test cases. The second main objective of this research is therefore to automated the oracle. Completely automating the oracle is hard to imagine without additional significant effort from the engineers, such as requiring that contracts be inserted into the source code (the so-called pre and post conditions for instance). Asking the engineer to specify what behaviour to expect for every single test case is also out of the question to us, especially when CP produces many tests for a non-trivial system under test (on the order of one hundred for instance). Our research instead relies on the specification of outputs of the system under test, similarly to what CP asks users to do with inputs. Executing the tests created with CP will reveal which output characterizations are actually used and machine learning will help the engineer identify whether the learned input-output relations make sense, thereby verifying the system under test. These specified/learned input-output relationships are metamorphic relations to be used in metamorphic testing. A third objective will then to add tool support to Melba for the proposed procedure and to experimentally study it.