Canada Spends Logo

Explore our Budget 2026 analysis

Grant: Engage Grants Program

University of Waterloo

$25,000.00

Analyzing causes of test failures

FY 2015-16

Summary
In this work, we propose to carry out a systematic analysis of test failures and suggest strategies to mitigate test flakiness. The work would investigate symptoms as observed in test execution logs, as well as potential causes including recent code changes; changes in tests' dependencies; and plugin load orders. It would also investigate correlations between test types and failure rates: for instance, some of Arista's tests are based on state machines. Do such tests fail more or less often than other tests? The immediate outcome of the work would be a diagnosis of the current state of failing tests along with suggestions to increase the reliability of the test suite. We also propose to build models of test cases which would encapsulate the reasons that they fail.
Program
Engage Grants Program
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data

As published

Application id
485177-2015
Name nom
Lam, Patrick
Department département
Electrical & Computer Engineering
Organization id
33
Institution établissement
University of Waterloo
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2015
Competition year année de concours
2015
Award amount
25000
Program id
EGP
Program name en
Engage Grants Program
Program name fr
Programme de subventions d'engagement partenariat
Group en
Research partnerships programs
Group fr
Programmes de partenariat
Committee code
1553
Committee name en
Ontario Internal Decision Committee
Committee name fr
Comité de décision interne pour l'Ontario
Area of application code
1207
Area of application group en
Advancement of knowledge
Area of application group fr
Avancement des connaissances
Area of application en
Information, computer and communication technologies
Area of application fr
Technologie de l'information, des ordinateurs et des réseaux de communications
Research subject code
2705
Research subject group en
Information technology
Research subject group fr
Technologie de l'information
Research subject en
Software and development
Research subject fr
Logiciels et développement
Partie
1
Nb partie
1
Application title
Analyzing causes of test failures
Keyword
NSERC
Application summary
In this work, we propose to carry out a systematic analysis of test failures and suggest strategies to mitigate test flakiness. The work would investigate symptoms as observed in test execution logs, as well as potential causes including recent code changes; changes in tests' dependencies; and plugin load orders. It would also investigate correlations between test types and failure rates: for instance, some of Arista's tests are based on state machines. Do such tests fail more or less often than other tests? The immediate outcome of the work would be a diagnosis of the current state of failing tests along with suggestions to increase the reliability of the test suite. We also propose to build models of test cases which would encapsulate the reasons that they fail.