Grant: Discovery Grants Program - Individual

My research begins by listening to the daily struggles of software developers and ultimately provides theoretically based general solutions to common software testing problems. Objective 1, Flaky tests: Flaky software tests fail without identifying a fault and waste developer time and reduce confidence in test outcomes. Flaky tests affect all projects and are particularly damaging on large projects or in complex test environments. The conventional practice is to re-run flaky tests or to quarantine them and require developers to fix them. However, I have observed that flaky tests have varying degrees of flakiness. For example, a test that has a flaky failure 1 in 100 runs is more reliable than a test that has 10 flaky failures 100 runs. By quantifying the historical FlakeRate, e.g. 10/100, we can use statistical tests to determine when a test is failing at its normal FlakeRate, 10%, or when it FlakeRate has changed, indicating instability and a potential product fault. My objective is to quantify the degree of test flakiness from prior runs and statistically identify changes in the FlakeRate as indications that a test has become unstable. Tool support will allow developers to rank tests by FlakeRate instability. This should reduce investigation effort because flaky failures that do not have a statistically significant corresponding change in FlakeRate are stable and less likely to need investigation. Objective 2, Commit failure risk: A common development practice is to test each commit individually to isolate immediately the commit that caused a test failure. Since most tests do not fail and most commits pass all tests, we propose to batch commits based on how likely a commit is to contain a failing test. A batch that contains 10 commits, will require 1 run of the test suite saving 9 executions over the test-all-passing scenario. However, on failure, a bisection process will require additional runs. We will adapt commit risk models that identify bug-introducing changes to the context of selecting low risk commits to be included in large batches and high risk commits that should be tested individually. The adaption and development of these statistical approaches is complex and evaluating them in production CI environments requires systematic rigor. Objective 3, Generalize and Tools: In the long-term, I will increase the use of software analytics by software testers and developers by simulating and demonstrating improved efficiency in a wide range of software environments. In my experience developers have a single focus of reducing faults and releasing software on time. This focus has limited the use of mining and analytics in practice. By focusing on developer problems, I have been able to and will continue to provide general solutions to time consuming, unpleasant, and expensive development problems.