Grant: Collaborative Research and Development Grants
Queen's University
$40,000.00
Hybrid Model for Optimization of Large-Scale Test Execution
EF 2014-15
Résumé
Modern companies like Intuit, Google and Mozilla often release a new version of their software every two weeks while start-ups like IMVU can release a new version up to 50 times a day. To enable rapid releases without sacrificing the quality of the software product, companies automatically execute a complete suite of tests after each change so that the developer responsible for a detected defect can be identified and notified immediately. Unfortunately, less time between releases means less time to execute tests. As a result, researchers have proposed optimization techniques to identify and execute only the most relevant tests. Unfortunately, these techniques have only been applied to small benchmark systems with limited validation on real industrial software. This creates a problem when test optimization techniques face a steady stream of changes. In the real world, many companies are hampered with a large-scale legacy code base in which quiet periods with few changes are interleaved with sudden bursts of major changes. The changes in quiet periods are not representative of those in busy periods so that the probability of detecting a sudden defect after a long quiet period becomes increasingly small. Sudden defects are costly, both in terms of repair time and of economic losses to the company. Our project will build hybrid techniques that will be able to identify, for a given code change: (1) those tests that are expected to fail and (2) a prioritized list of tests that might fail unexpectedly. Developers would run all tests of (1) and as many tests of (2) as time allows. We will evaluate our approach on real-world, large test suites provided by our industrial partner. Our approach to test suite optimization will allow the Canadian software industry to fully embrace the rapid release movement while reducing software quality costs. As a result, end users will have rapid access to new, defect-free, functionality. The two HQP trained through this project will gain skills in empirical software engineering, in test suite hybrid optimization techniques and essential communication and presentation skills.
Programme
Collaborative Research and Development Grants
Bénéficiaire
Emplacement
Ontario, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data
Tel que publié
Application id
451570-2013
Name nom
Hassan, Ahmed
Department département
Computing, School of
Organization id
29
Institution établissement
Queen's University
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2014
Competition year année de concours
2013
Award amount
40000
Program id
CRDPJ
Program name en
Collaborative Research and Development Grants
Program name fr
Subventions de recherche et développement coopérative
Group en
Research partnerships programs
Group fr
Programmes de partenariat
Committee code
96
Committee name en
RPP Internal Decision Cttee
Committee name fr
Cté de décision interne - PPR
Area of application code
802
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Information systems and technology
Area of application fr
Systèmes d'information et technologie de l'information
Research subject code
2705
Research subject group en
Information technology
Research subject group fr
Technologie de l'information
Research subject en
Software and development
Research subject fr
Logiciels et développement
Partie
2
Nb partie
2
Application title
Hybrid Model for Optimization of Large-Scale Test Execution
Keyword
NSERC
Application summary
Modern companies like Intuit, Google and Mozilla often release a new version of their software every two weeks while start-ups like IMVU can release a new version up to 50 times a day. To enable rapid releases without sacrificing the quality of the software product, companies automatically execute a complete suite of tests after each change so that the developer responsible for a detected defect can be identified and notified immediately. Unfortunately, less time between releases means less time to execute tests. As a result, researchers have proposed optimization techniques to identify and execute only the most relevant tests. Unfortunately, these techniques have only been applied to small benchmark systems with limited validation on real industrial software. This creates a problem when test optimization techniques face a steady stream of changes. In the real world, many companies are hampered with a large-scale legacy code base in which quiet periods with few changes are interleaved with sudden bursts of major changes. The changes in quiet periods are not representative of those in busy periods so that the probability of detecting a sudden defect after a long quiet period becomes increasingly small. Sudden defects are costly, both in terms of repair time and of economic losses to the company. Our project will build hybrid techniques that will be able to identify, for a given code change: (1) those tests that are expected to fail and (2) a prioritized list of tests that might fail unexpectedly. Developers would run all tests of (1) and as many tests of (2) as time allows. We will evaluate our approach on real-world, large test suites provided by our industrial partner. Our approach to test suite optimization will allow the Canadian software industry to fully embrace the rapid release movement while reducing software quality costs. As a result, end users will have rapid access to new, defect-free, functionality. The two HQP trained through this project will gain skills in empirical software engineering, in test suite hybrid optimization techniques and essential communication and presentation skills.