Grant: Discovery Launch Supplement
University of Waterloo
$12,500.00
Execution-Guided Machine Learning Models for Test Generation and Maintenance
EF 2024-25
Résumé
Context. Software testing is an essential and widely-used technique for assessing software reliability and safety. Developers need to invest substantial effort to write and maintain tests, taking up approximately 50% of their time. Given the pressure to release software at a rapid pace, tests may remain incomplete (not covering all software behaviors) or outdated (not reflecting the latest software changes), leaving room for undetected software bugs. Recent advances in machine learning, especially large language models, present new opportunities to assist developers in obtaining high-quality tests by learning from the large number of existing tests. However, the unique structures and semantics of software code and tests are difficult to capture since machine learning models have been designed for processing natural language text. Our recent work addresses this challenge by integrating the semantics of software execution and software evolution with machine learning, resulting in execution-guided and evolution-aware models, such as TeCo for test completion and CoditT5 for editing code and natural language elements in software. Yet challenges remain when applying these models to testing: (1) many tests generated by machine learning models are not executable; (2) current solutions focus on generating new tests but ignore the need for evolving existing tests; (3) developers have little control on which part of software behaviors to generate tests for.
Objectives. My long-term research goal is boosting developers' productivity in software development, testing, and maintenance, leveraging a combination of software engineering techniques and machine learning models. This proposal focuses on developers' productivity in testing. Specifically, we propose the following three research objectives:
RO1: Design execution-guided models for robustly generating executable tests, by pretraining large language models with execution data and interleaving execution into models' generation workflow.
RO2: Combine execution-guided and evolution-aware models to support test maintenance tasks, such as updating tests when the associated code changes (and vice versa) and refining tests to improve their quality.
RO3: Bridge test generation models and natural language elements in software (e.g., comments and code reviews), so that developers can easily control and interact with the models.
Impact. Our research serves the 45K software companies in the Canadian information and communications technologies industry ($104.5B yearly GDP), by implementing high-quality test generation and maintenance techniques and recovering billions of dollars lost due to software bugs. The design of execution-guided and evolution-aware models in this proposal is transferrable to models for other software engineering tasks. The 11 students involved in this proposal will become experts in the cutting-edge interdisciplinary area of software engineering and machine learning.
Programme
Discovery Launch Supplement
Bénéficiaire
Emplacement
Ontario, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data
Tel que publié
Application id
DGECR-2024-00351
Name nom
Nie, Pengyu
Department département
David R. Cheriton School of Computer Science
Organization id
33
Institution établissement
University of Waterloo
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2024
Award amount
12500
Program id
DGECR
Program name en
Discovery Launch Supplement
Program name fr
Supplément Tremplin vers la découverte
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
801
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Computer software
Area of application fr
Logiciel
Research subject code
2706
Research subject group en
Information technology
Research subject group fr
Technologie de l'information
Research subject en
Software engineering
Research subject fr
Génie logiciel
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Execution-Guided Machine Learning Models for Test Generation and Maintenance
Keywords
test generation; test maintenance; execution-guided machine learning models; evolution-aware machine learning models; machine learning; large language models; machine learning for software engineering
Application summary
Context. Software testing is an essential and widely-used technique for assessing software reliability and safety. Developers need to invest substantial effort to write and maintain tests, taking up approximately 50% of their time. Given the pressure to release software at a rapid pace, tests may remain incomplete (not covering all software behaviors) or outdated (not reflecting the latest software changes), leaving room for undetected software bugs. Recent advances in machine learning, especially large language models, present new opportunities to assist developers in obtaining high-quality tests by learning from the large number of existing tests. However, the unique structures and semantics of software code and tests are difficult to capture since machine learning models have been designed for processing natural language text. Our recent work addresses this challenge by integrating the semantics of software execution and software evolution with machine learning, resulting in execution-guided and evolution-aware models, such as TeCo for test completion and CoditT5 for editing code and natural language elements in software. Yet challenges remain when applying these models to testing: (1) many tests generated by machine learning models are not executable; (2) current solutions focus on generating new tests but ignore the need for evolving existing tests; (3) developers have little control on which part of software behaviors to generate tests for.
Objectives. My long-term research goal is boosting developers' productivity in software development, testing, and maintenance, leveraging a combination of software engineering techniques and machine learning models. This proposal focuses on developers' productivity in testing. Specifically, we propose the following three research objectives:
RO1: Design execution-guided models for robustly generating executable tests, by pretraining large language models with execution data and interleaving execution into models' generation workflow.
RO2: Combine execution-guided and evolution-aware models to support test maintenance tasks, such as updating tests when the associated code changes (and vice versa) and refining tests to improve their quality.
RO3: Bridge test generation models and natural language elements in software (e.g., comments and code reviews), so that developers can easily control and interact with the models.
Impact. Our research serves the 45K software companies in the Canadian information and communications technologies industry ($104.5B yearly GDP), by implementing high-quality test generation and maintenance techniques and recovering billions of dollars lost due to software bugs. The design of execution-guided and evolution-aware models in this proposal is transferrable to models for other software engineering tasks. The 11 students involved in this proposal will become experts in the cutting-edge interdisciplinary area of software engineering and machine learning.