Canada Spends Logo

Explore our Budget 2026 analysis

Grant: Research Tools and Instruments

Queen's University

$129,760.00

An Experimental Platform to Comparatively Evaluate Large Pre-trained Models on SE Tasks

FY 2023-24

Summary
In an era marked by rapid advancements in large pre-trained models (PTMs) for automating software engineering (SE) tasks, a critical necessity arises for a fully configurable, scalable computing platform to evaluate and optimize the capabilities of PTMs in SE domains in the context of research collaborations with Canadian industry. The specialized PTM computing platform outlined by this proposal bridges this technological gap and nurtures innovation within Canadian SE industry, while fostering a rich, hands-on learning environment for the highly qualified personnel (HQP) at Queen's University. The proposed platform is equipped with high-performance GPU servers, aiding in prompt engineering, fine-tuning, and empirical evaluation of open-source large PTMs for SE tasks. The equipment will serve as the backbone for three key research activities: advancing novel prompt engineering and fine-tuning methodologies, enabling controlled, secure empirical evaluation and selection of PTMs, and ensuring effective deployment and monitoring of optimized PTMs. These activities are unique and fundamental for addressing the challenges faced by Canadian companies in optimizing PTMs to their specific contexts, which is often resource-intensive and costly. The proposed platform will significantly impact the training and practical exposure of around 30 HQP annually, across two labs directly and several other SE and AI research labs indirectly within the School of Computing. Through collaborative workshops, a graduate-level course, and real-world SE challenges, the platform will provide hands-on experience in managing (via direct access to the infrastructure of this computing platform), deploying, and optimizing PTMs, thereby furnishing HQP with marketable skills essential in the intertwined realms of SE and AI.
Program
Research Tools and Instruments
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data

As published

Application id
RTI-2024-00166
Name nom
Tian, Yuan
Department département
School of Computing
Organization id
29
Institution établissement
Queen's University
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2023
Competition year année de concours
2024
Award amount
129760
Program id
RTI
Program name en
Research Tools and Instruments
Program name fr
Outils et instruments de recherche
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1607
Committee name en
Computer, Mathematical and Statistical Sciences (RTI)
Committee name fr
Sciences informatiques, mathématiques et statistiques (OIR)
Area of application code
801
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Computer software
Area of application fr
Logiciel
Research subject code
2700
Research subject group en
Information technology
Research subject group fr
Information technology
Research subject en
Technologie de l'information
Research subject fr
Technologie de l'information
Application title
An Experimental Platform to Comparatively Evaluate Large Pre-trained Models on SE Tasks
Keyword
Automated Software Engineering; Evaluation and Selection of Large Pre-trained Models; Experimental Platform; Large Pre-trained Model Reuse; Maintenance of Large Language Models in Production; Pre-trained Models for Code; Responsible AI
Application summary
In an era marked by rapid advancements in large pre-trained models (PTMs) for automating software engineering (SE) tasks, a critical necessity arises for a fully configurable, scalable computing platform to evaluate and optimize the capabilities of PTMs in SE domains in the context of research collaborations with Canadian industry. The specialized PTM computing platform outlined by this proposal bridges this technological gap and nurtures innovation within Canadian SE industry, while fostering a rich, hands-on learning environment for the highly qualified personnel (HQP) at Queen's University. The proposed platform is equipped with high-performance GPU servers, aiding in prompt engineering, fine-tuning, and empirical evaluation of open-source large PTMs for SE tasks. The equipment will serve as the backbone for three key research activities: advancing novel prompt engineering and fine-tuning methodologies, enabling controlled, secure empirical evaluation and selection of PTMs, and ensuring effective deployment and monitoring of optimized PTMs. These activities are unique and fundamental for addressing the challenges faced by Canadian companies in optimizing PTMs to their specific contexts, which is often resource-intensive and costly. The proposed platform will significantly impact the training and practical exposure of around 30 HQP annually, across two labs directly and several other SE and AI research labs indirectly within the School of Computing. Through collaborative workshops, a graduate-level course, and real-world SE challenges, the platform will provide hands-on experience in managing (via direct access to the infrastructure of this computing platform), deploying, and optimizing PTMs, thereby furnishing HQP with marketable skills essential in the intertwined realms of SE and AI.