Canada Spends Logo

Explorez notre analyse du budget 2026

Grant: Collaborative Research and Development Grants

University of Calgary

$68,000.00

Performance management tools for big data applications

EF 2017-18

Résumé
The proposed project will develop tools that will allow users and operators of big data systems to better analyze and optimize the performance of their applications. Enterprises are increasingly relying on big data cluster systems to support their analytics requirements. Newer generation big data platforms such as Apache Spark provide a rich set of Application Programming Interfaces (APIs) that allow data scientists to quickly develop and deploy analytics applications. However, despite recent advances in big data platforms, analyzing and optimizing the performance of big data applications remains a challenge. Users and operators of big data clusters often face challenges in understanding the reasons for poor application performance and optimizing applications and clusters to meet performance requirements. To address these challenges, the proposed project will focus on two long term objectives. First, the project will devise an intuitive and intelligent interface that will support rapid performance analysis and optimization of big data applications. Using the interface, a user will be able to quickly identify bottlenecks that limit an application's performance. The interface can also be used to visualize and understand an application's performance scalability. Such features can in turn lead to increased productivity for users and more efficient usage of clusters for operators. Second, the project will develop automated service level management techniques for big data systems. Typically, big data systems use schedulers to control the allotment of resources to applications. Although production scheduling systems are sophisticated, they currently lack the ability to drive scheduling decisions so as to automatically meet user service level requirements, e.g., a specified deadline for completing an application, while simultaneously achieving efficient resource usage. Consequently, users and cluster operators often resort to tedious trial and error approaches, which can cause both missed service levels as well as inefficient resource usage. The proposed project will devise new scheduling techniques to address this limitation.
Programme
Collaborative Research and Development Grants
Bénéficiaire
Emplacement
Alberta, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data

Tel que publié

Application id
513200-2017
Name nom
Krishnamurthy, Diwakar
Department département
Electrical and Computer Engineering
Organization id
11
Institution établissement
University of Calgary
Province en
Alberta
Province fr
Alberta
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2017
Competition year année de concours
2017
Award amount
68000
Program id
CRDPJ
Program name en
Collaborative Research and Development Grants
Program name fr
Subventions de recherche et développement coopérative
Group en
Research partnerships programs
Group fr
Programmes de partenariat
Committee code
96
Committee name en
RPP Internal Decision Cttee
Committee name fr
Cté de décision interne - PPR
Area of application code
802
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Information systems and technology
Area of application fr
Systèmes d'information et technologie de l'information
Research subject code
2720
Research subject group en
Information technology
Research subject group fr
Technologie de l'information
Research subject en
Computer systems software
Research subject fr
Logiciel de systèmes informatiques
Partie
1
Nb partie
1
Application title
Performance management tools for big data applications
Keyword
NSERC
Application summary
The proposed project will develop tools that will allow users and operators of big data systems to better analyze and optimize the performance of their applications. Enterprises are increasingly relying on big data cluster systems to support their analytics requirements. Newer generation big data platforms such as Apache Spark provide a rich set of Application Programming Interfaces (APIs) that allow data scientists to quickly develop and deploy analytics applications. However, despite recent advances in big data platforms, analyzing and optimizing the performance of big data applications remains a challenge. Users and operators of big data clusters often face challenges in understanding the reasons for poor application performance and optimizing applications and clusters to meet performance requirements. To address these challenges, the proposed project will focus on two long term objectives. First, the project will devise an intuitive and intelligent interface that will support rapid performance analysis and optimization of big data applications. Using the interface, a user will be able to quickly identify bottlenecks that limit an application's performance. The interface can also be used to visualize and understand an application's performance scalability. Such features can in turn lead to increased productivity for users and more efficient usage of clusters for operators. Second, the project will develop automated service level management techniques for big data systems. Typically, big data systems use schedulers to control the allotment of resources to applications. Although production scheduling systems are sophisticated, they currently lack the ability to drive scheduling decisions so as to automatically meet user service level requirements, e.g., a specified deadline for completing an application, while simultaneously achieving efficient resource usage. Consequently, users and cluster operators often resort to tedious trial and error approaches, which can cause both missed service levels as well as inefficient resource usage. The proposed project will devise new scheduling techniques to address this limitation.