Grant: Collaborative Research and Development Grants
University of Waterloo
$62,117.00
Reliable measurement contexts for empirical performance evaluation
EF 2014-15
Résumé
The usual yardstick for measuring results in systems research is performance data. For example, the question is whether a particular change in the compiler or the operating system increases or decreases the system throughput. However, recent results show that minor aspects can have a significant impact on these measurements. Consequently, the conclusions drawn from the measurements might be incorrect. So the fundamental question is how to produce reliable, robust, and repeatable measurements.The proposed projects targets this problem in collaboration with Google. We will explore the nature of measurement contexts in single computer experiments, and then combining the results in a public-service infrastructure. We will characterize measurement contexts to identify the key elements that must be reported, investigate assessment techniques for measurement contexts, and finally integrate this knowledge into a framework for empirical benchmarking. This framework will be a public-service infrastructure and of use to research in performance evaluation and the causal researcher performance empirical evaluations. Getting reliable, robust, and repeatable measurements is a key problem for Google to make well-informed decisions on their software architecture and hardware purchases.
Programme
Collaborative Research and Development Grants
Bénéficiaire
Emplacement
Ontario, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data
Tel que publié
Application id
462664-2013
Name nom
Fischmeister, Sebastian
Department département
Electrical & Computer Engineering
Organization id
33
Institution établissement
University of Waterloo
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2014
Competition year année de concours
2013
Award amount
62117
Program id
CRDPJ
Program name en
Collaborative Research and Development Grants
Program name fr
Subventions de recherche et développement coopérative
Group en
Research partnerships programs
Group fr
Programmes de partenariat
Committee code
96
Committee name en
RPP Internal Decision Cttee
Committee name fr
Cté de décision interne - PPR
Area of application code
801
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Computer software
Area of application fr
Logiciel
Research subject code
2720
Research subject group en
Information technology
Research subject group fr
Technologie de l'information
Research subject en
Computer systems software
Research subject fr
Logiciel de systèmes informatiques
Partie
1
Nb partie
3
Application title
Reliable measurement contexts for empirical performance evaluation
Keyword
NSERC
Application summary
The usual yardstick for measuring results in systems research is performance data. For example, the question is whether a particular change in the compiler or the operating system increases or decreases the system throughput. However, recent results show that minor aspects can have a significant impact on these measurements. Consequently, the conclusions drawn from the measurements might be incorrect. So the fundamental question is how to produce reliable, robust, and repeatable measurements.The proposed projects targets this problem in collaboration with Google. We will explore the nature of measurement contexts in single computer experiments, and then combining the results in a public-service infrastructure. We will characterize measurement contexts to identify the key elements that must be reported, investigate assessment techniques for measurement contexts, and finally integrate this knowledge into a framework for empirical benchmarking. This framework will be a public-service infrastructure and of use to research in performance evaluation and the causal researcher performance empirical evaluations. Getting reliable, robust, and repeatable measurements is a key problem for Google to make well-informed decisions on their software architecture and hardware purchases.