Grant: Discovery Grants Program - Individual

For a range of major scientific computing challenges that span fundamental and applied science, the deployment of Big Data Analytics (hereafter BDAs) on a large-scale system, such as an internal or external cloud, a cluster or even distributed public resources (crowd computing“), needs to be offered with guarantees of predictable performance and utilization cost. Currently, however, this is not possible, because scientific communities lack the technology, both at the level of modelling and analytics, that identifies the key characteristics of BDAs and their impact on performance. There is also little data or simulations available that address the role of the system operation and infrastructure in defining overall performance. The overall goal of the proposed research is to fill this gap by producing a deeper understanding of how to optimize the deployment of BDAs that run on systems operating on large infrastructures, and even hybrid infrastructures, in order to achieve optimal performance, while taking into account running costs. Overall, we will achieve this using a novel combination of big data analytics and modeling results.Our research will involve the modeling of BDAs with respect to dimensions of workload, data and resources, and profiling of BDAs with respect to the proposed modeling. We will explore alternative systems for the deployment of BDAs, ranging from private clusters, to cloud computing and crowd computing. We will develop a methodology for the performance prediction of BDAs deployed in hybrid systems of cloud, cluster and crowd. This will be achieved with the creation of a multi-agent utilization model and the analysis of the computing environment with numerical and analytical methods. We will employ the predictions to create schemes for performance optimization with respect to cost limitations for system utilization. The schemes will accommodate execution by adapting, i.e. expanding or hybridizing, the system. In our research we will observe and experiment with a range of scientific and business applications of BDAs, as well as a variety of systems, which present qualitative and quantitative differences. These will give us the opportunity to create solutions that are applicable to wide range of BDAs environments, but also mine the limitations of performance optimization based on generic guidelines for profiling, prediction and deployment, with respect to the characteristics of BDAs and the characteristics of the system.