Grant: Discovery Grants Program - Individual
Concordia University
$20,000.00
A Scalable Middleware for Coordinating Data Streams on Clouds
FY 2014-15
Summary
This research explores the problem of scalable coordination of data streams in many domain applications, such as Smart Grids to support diverse information sources. In such a system, besides sensors and instruments monitoring the infrastructure to produce an avalanche of data, information on electrical equipment, organizations, market regulations and weather are also engaged to gain situation awareness. Coordination involves synchronization, dissemination, and aggregation of multi-source, continuous data streams in and between data processing operations. A coordination process is formulated as a graph of data flows that running operators in cloud programming models. To achieve scalable coordination, this proposal advocates two key principles: first, incremental data stream processing should be introduced to the coordination middleware to handle frequent and small updates; secondly, since data frequency, format, quality, causality and synchronization criteria are diverse, skew often occurs in data processing operations. When skew arises, a portion of a data flow takes significantly longer time to process the task than others and thus slows down the entire application. Therefore, detecting skews and adapting the coordination middleware play another key role in achieving overall scalability of a domain application. The long term objective is to investigate a coordination platform to define, manage and resolve coordination of multi-source data streams, and contribute to industry standards involving broad distributed data services with enhanced middleware architecture and software components. The short term goal is to elaborate a scalable coordination middleware that engages incremental data processing, monitoring and skew detection into a whole control plane to improve responsiveness and scalability. This research program focuses on four integral aspects: (1) incremental data processing that is embedded in the coordination middleware. A dynamic load distribution algorithm triggers operators on incremental data and balances the computation and communication costs in the data flow; (2) skew detection. This requires monitoring methods and statistical models to determine a skew at runtime; (3) enhanced control plane. The control plane utilizes the monitoring and skew detection components and triggers reaction of the coordinate process when a skew occurs; and (4) middleware architecture facilitates communications in the coordination loop of monitoring, detecting and adapting. The architecture is evaluated under given uncertainties in the system architecture design, data characters, communication infrastructure and cloud environment. The research is applicable to a variety of practical applications in smart grids, cyber-physical systems, tele-communication and mobile services, health care monitoring and social networking. The research results will be disseminated through conferences and journal papers. The software will also be made open source to the public to help application specific development. More important, the award of this proposal will help provide HQP training for graduate students to complete six theses (two PhD and four MASc). In addition, the award will also help establish a strong position to a long-term collaboration with Ericsson in Montreal on developing a general monitoring and diagnosis framework for high available cloud services in the telecommunication domain.
Program
Discovery Grants Program - Individual
Recipient
Location
Quebec, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
RGPIN-2014-06254
Name nom
Liu, Yan
Department département
Electrical and Computer Engineering
Organization id
70
Institution établissement
Concordia University
Province en
Québec
Province fr
Québec
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2014
Competition year année de concours
2014
Award amount
20000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
Discovery grants programs
Group fr
Programmes de subventions à la découverte
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
801
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Computer software
Area of application fr
Logiciel
Research subject code
2704
Research subject group en
Information technology
Research subject group fr
Technologie de l'information
Research subject en
Distributed and parallel processing
Research subject fr
Traitement réparti et simultané
Partie
1
Nb partie
5
Application title
A Scalable Middleware for Coordinating Data Streams on Clouds
Keyword
NSERC
Application summary
This research explores the problem of scalable coordination of data streams in many domain applications, such as Smart Grids to support diverse information sources. In such a system, besides sensors and instruments monitoring the infrastructure to produce an avalanche of data, information on electrical equipment, organizations, market regulations and weather are also engaged to gain situation awareness. Coordination involves synchronization, dissemination, and aggregation of multi-source, continuous data streams in and between data processing operations. A coordination process is formulated as a graph of data flows that running operators in cloud programming models. To achieve scalable coordination, this proposal advocates two key principles: first, incremental data stream processing should be introduced to the coordination middleware to handle frequent and small updates; secondly, since data frequency, format, quality, causality and synchronization criteria are diverse, skew often occurs in data processing operations. When skew arises, a portion of a data flow takes significantly longer time to process the task than others and thus slows down the entire application. Therefore, detecting skews and adapting the coordination middleware play another key role in achieving overall scalability of a domain application. The long term objective is to investigate a coordination platform to define, manage and resolve coordination of multi-source data streams, and contribute to industry standards involving broad distributed data services with enhanced middleware architecture and software components. The short term goal is to elaborate a scalable coordination middleware that engages incremental data processing, monitoring and skew detection into a whole control plane to improve responsiveness and scalability. This research program focuses on four integral aspects: (1) incremental data processing that is embedded in the coordination middleware. A dynamic load distribution algorithm triggers operators on incremental data and balances the computation and communication costs in the data flow; (2) skew detection. This requires monitoring methods and statistical models to determine a skew at runtime; (3) enhanced control plane. The control plane utilizes the monitoring and skew detection components and triggers reaction of the coordinate process when a skew occurs; and (4) middleware architecture facilitates communications in the coordination loop of monitoring, detecting and adapting. The architecture is evaluated under given uncertainties in the system architecture design, data characters, communication infrastructure and cloud environment. The research is applicable to a variety of practical applications in smart grids, cyber-physical systems, tele-communication and mobile services, health care monitoring and social networking. The research results will be disseminated through conferences and journal papers. The software will also be made open source to the public to help application specific development. More important, the award of this proposal will help provide HQP training for graduate students to complete six theses (two PhD and four MASc). In addition, the award will also help establish a strong position to a long-term collaboration with Ericsson in Montreal on developing a general monitoring and diagnosis framework for high available cloud services in the telecommunication domain.