Canada Spends Logo

Explorez notre analyse du budget 2026

Grant: Discovery Grants Program - Individual

University of Toronto

$21,191.66

Context-based Pattern Recognition for Automating Big Data Management in Clouds

EF 2023-24

Résumé
We propose to build data models for real-time, on-the-fly pattern recognition in the large volumes of operational data produced by Cloud environments. Our pattern recognition techniques will be applied to a range of automated management tasks for Cloud environments, such as, anomaly diagnosis, and resource provisioning. We will also investigate the extensibility and applicability of our pattern recognition techniques to other types of big data, such as, biomedical and biometric data hosted on Cloud environments.Objectives1. Data CollectionWe will capture various types of Cloud operational data and tag them with the high level application scope in which they occur, at the collection points, for all the application scopes of interest e.g., workload phase, parametrized request, or method invocation. The types of operational data we plan to capture and investigate include routine logging and monitoring information, such as, throughput, latency, dependability, or other application QoS metrics, as well as log statements and resource consumption time series data for CPU, network bandwidth, disk and memory consumption.2. Pattern Learning and RecognitionAs data is collected, one or more statistical learning techniques are deployed in order to learn mathematical data models, per context. For contexts that statistically diverge from the learned data model, we will trigger an automated system adaptation e.g., in terms of resource allocation and/or provide context-rich problem digests for human inspection.MethodologyOur key idea is to leverage application contexts within our on-the-fly pattern learning and recognition framework, by explicitly augmenting routine system logging and monitoring data collection with the application contexts where these data occur. We expect that several, complementary data modelling and analysis methods will be applied jointly, or independently, for the various contexts, e.g., for the same application scope, for different scopes of the same granularity, and/or for nested scopes of different granularities. Different methods may work better for different data and different contexts. For example, a classification method based on clustering may work well for log template data, but a Neural Network method for pattern matching in time series data for resource consumption could provide better pattern extraction, learning and matching accuracy. Both methods could provide higher accuracy for anomaly detection than either method alone. In order to detect application contexts we currently use static analysis on application source code. However, we plan on including more general techniques based on binary code inspection as our project progresses. In our experimental evaluation of our prototype, we will use open-source systems in our Cloud software stack and experiment with clustering, dynamic time warp, Neural Networks and other statistical learning methods.
Programme
Discovery Grants Program - Individual
Bénéficiaire
Emplacement
Ontario, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data

Tel que publié

Application id
RGPIN-2017-06925
Name nom
Amza, Cristiana
Department département
Electrical and Computer Engineering
Organization id
31
Institution établissement
University of Toronto
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2023
Competition year année de concours
2017
Award amount
21191.66
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
802
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Information systems and technology
Area of application fr
Systèmes d'information et technologie de l'information
Research subject code
9999
Research subject group en
Not available
Research subject group fr
Not available
Research subject en
Non disponible
Research subject fr
Non disponible
Application title
Context-based Pattern Recognition for Automating Big Data Management in Clouds
Keyword
anomaly detection; automated learning; Big data analytics; Cloud computing; context awareness; dynamic adaptation; experimental evaluation; on-the-fly system modelling; pattern recognition; performance and reliability
Application summary
We propose to build data models for real-time, on-the-fly pattern recognition in the large volumes of operational data produced by Cloud environments. Our pattern recognition techniques will be applied to a range of automated management tasks for Cloud environments, such as, anomaly diagnosis, and resource provisioning. We will also investigate the extensibility and applicability of our pattern recognition techniques to other types of big data, such as, biomedical and biometric data hosted on Cloud environments.Objectives1. Data CollectionWe will capture various types of Cloud operational data and tag them with the high level application scope in which they occur, at the collection points, for all the application scopes of interest e.g., workload phase, parametrized request, or method invocation. The types of operational data we plan to capture and investigate include routine logging and monitoring information, such as, throughput, latency, dependability, or other application QoS metrics, as well as log statements and resource consumption time series data for CPU, network bandwidth, disk and memory consumption.2. Pattern Learning and RecognitionAs data is collected, one or more statistical learning techniques are deployed in order to learn mathematical data models, per context. For contexts that statistically diverge from the learned data model, we will trigger an automated system adaptation e.g., in terms of resource allocation and/or provide context-rich problem digests for human inspection.MethodologyOur key idea is to leverage application contexts within our on-the-fly pattern learning and recognition framework, by explicitly augmenting routine system logging and monitoring data collection with the application contexts where these data occur. We expect that several, complementary data modelling and analysis methods will be applied jointly, or independently, for the various contexts, e.g., for the same application scope, for different scopes of the same granularity, and/or for nested scopes of different granularities. Different methods may work better for different data and different contexts. For example, a classification method based on clustering may work well for log template data, but a Neural Network method for pattern matching in time series data for resource consumption could provide better pattern extraction, learning and matching accuracy. Both methods could provide higher accuracy for anomaly detection than either method alone. In order to detect application contexts we currently use static analysis on application source code. However, we plan on including more general techniques based on binary code inspection as our project progresses. In our experimental evaluation of our prototype, we will use open-source systems in our Cloud software stack and experiment with clustering, dynamic time warp, Neural Networks and other statistical learning methods.