Canada Spends Logo

Explorez notre analyse du budget 2026

Grant: Engage Grants Program

University of British Columbia

$25,000.00

Optimizing compute task scheduling at Shopify

EF 2017-18

Résumé
Big Data processing frameworks allow enterprises to drive their business based on key insights extracted fromnear real-time data. Apache Spark is a framework that Shopify leverages to collect and analyze customerinformation and expose data insights to the shop owners who are users of the Shopify platform. For example,the enterprise which built its online store using the Shopify platform can send pre-set queries to the platform tolearn about the impact of their recent marketing campaign (e.g., with an improved Google ad keywords) bygeographical region or by age group, and see if the marketing campaign resulted in actual product sales.Shopify uses Spark to analyze data by passing the data through a processing pipeline. The pipeline consists ofmultiple jobs, each applying some computation to the data. A set of connected jobs form a flow which mightcorrespond to the query exposed to the shop owner. In this project we will explore algorithmic solutions tomaintain a useful flow execution frequency while absorbing the variance caused by job failures, and ensuringco-execution of pre-scheduled and ad-hoc jobs without resource starvation. We intend to apply approachesbased on heuristics and constraint satisfaction, study their tradeoffs, and develop a solution that works best inpractice.
Programme
Engage Grants Program
Emplacement
British Columbia, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data

Tel que publié

Application id
514614-2017
Name nom
Beschastnikh, Ivan
Department département
Computer Science
Organization id
2
Institution établissement
University of British Columbia
Province en
British Columbia
Province fr
Colombie-Britannique
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2017
Competition year année de concours
2017
Award amount
25000
Program id
EGP
Program name en
Engage Grants Program
Program name fr
Programme de subventions d'engagement partenariat
Group en
Research partnerships programs
Group fr
Programmes de partenariat
Committee code
1555
Committee name en
Prairie Internal Decision Committee
Committee name fr
Comité de décision interne pour les Prairies
Area of application code
802
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Information systems and technology
Area of application fr
Systèmes d'information et technologie de l'information
Research subject code
2710
Research subject group en
Information technology
Research subject group fr
Technologie de l'information
Research subject en
Information systems design
Research subject fr
Conception de systèmes d'information
Partie
1
Nb partie
1
Application title
Optimizing compute task scheduling at Shopify
Keyword
NSERC
Application summary
Big Data processing frameworks allow enterprises to drive their business based on key insights extracted fromnear real-time data. Apache Spark is a framework that Shopify leverages to collect and analyze customerinformation and expose data insights to the shop owners who are users of the Shopify platform. For example,the enterprise which built its online store using the Shopify platform can send pre-set queries to the platform tolearn about the impact of their recent marketing campaign (e.g., with an improved Google ad keywords) bygeographical region or by age group, and see if the marketing campaign resulted in actual product sales.Shopify uses Spark to analyze data by passing the data through a processing pipeline. The pipeline consists ofmultiple jobs, each applying some computation to the data. A set of connected jobs form a flow which mightcorrespond to the query exposed to the shop owner. In this project we will explore algorithmic solutions tomaintain a useful flow execution frequency while absorbing the variance caused by job failures, and ensuringco-execution of pre-scheduled and ad-hoc jobs without resource starvation. We intend to apply approachesbased on heuristics and constraint satisfaction, study their tradeoffs, and develop a solution that works best inpractice.