Grant: Alliance Grants
University of Toronto
$239,690.00
Efficient log data compression and analytics system
EF 2022-23
Résumé
The pandemic has brought a furious growth of computer log data, i.e., the runtime data generated by computer systems. This is catalyzed by two trends. First, during the pandemic many activities are moved to online, hence there is an unprecedented growth of software services, which, in turn, generate an unprecedented amount of log data. In addition, software companies are increasingly relying on analysis of data to make data-driven decisions. They build large scale big data analytics systems that run hundreds of jobs per day. These data analytics systems end up generating even more log data. As a result, it is common for a company to generate Petabytes of log data per day, with 50% - 100% year-over-year growth.This unprecedented amount of log data creates a number of new challenges. First, conventional log analytics systems no longer scale to such large data sizes. In addition, managing such large data is extremely costly at every level, from storage cost, to network bandwidth, and to the cost of compute resources. This research proposes a novel system called CLP (Compressed Log Processor). CLP compresses the logs to unprecedented compression ratio, and more importantly, it allows one to search the compressed logs without decompression. The success of CLP will reduce the cost of log management and storage by over 40x, saving internet companies hundreds of millions of dollars per year. More importantly, it enables users to quickly search through Petabytes of logs efficiently, something that was not possible. The partnership with Uber will make CLP production-ready, and establish Canada as a leader in this emerging area of scalable and efficient log management. It will also result in increased employments of skilled software engineers in Canada, significant enhancements of the competitiveness of Canada's data center technology, and reduced emission of Canadian data centers.
Programme
Alliance Grants
Bénéficiaire
Emplacement
Ontario, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data
Tel que publié
Application id
570524-2021
Name nom
Yuan, Ding D
Department département
Electrical & Computer Engineering, Edward S. Rogers Sr. Department of
Organization id
31
Institution établissement
University of Toronto
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2022
Competition year année de concours
2021
Award amount
239690
Program id
ALLRP
Program name en
Alliance Grants
Program name fr
Subventions Alliance
Group en
RESEARCH PARTNERSHIPS
Group fr
PARTENARIATS DE RECHERCHE
Committee code
96
Committee name en
RPP Internal Decision Cttee
Committee name fr
Cté de décision interne - PPR
Area of application code
9999
Area of application group en
Not available
Area of application group fr
Non disponible
Area of application en
Not available
Area of application fr
Non disponible
Research subject code
2720
Research subject group en
Information technology
Research subject group fr
Technologie de l'information
Research subject en
Computer systems software
Research subject fr
Logiciel de systèmes informatiques
Installment
inst:2
Partie
1
Nb partie
2
Application title
Efficient log data compression and analytics system
Application summary
The pandemic has brought a furious growth of computer log data, i.e., the runtime data generated by computer systems. This is catalyzed by two trends. First, during the pandemic many activities are moved to online, hence there is an unprecedented growth of software services, which, in turn, generate an unprecedented amount of log data. In addition, software companies are increasingly relying on analysis of data to make data-driven decisions. They build large scale big data analytics systems that run hundreds of jobs per day. These data analytics systems end up generating even more log data. As a result, it is common for a company to generate Petabytes of log data per day, with 50% - 100% year-over-year growth.This unprecedented amount of log data creates a number of new challenges. First, conventional log analytics systems no longer scale to such large data sizes. In addition, managing such large data is extremely costly at every level, from storage cost, to network bandwidth, and to the cost of compute resources. This research proposes a novel system called CLP (Compressed Log Processor). CLP compresses the logs to unprecedented compression ratio, and more importantly, it allows one to search the compressed logs without decompression. The success of CLP will reduce the cost of log management and storage by over 40x, saving internet companies hundreds of millions of dollars per year. More importantly, it enables users to quickly search through Petabytes of logs efficiently, something that was not possible. The partnership with Uber will make CLP production-ready, and establish Canada as a leader in this emerging area of scalable and efficient log management. It will also result in increased employments of skilled software engineers in Canada, significant enhancements of the competitiveness of Canada's data center technology, and reduced emission of Canadian data centers.