Grant: Discovery Grants Program - Individual
University of Toronto
$55,000.00
Practical, Efficient, and Intelligent Log Compression and Search
FY 2024-25
Summary
Software systems are vital in almost every aspect of our lives today, from powering Internet service like Google and Uber to running our cars and phones. Unfortunately, failures inevitably occur. Prolonged system outages not only are costly, but can have deadly consequences in safety critical systems. The software industry spends $244 billion on failure recovery annually, and programmers today spend the majority of their time diagnosing and fixing failures instead of developing new code.
Failure diagnosis is fundamentally a data problem. Engineers analyze diagnostic data stored in logs, which record important events during software's execution. Yet, there is a furious growth of the log data fueled by the explosion of Internet services. Large Internet services like Uber and Facebook can easily produce tens of Petabytes of logs per day. The annual storage cost alone of logs at this scale would exceed $100 million.
Existing log analytics tools operate by creating indexes of the logs. However, the size of the index is on the same order of magnitude as the original logs, making them too costly to be used for logs at petabyte scale and beyond.
Recently, we invented a log compression and search technique called CLP (Compressed Log Processor). CLP proposed a domain-specific algorithm that compresses the logs, and allows one to search the compressed logs without full decompression. It achieves an unprecedented compression ratio, outperforming general purpose compressors like Zstandard and Gzip by 2x. Search on CLP uses orders of magnitude less computing resources compared to conventional index-based log search tools. CLP has been open-sourced, and it has been deployed at large scale by industry giants including Uber and Walmart.
This research focuses on improving CLP to compress and search the logs. Specifically, it has three objectives. First, we plan to compress and search semi-structured logs, such as JSON, Protocol Buffers, and YAML. CLP only supports unstructured logs (i.e. free text). However, in practice, semi-structured logs are much more important. For example, at Uber, the size of JSON logs is 10x of the size of unstructured logs. We propose a system that can efficiently compress semi-structured logs, and allows search without decompression.
The second objective is to optimize CLP's handling of unstructured logs. Specifically, we aim to significantly improve its search performance; not only making it faster, but also more predictable. Our design should also improve the compression ratio.
The third objective is an intelligent log parser that can automatically extract the schema of any logs. CLP, as well as most log management tools, requires user-provided rules (i.e., schemas) to parse the logs. Manually providing schemas is cumbersome, and such manual effort is required every time the software evolves. We propose an algorithm that can automatically infer the schemas of the logs, eliminating the manual effort.
Program
Discovery Grants Program - Individual
Recipient
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
RGPIN-2024-04337
Name nom
Yuan, Ding
Department département
Electrical and Computer Engineering
Organization id
31
Institution établissement
University of Toronto
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2024
Competition year année de concours
2024
Award amount
55000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
801
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Computer software
Area of application fr
Logiciel
Research subject code
2720
Research subject group en
Information technology
Research subject group fr
Technologie de l'information
Research subject en
Computer systems software
Research subject fr
Logiciel de systèmes informatiques
Field of research list names en
Not available
Field of research list names fr
Non disponible
Socioeconomic objective list names en
Not available
Socioeconomic objective list names fr
Non disponible
Application title
Practical, Efficient, and Intelligent Log Compression and Search
Keywords
systems software; big data; log compression; log analytics; system management; observability
Application summary
Software systems are vital in almost every aspect of our lives today, from powering Internet service like Google and Uber to running our cars and phones. Unfortunately, failures inevitably occur. Prolonged system outages not only are costly, but can have deadly consequences in safety critical systems. The software industry spends $244 billion on failure recovery annually, and programmers today spend the majority of their time diagnosing and fixing failures instead of developing new code.
Failure diagnosis is fundamentally a data problem. Engineers analyze diagnostic data stored in logs, which record important events during software's execution. Yet, there is a furious growth of the log data fueled by the explosion of Internet services. Large Internet services like Uber and Facebook can easily produce tens of Petabytes of logs per day. The annual storage cost alone of logs at this scale would exceed $100 million.
Existing log analytics tools operate by creating indexes of the logs. However, the size of the index is on the same order of magnitude as the original logs, making them too costly to be used for logs at petabyte scale and beyond.
Recently, we invented a log compression and search technique called CLP (Compressed Log Processor). CLP proposed a domain-specific algorithm that compresses the logs, and allows one to search the compressed logs without full decompression. It achieves an unprecedented compression ratio, outperforming general purpose compressors like Zstandard and Gzip by 2x. Search on CLP uses orders of magnitude less computing resources compared to conventional index-based log search tools. CLP has been open-sourced, and it has been deployed at large scale by industry giants including Uber and Walmart.
This research focuses on improving CLP to compress and search the logs. Specifically, it has three objectives. First, we plan to compress and search semi-structured logs, such as JSON, Protocol Buffers, and YAML. CLP only supports unstructured logs (i.e. free text). However, in practice, semi-structured logs are much more important. For example, at Uber, the size of JSON logs is 10x of the size of unstructured logs. We propose a system that can efficiently compress semi-structured logs, and allows search without decompression.
The second objective is to optimize CLP's handling of unstructured logs. Specifically, we aim to significantly improve its search performance; not only making it faster, but also more predictable. Our design should also improve the compression ratio.
The third objective is an intelligent log parser that can automatically extract the schema of any logs. CLP, as well as most log management tools, requires user-provided rules (i.e., schemas) to parse the logs. Manually providing schemas is cumbersome, and such manual effort is required every time the software evolves. We propose an algorithm that can automatically infer the schemas of the logs, eliminating the manual effort.