Grant: Discovery Grants Program - Individual

Software applications produce logs to keep records of execution. Such logs are analyzed to detect, track any events, errors, faults, or exceptions. Emerging software applications are becoming large-scale, complex, and data-intensive. Such applications are extensively utilized and produce logs as data with massive size, high speed, and wide variety. Managing and analyzing logs at a large-scale becomes challenging due to lack of standards, guidelines, best practices on what and how to log. Currently available solutions rely on limited information and are mostly confined to producing logs as semi-structured text files with descriptions in natural language. Processing of such logs as text files include manual scanning, or usage of tools that carry out basic crawling and multiple traversals. This makes monitoring and management of software applications to be challenging. Key limitations are (1) lack of formal data modeling for logs, (2) lack of guidelines for structuring log data, (3) lack of analytical techniques to process logs, produced by software applications, at a scale of big data. This research program aims to discover and carry out advancements in data modeling and analytics of logs for software applications, especially that are large-scale, complex, and data-intensive. We will approach the challenges by building integrated solutions based on data modeling and big data analytics to efficiently collect, organize, and effectively analyze logs. Firstly, data modeling techniques will be built to make logs highly structured, well-expressed, and machine-readable. It will serve as a basis for building standards for logging in software applications. Secondly, specialized, scalable, and effective quantitative and qualitative analytical techniques, integrated with data modeling techniques, will be built to manage, and perform analytics on logs produced by large-scale and data-intensive software applications, while maximally utilizing structured, formally modeled and high expressivity of log data. Long-term goals are to build an integrated big data modeling and analytics framework for logs. It will (1) include data modeling and analytics on logs, and (2) act as an information system for developers, system administrators, and other related staff to facilitate automated analysis, and (3) help in setting up standards and best practices to effectively monitor and manage software applications. This is a timely research which will significantly improve monitoring of next-generation large-scale and data-intensive software applications. Impact and usefulness of the research will be demonstrated by applying it to real-life applications. This research program will contribute significantly to the training of highly qualified personnel (HQPs). It will benefit Canada in gaining a leading position in information management and data science where demand such skills is rapidly increasing.