Grant: Discovery Grants Program - Individual
University of Toronto
$96,000.00
Enabling Data Science over Open Data and Massive Data Lakes
FY 2022-23
Summary
In 2016, Forbes assessed that "data preparation accounts for about 80% of the work of data scientists" where preparation includes finding and collecting data, cleaning and integrating data, and managing data for data analysis. They concluded that this is also the least enjoyable part of a data scientist's job. As scientists, they would rather be deriving new knowledge and insights. The paradox is that without principled data management and preparation, those new insights are suspect at best. Data preparation and data management in support of analysis is so time consuming and unenjoyable because of the lack of tools, scientific frameworks, and mathematical foundations to support principled data preparation. I plan to devote the next five years to addressing and helping to correct this deficit. As part of my methodology, I will focus on open data, both because of its availability for scientific research and because of its importance to governments and society.Today, most of Canada's structured open data is in CSV (comma--separated--value) files, with some in JSON, and a tiny amount in RDF. Little if any of these datasets are being published with a schema following the W3C recommendation "Data on the Web Best Practices" or other open data best practices. And the files rarely contain attribute names beyond uninformative tags (like name10). Despite the large effort in making datasets publicly available, open data publishers, like the Canadian government do not provide search functionality beyond simple keyword search on the metadata. This metadata varies greatly in quality across different datasets and publishers. Even if data publishers were to include schemas and other valuable metadata, the lack of sophisticated search functionality creates problems for data scientists who want to use open data. The heterogeneity and incompleteness of the data create new problems for understanding the true structure of the data and how it can best be aligned with other data and ultimately used effectively for data science.My research will propose new methods for 1) finding relevant datasets (e.g., letting a data scientist find all datasets that join with hers or even all datasets that union meaningfully with hers) at interactive speeds over massive repositories of data; 2) new methods and mathematical foundations for the discovery of structure over open data (e.g., in CSV files that contain misaligned or pivoted data); and 3) new methods for aligning open data and data within massive public or private data lakes. I plan to make the methods I develop open source along with benchmarks for helping other scientists to develop and evaluate data preparation, collection and management solutions for massive open data. This work will extend the societal reach of Open Data at all levels (federal, provincial, and municipal), allowing this valuable data to be used more easily, in more effective and principled ways, for more purposes.
Program
Discovery Grants Program - Individual
Recipient
Location
Ontario, Canada
Recipient Type
Academia
Type
Grant
Source
NSERC's Awards Data
As published
Application id
RGPIN-2018-06012
Name nom
Miller, Renee
Department département
Computer Science (St. George Campus) - Computer Science (St. George Campus)
Organization id
31
Institution établissement
University of Toronto
Province en
Ontario
Province fr
Ontario
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2022
Competition year année de concours
2018
Award amount
96000
Program id
RGPIN
Program name en
Discovery Grants Program - Individual
Program name fr
Programme de subventions à la découverte - individuelles
Group en
DISCOVERY RESEARCH
Group fr
RECHERCHE AXÉE SUR LA DÉCOUVERTE
Committee code
1507
Committee name en
Computer Science
Committee name fr
Sciences informatiques
Area of application code
802
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Information systems and technology
Area of application fr
Systèmes d'information et technologie de l'information
Research subject code
2700
Research subject group en
Information technology
Research subject group fr
Technologie de l'information
Research subject en
Information technology
Research subject fr
Technologie de l'information
Installment
inst:5;6
Partie
2
Nb partie
6
Application title
Enabling Data Science over Open Data and Massive Data Lakes
Application summary
In 2016, Forbes assessed that "data preparation accounts for about 80% of the work of data scientists" where preparation includes finding and collecting data, cleaning and integrating data, and managing data for data analysis. They concluded that this is also the least enjoyable part of a data scientist's job. As scientists, they would rather be deriving new knowledge and insights. The paradox is that without principled data management and preparation, those new insights are suspect at best. Data preparation and data management in support of analysis is so time consuming and unenjoyable because of the lack of tools, scientific frameworks, and mathematical foundations to support principled data preparation. I plan to devote the next five years to addressing and helping to correct this deficit. As part of my methodology, I will focus on open data, both because of its availability for scientific research and because of its importance to governments and society.Today, most of Canada's structured open data is in CSV (comma--separated--value) files, with some in JSON, and a tiny amount in RDF. Little if any of these datasets are being published with a schema following the W3C recommendation "Data on the Web Best Practices" or other open data best practices. And the files rarely contain attribute names beyond uninformative tags (like name10). Despite the large effort in making datasets publicly available, open data publishers, like the Canadian government do not provide search functionality beyond simple keyword search on the metadata. This metadata varies greatly in quality across different datasets and publishers. Even if data publishers were to include schemas and other valuable metadata, the lack of sophisticated search functionality creates problems for data scientists who want to use open data. The heterogeneity and incompleteness of the data create new problems for understanding the true structure of the data and how it can best be aligned with other data and ultimately used effectively for data science.My research will propose new methods for 1) finding relevant datasets (e.g., letting a data scientist find all datasets that join with hers or even all datasets that union meaningfully with hers) at interactive speeds over massive repositories of data; 2) new methods and mathematical foundations for the discovery of structure over open data (e.g., in CSV files that contain misaligned or pivoted data); and 3) new methods for aligning open data and data within massive public or private data lakes. I plan to make the methods I develop open source along with benchmarks for helping other scientists to develop and evaluate data preparation, collection and management solutions for massive open data. This work will extend the societal reach of Open Data at all levels (federal, provincial, and municipal), allowing this valuable data to be used more easily, in more effective and principled ways, for more purposes.