Grant: Collaborative Research and Development Grants

Enterprises are building massive storage repositories (often referred to as data lakes) that hold data in its native format (text, JSON, CSV files, relational databases, etc). The aim is to increase the amount of usable information by allowing fast acquisition of data from many sources as soon as they are available, instead of waiting for traditional ETL (extract-transform-load) stacks to curate the data and integrate it in a trustworthy warehouse. Unfortunately, with the massive number of external data sources, and the increasing heterogeneity in the formats of these sources (text, feeds, tweets, posts, blogs, events, etc.), handling unstructured and semi-structured data in a unified framework becomes key. In this project, we build an end-to-end system for handling (semi-) structured and unstructured in a unified way, in an open-source test-bed we call DSTLR (short for the data distiller). The DSTLR project combines advances in natural language processing (NLP) and information extraction (IE) particularly with deep learning, data cleaning, and managing RDF data to enable treating all types of data in a common format that allows for truth finding, question answering and structured data enrichment. The main objective of the proposal is to identify an investigate the technical challenges in building such a system including: (1) identifying the necessary provenance and lineage information to describe the context of the IE system; (2) efficient propagation of such information across the modules; (3) leveraging this rich extraction context in cleaning the extracted information; and (4) exploring novel ways to link the cleaning of the output to the information extraction subsystem in an iterative loop, with judicious involvement of humans in the whole life cycle. For this, the proposal identifies multiple concrete research tasks and expected outcome of these investigations.