Grant: Collaborative Research and Development Grants
Simon Fraser University
$79,147.00
Optimizing hadoop to scale to big systems and big-data
EF 2016-17
Résumé
The overall `big-data' market is expected to be worth more than \$100 billion and growing roughly twice as fast as the software business as a whole [Deloitte]. The emergence of ""big data"", such as social media, has played critical roles in business analytics. As an IDC report indicates data analysis will play a key role in managing traffic in canadian cities (e.g., Toronto) and healthcare. Such big data is often stored in the new generation of NoSQL databases, and processed by massively parallel MapReduce/Hadoop based infrastructure. The industryis facing an essential challenge: can millions of existing successful applications based on SQL take the advantage of big data infrastructure? Many critical challenges remain including the design of optimal data storage formats and efficient communication of data between data storage nodes. In this project, we will be working with our partner, Simba, an industry leader of big data technology and has guided us towards the challenges that forsee from their client's perspective. We will be making three specific contributions i) Low-latency: We will be adapting Apache Hive (open source Hadoop-based NoSQL framework) to take advantage of RDMA (Remote Direct Memory Accesses) networks and scale out to take advantage of rack scale memory resources. ii) Reducing Bandwidth: We will be developing data-specific compression mechanisms to minimize the data movement across the compute rack and enable both an increase in database volume and data access speed, and iii) ``Flexibility"": We will be developing a tool for enabling Apache Hive to dynamically enforce schemas and enable end-client SQL-based business analytic tools to interact with NoSQL database backends employed by Hadoop.We also plan to integrate our changes into the open source Apache Hive platform to benefit the wider Big data community.
Programme
Collaborative Research and Development Grants
Bénéficiaire
Emplacement
British Columbia, Canada
Type de bénéficiaire
Academia
Type
Grant
Source
NSERC's Awards Data
Tel que publié
Application id
485325-2015
Name nom
Shriraman, Arrvindh
Department département
Computing Science, School of
Organization id
5
Institution établissement
Simon Fraser University
Province en
British Columbia
Province fr
Colombie-Britannique
Country en
CANADA
Country fr
CANADA
Fiscal year exercice financier
2016
Competition year année de concours
2015
Award amount
79147
Program id
CRDPJ
Program name en
Collaborative Research and Development Grants
Program name fr
Subventions de recherche et développement coopérative
Group en
Research partnerships programs
Group fr
Programmes de partenariat
Committee code
96
Committee name en
RPP Internal Decision Cttee
Committee name fr
Cté de décision interne - PPR
Area of application code
801
Area of application group en
Information and communication services
Area of application group fr
Services d'information et de télécommunications
Area of application en
Computer software
Area of application fr
Logiciel
Research subject code
2711
Research subject group en
Information technology
Research subject group fr
Technologie de l'information
Research subject en
Database management
Research subject fr
Gestion de bases de données
Partie
1
Nb partie
3
Application title
Optimizing hadoop to scale to big systems and big-data
Keyword
NSERC
Application summary
The overall `big-data' market is expected to be worth more than \$100 billion and growing roughly twice as fast as the software business as a whole [Deloitte]. The emergence of ""big data"", such as social media, has played critical roles in business analytics. As an IDC report indicates data analysis will play a key role in managing traffic in canadian cities (e.g., Toronto) and healthcare. Such big data is often stored in the new generation of NoSQL databases, and processed by massively parallel MapReduce/Hadoop based infrastructure. The industryis facing an essential challenge: can millions of existing successful applications based on SQL take the advantage of big data infrastructure? Many critical challenges remain including the design of optimal data storage formats and efficient communication of data between data storage nodes. In this project, we will be working with our partner, Simba, an industry leader of big data technology and has guided us towards the challenges that forsee from their client's perspective. We will be making three specific contributions i) Low-latency: We will be adapting Apache Hive (open source Hadoop-based NoSQL framework) to take advantage of RDMA (Remote Direct Memory Accesses) networks and scale out to take advantage of rack scale memory resources. ii) Reducing Bandwidth: We will be developing data-specific compression mechanisms to minimize the data movement across the compute rack and enable both an increase in database volume and data access speed, and iii) ``Flexibility"": We will be developing a tool for enabling Apache Hive to dynamically enforce schemas and enable end-client SQL-based business analytic tools to interact with NoSQL database backends employed by Hadoop.We also plan to integrate our changes into the open source Apache Hive platform to benefit the wider Big data community.