BigData Analysis with Project Spark and Shark

SPARK :

• Developed by AMPLabs, UC Berkeley
•  Developers Michael Franklin and Matei Zaharia 
•  Alternative to MapReduce parallel processing engine.
•  In-memory storage for very fast iterative queries removing temporary writes of intermediate data like MapReduce jobs. 
•  After each map and shuffle the data is written to local disk in Hadoop. Which increases the further  execution time. This bottle neck removed in SPARK by making the results available in the memory itself.
• Spark writes data to RDD (Resilient Distributed Datasets) which can live memory and hence Spark provides the necessary execution improvements.
• Up to 100x faster than Hadoop.
• Compatible with existing Hadoop ecosystem and works well with existing HDFS systems. 
• Spark can co-exist with existing Hadoop cluster using Mesos cluster manager. 
• It is better suited for iterative algorithms like Logistic Regression and Matrix Factorization compared plain data processing algorithms.
• Developed by Scala and provides clean APIs in Java and Scala. Python APIs will be added soon.

SHARK :

• Meant for Hive replacement with high degree of speed improvement.
• Built on top of SPARK data-parallel execution engine.
• Uses SQL like declarative language and works on SPARK infrastructure.
• Can execute complex queries using JOINs and GROPU BY
• Uses column-oriented store to improve performance. The columnar compression provides better reduction in storage.
• All the queries run in memory to improve the performance.
• Shark provides descent integration with Machine Learning using (Resilient Distributed Datasets). User can call these functions using SQL like syntax. This minimizes the complexity involved in using machine language.
• The entire software stack ( SHARK + SPARK + ecosystem ) is called as BDAS ( Berkeley Data Analysis Stack ).

Platfora and Qubole are taking Hadoop to next level with realtime interactive queries

As I have discussed earlier, one of the important disruption happening currently  in the Hadoop ecosystem is to make Hadoop more real time. Cloudera has already developed Impala which provides user to make real time queries. Mapr with Apache Drill project provides a declarative language like SQL to query underlined Hadoop data in real time. Google is already has its proprietary analytics engine and provides service called BigQuery to query petabytes of data. Facebook is also on other hand putting lot of efforts to increase performance of Hive and its ecosystem.

There are other start-ups like Platfora and QuBole taking different route for solving this big puzzle still using Hadoop. According to Ben Werther founder of Platfora,  business cannot wait days and weeks to get answers. It has to be instantaneous. The true innovation lies in the data agility and exploration made possible by Hadoop.

Hadoop is a great platform for solving the Bigdata. It has one of the big ecosystem in Bigdata. But the real issue with Hadoop is, it is not meant for all. Still the platform is restricted only for handful of researchers and calls for substantial amount of learning. This is the gap where most of the companies wants fill in many different ways.

Platfora got 20 million, Series B funding recently and has been working for more than a year to make Hadoop available to business users.  There are other first generation Hadoop start-ups built technologies around Hadoop. But most of them are run queries in batch oriented. This is where Platfora and QuBole differ in their implementation. Platfora has built an analytics platform from ground up. It has Vizborads for analytics, scale out in-memory data processing engine and hadoop data refinery as key components. Platfora wants to change the way traditional BI works.

One more start-up called QuBole also heading on the same lines. They have released QuBole data platform as service which run on Amazon  EC2 and Hadoop. QuBole founders Asish Thusoo and Joydeep Sen Sarma were from Facebook data infrastructure team where they have used and implemented Hive platform and they knew the bottlenecks of Hadoop and Hive. One of the main design goal of QuBole is to make Hadoop more real time and a simple platform for business user to run their queries.

To make queries run interactively, they use HQL and Hive but the entire platform is designed to run queries in real time.  The platform scales seamlessly up and down. The users need not to provision the no of systems needed upfront. In typical scenario it is unlike that user knows how many machines are required for query.  It is underlined platform’s job to allocate and free the machines in the cluster on demand based on the workload and past statistics.

It is good to see the NON Hadoop in memory databases like SAP HANA and recent announcement of Amazon RedShift also trying to build their technologies to solve big data more like traditional transactional database.

We need to see how much Hadoop can leap frog in solving this. As of now there is lot of innovation going on to make Hadoop more real time and more simple for many different use cases. Recently, Microsoft has also provided support for Hadoop on its Azure platform. This implementation is pure Hadoop Vanilla. Can we see Microsoft bringing something innovative to the Hadoop table or simply it follows the crowd ?

An alternative framework for Mahout : CRAB

Have you imagined every time you purchase any item from online sites like Amazon, BestBuy etc you might have figured out there are other items have been displayed as recommendations for you. For example I buy book from Amazon, it recommends list of books purchased by other shoppers with similar interest. This is possible as the online stores actual process millions of data and finds out the item purchased by similar users who are having a common buying behaviour. This helps the online sites sell more to their users based on the user preferences and their online behaviours. Most of the times users will not know what items they are looking for over the net. The recommendation system helps them to discover similar items based on their interest.  

In today’s overcrowded world with millions of items, it is very difficult to search and narrow down our requirements. In that context the online stores provides a filtration of data and presented in most pleasant way. At times we discover items which we might not heard of.

The same is not only true for online retailers. In most of the social sites, we discover friends and people with similar interest. This is done by processing all the socio interests expressed over the net and finding similarities between them. In linkedin you will find jobs, professionals, groups with similar interest. This is a facilitated by underline recommendation system infrastructure. Building a recommendation system could become a fairly complex process as the number of variables are going to increase. Of course the important variable is your amount of data to be processed.

Mahout played a very critical role in solving this problem. But it is not that trivial to build applications with Mahout. Though, it provides a comprehensive set of tools to work with Machine Learning. This is where CRAB fits the bill. The main objective of CRAB is to provide a very simple way to build the recommendation engine.

Crab is a flexible, fast recommender engine for python that integrates classic information filtering recommendation algorithms in the world of scientific Python packages (NumPy, SciPy, Metaplotlib).

The project is started in 2010 by Muricoca incorporated as an alternative to Mahout. It is developed using python so it is much easier to work with for an average programmer compared to Mahout which is built using Java. It has implemented User based, Item based and sloped based Collaborative filtering algorithms.

Demo Example can be found at:

https://github.com/marcelcaraciolo/crab/blob/master/crab/tests/test_recommender.py

Find more at:  https://github.com/muricoca/crab

Resource : http://www.slideshare.net/marcelcaraciolo/crab-a-python-framework-for-building-recommender-systems