Introduction to MapReduce Data Transformations ( SQL/MR )

Check out this SlideShare Presentation:

Introduction to MapReduce Data Transformations

View more presentations from swooledge.

Big Database List

Here is the Big Database List.

http://www.bigdatabaselist.com/wiki/Big_Database_List

A List of ETL tools

http://www.etltool.com/etltoolslist.htm.

To the above list I would like to add the following ETL products.

Jitterbit ETL - Jitterbit

Expressor Integrator - Expressor

Snap Logic : The data flow company

Most of the commercial data integration companies today provide data integration in very traditional way. All the integration and ETL jobs run in on premise and integration is mostly focused on structured data with in corporate boundaries. Though most of them support unstructured data, the basic premise of integration never changed. As we know for past few years there are lot of web standards have been emerged and most of the software vendors embraced them. One of the important focuses was to provide a very loose coupling between different services, so that we can truly create a collage of services and able to deliver on demand agile applications. Web service and XML has taken center stage in this paradigm shift. Unfortunately this did not extend to real enterprise application integration. Always the integration was done more on using native APIs. And hence there is always risk of failure, when more no of monolithic applications communicates with each other.

On the other hand the types of the data being generated by various applications has also changed over the years. Today, the unstructured data like feeds, atoms, xml, csv files are generated more than the structured data. More over the unstructured data is not only getting generated with in corporates but also from outside like blogs, wikis. So the traditional integration is bound to fail as they were never built on these premises.

In recent years SaaS ( Software as service ) emerged as a very promising business model. This has changed the complete business dynamics. With the help of SaaS, Now vendors can offer and target any customer with in any price bracket for their applications. This completely reduces customer initial upfront investment on the software licenses. On the same time it has raised some more challenges. There are many corporate legacy applications have to talk to these externally hosted managed services . For example a company wants to integrate salesforce application with its in house legacy ERP application then we need to have to support web service compared to traditional data access layers.

So if we sum up all, in today’s data integration, we need to support various types of data from various sources with various data access apis/standards/protocols. This has to be accomplished in more loosely integrated, without any boundaries, securely, fast and agile unlike traditional data integration projects. This is what SnapLogic is trying to fill.

Snap logic is a open source data integration company. SnapLogic DataFlow is a scalable data integration platform that leverages Web technology and standards to provide organizations of all sizes with a flexible and cost-effective solution for on-demand data integration. It can connect and fetch data from various data sources like traditional data bases, SaaS apps like sales force, netsuite , social networking sites. The dataflow also provides a way to create custom components and integrate with Snaplogic.