Snap Logic : The data flow company

Most of the commercial data integration companies today provide data integration in very traditional way. All the integration and ETL jobs run in on premise and integration is mostly focused on structured data with in corporate boundaries. Though most of them support unstructured data, the basic premise of integration never changed. As we know for past few years there are lot of web standards have been emerged and most of the software vendors embraced them. One of the important focuses was to provide a very loose coupling between different services, so that we can truly create a collage of services and able to deliver on demand agile applications. Web service and XML has taken center stage in this paradigm shift. Unfortunately this did not extend to real enterprise application integration. Always the integration was done more on using native APIs. And hence there is always risk of failure, when more no of monolithic applications communicates with each other.

On the other hand the types of the data being generated by various applications has also changed over the years. Today, the unstructured data like feeds, atoms, xml, csv files are generated more than the structured data. More over the unstructured data is not only getting generated with in corporates but also from outside like blogs, wikis. So the traditional integration is bound to fail as they were never built on these premises.

In recent years SaaS ( Software as service ) emerged as a very promising business model. This has changed the complete business dynamics. With the help of SaaS, Now vendors can offer and target any customer with in any price bracket for their applications. This completely reduces customer initial upfront investment on the software licenses. On the same time it has raised some more challenges. There are many corporate legacy applications have to talk to these externally hosted managed services . For example a company wants to integrate salesforce application with its in house legacy ERP application then we need to have to support web service compared to traditional data access layers.

So if we sum up all, in today’s data integration, we need to support various types of data from various sources with various data access apis/standards/protocols. This has to be accomplished in more loosely integrated, without any boundaries, securely, fast and agile unlike traditional data integration projects. This is what SnapLogic is trying to fill.

Snap logic is a open source data integration company. SnapLogic DataFlow is a scalable data integration platform that leverages Web technology and standards to provide organizations of all sizes with a flexible and cost-effective solution for on-demand data integration. It can connect and fetch data from various data sources like traditional data bases, SaaS apps like sales force, netsuite , social networking sites. The dataflow also provides a way to create custom components and integrate with Snaplogic.

The Big Data Puzzle

The gen next new age database vendors claim support for petabytes of data. This list is ever increasing day by day. It’s quite interesting to watch the trends from EDW ( Enterprise data warehouse ) to EDA ( Enterprise data appliances ) to EDC ( Enterprise data cloud ). The EDC is the latest band wagon from Greenplum database vendor. Today most of the Big Data vendors singing the same tunes and at the core of it the following technologies responsible.

1. Column oriented architecture
2. Cloud support & Integration
3. Map Reduce to process the data
4. Massively Parallel Processing
5. Data grid with Commoditized H/W

Companies like Vertica, ParAccel , GreenPlum , InfoBright , Sybase are built databases around above key technologies.

Column Oriented Architecture :

A column oriented database stores its content in columns rather than storing in traditional row format like OLTP database systems. This architecture has a definite incremental advantage of processing massive amount of data compared the row based architecture. This is most suitable for read operation and queries. In a typical data warehouse, the no of columns might be more than 100 per table. In those cases when the user queries only few columns, there is no need for processing the entire row. We can easily retrieve the necessary columns much faster compared to row based storage.

Cloud support & Integration:

All most all vendors provide a way to integrate their databases with clouds. Inherently cloud architecture provide the necessary tenets for solving the peta bytes of data problem and on the same hand it also provides better economy of scale. Typically, cloud computing provides the following key features which is the key for big data vendors.

1. On demand Scale out and provisioning provides low capex which attracts all sized vendors.
2. The whole analytical process can be configured with in external cloud. They need not be in house. This helps in reduction upfront cost of H/W installation.

Map Reduce:

Map reduce framework provides yet another dimension for solving this equation. Map Reduce provides a way for processing huge datasets by chopping in to small sets and aggregating back them in a distributed environment. Initially this technique was tried and implemented by Big Daddy Google. There are lot of architecture built on this framework. One of the notable variant is Hadoop, which can solve massive amount of data in key and value pair. The Map Reduce is extended to SQL by GreenPlum and AsterData systems. Some company feel this will help developers as they need not to learn a new language while working with these technologies.

Massively Parallel Processing:

The massively parallel processing with shared nothing technology is one of the key technologies behind today’s distributed computing and this provides the so called on demand scale out in cloud infrastructure. And hence this is one of the key ingredients within the Big Data equation.

Data grid with Commoditized H/W:

This is last but not the least technology used in the above stack. Above technological rise is happening due to availability of today’s low cost commodity H/W. This H/W commoditization led to proliferation of large grid architecture with low cost. Instead spending money on high-end H/W, cloud computing provides an opportunity to use low cost H/W and allows to reap the advantages with the same proprietary H/W.

Scale UP versus Scale Out

Scale UP : This process is also called as Vertical scaling. With this approach you add more hardware like RAM, Processors and Network Interface cards. This provides more memory real estate and bandwidth for computation and hence your application will be faster. Having said that, if we can keep on adding new hardware to existing servers at one point of time they fail or they give us negative performance. So Scale up is always has limitations. While scaling up, the OS and application software also has to support the increased H/W and provide us better performance.

Pros :

• 1. It doesn’t introduce additional maintenance cost and support cost.
• 2. The current blade server provide plug in architecture to add new H/W. So adding new H/W is a fairly simple process.
• 3. Lot of software vendors support Scale up, so getting support is more easier.
• 4. It’s claimed that Scale up provides much better performance and low maintenance cost compared to Scale out process keeping all the variables constant.

Cons :

• 1. We cannot simply keep on adding new H/W to existing servers. It whole lot depends on the server architecture and the software too.
• 2. Change in technology compatibility. As technology is changing rapidly, we are not sure how well the new H/W will have the compatibility with old ones.
• 3. The underlined software also should be designed for H/W Fail overs.

Scale Out : Scale out also called as Horizontal Scaling. You add more no of servers to existing infrastructure and do a load balancing. This approach catching up due the technologies like Map/Reduce. In scaling out there is no limitation. Until you can do the load balancing, you can keep adding the new servers to existing cluster and these servers need not to be high end like Scale Up scenarios. They can be commodity H/W.

Pros :

• 1. Easier to add low end servers as per the demand.
• 2. The servers can be instantly scaled out or scaled down based on the demand.
• 3. This provides right economy of scale.
• 4. In case one node fails in the grid, the system does not stand still, the other nodes will take the load automatically and hence the H/W failure is not a big risk.
• 5. Companies like Aster Data Systems and GreenPlum Software provided enough ground on the Map/Reduce technologies.

Cons :

• 1. Maintenance and support cost may be higher compared Scale Up process.
• 2. The scalability is much more depends on the net work bandwith provided by the infrastructure. However in practice Scale out process will provide less performance over scale up process.
• 3. Unless you use open source, the licensing cost and initial acquisition cost is more in Scale Out approach.

In typical enterprise, both the Vertical and Horizontal scaling used to take the benefit of both the approaches and they are co-existence. They start with a large vertical architecture server, adding resources as-needed. Once the single server approaches its limit then they scale out. In some cases the scale out is more practical than scale up. For example companies like social networking, they need to scale out rather than scale up. In case the performance is the key, then scale up may be the better choice over scale out.

Comparing JSF, Spring MVC, Stripes, Struts 2, Tapestry and Wicket

This presentation is bit old but contains very nice info about various web frameworks. In my opinion Raible slid show provides an overview of various web framewroks but in real life it is not that easy to compare and start choosing the best web framework. It always depends the kind of the project you really have. Having said that it provides a good start for understanding the various frame works.

Check out this SlideShare Presentation:

Comparing JSF, Spring MVC, Stripes, Struts 2, Tapestry and Wicket

View more documents from Matt Raible.

Introduction To Enterprise Data Mashups

Check out this SlideShare Presentation:

Introduction To Enterprise Data Mashups

View more presentations from Justo Hidalgo.

Cloud Vendors A to Z (Revised)

The following is good post

http://www.johnmwillis.com/cloud-computing/cloud-vendors-a-to-z-revised/

all the buzz around the cloud...

Over the last couple of years there is lot of buzz surrounding cloud computing. Every day there are mushrooms of startups built around this new computing platform and business model. And there is lot of VC investment happening around these companies. I am hoping we are not again in the same clouds of early 2000 when all VCs put lot of ambitions and aspirations and all the internet startups were overvalued and finally the bubble got collapsed. I am not sure whether this cloud syndrome is a real phenomenon. There are authors equated this cloud era data centers to electricity grids. This might be true but, I agree with others who say all the computing model exists and cloud computing will be one of the computing models in computing eco system. No doubt we are progressing towards much more deep collaborative environment. The latest raise in social networks and gaining importance in online collaborative tools and sites is true witness of these patterns. In all these the internet becomes main stream computing platform.

Will all the applications move to cloud? May be the answer is NO.

Every application written in the universe is very different in terms of its requirement and usage. There was a time when desktop PCs gaining a lot of power, everyone thought there is an end to main frame era and powerful desktops will replace the main frame computers. The power of computing will be decentralized. The fact that today mainframes are still ruling. Today’s majority of mission critical application are still running in mainframe and the interesting phenomenon is we are going back to the centralized computing model like main frame.

The cloud provides one of the strong business model, which justifies a perfect ROI and hence there is lot of buzz around this and this platform is new and immature. There are very few companies who mastered this art. However there is lot of proof has been made on the ground in terms of the business values this new platform provides. One of the earliest company to ripe the benefits of this platform is salesforce.com. Currently, there are lots of big software giants like Microsoft, Google, Amazon, Rack Space, IBM, SUN having their own definition of Cloud and each one provides their own infrastructure and utilities around their technologies. This is bound to happen and things will consolidate in form of standards in future as the technology matures. Though the cloud computing is gaining momentum rapidly, still there are lot of concerns in the minds of corporate. The most important concern is security and critical data governance. The current infrastructure and utilities are provided to the end user more like vendor lock in. In case the user wants to switch between other cloud providers what are the options. We really do not know yet and hence this is pretty clear that the data and processes which are not critical to an enterprise can be deployed on an on demand cloud to gain the much wanted business value where as the critical data and processes could be managed in house. This will result in having hybrid models with different computing strategies for different kinds of requirements with in an enterprise.