Don’t Neglect Big Data Integration

In the excitement to implement a big data platform like Splunk or Hadoop, many enterprises put data integration on the back-burner or figure it can be ‘worked out’, once the platform is in place. However, data integration is a key part of a successful big data intelligence strategy and must be given appropriate consideration.

Data intelligence involves the correlation, searching and performing of advanced analytics on data from many – potentially thousands of – different data sources. Enterprises without a good data integration strategy and an inadequate tool-set often find themselves spending months integrating data sources; developing point-to-point database connections, writing scripts that tap specific APIs, and/or batch processing data using ETL-type tools. This approach may work, but ultimately you have built a fragile house of cards, with no visibility into when critical data integrations fail and will have to continually rework existing integrations that break or were found to not scale. Without careful thought, data integration can rapidly become a burden and a roadblock to achieving your data intelligence goals.

Cloud services have come to represent the new reality, with even the most resistant enterprises now flocking to them. Cloud services may remove the headache of implementing and managing applications and infrastructure, but they compound the issue of data integration complexity by increasing the disparate nature of data across the enterprise.  Data is no longer sitting in one internal data centre, but is spread across several third-party services in data centres all over the world.

Many cloud services silo data and therefore, ultimately silo your data visibility and intelligence gathering capabilities.

When selecting cloud services, thought must be given to how data integration fits into the picture. For example, How will you access this data? How will you correlate this data with your own internal data? Does this data need to go to, or integrate with, other cloud services? 

In today’s big data-centric environment, data integration has had to evolve. Legacy and hugely costly ETL and ESB solutions, that batch data transfer and poll databases, no longer cut it in a world where time is money. Technology has evolved, huge data scales prevail and there are now cutting-edge technologies that rewrite the rule book on data integration. As is common, these new integration technologies are either completely or partially cloud based. However, this should not put you off, as you probably already leverage some cloud services and will be increasingly doing so in the future.

The right technology must not only solve today’s data integration challenges but be open enough to work on future initiatives.

These new integration technologies have been developed from the ground up, using open, RESTful based technology and designed to scale with your data volumes. They also make data integration simple; providing usable drag and drop interfaces that most IT personnel can quickly get to grips with. Out of the box connections for many popular cloud providers and common data integration elements, greatly expedite data integration. Our managed data integration-platform-as-a-service (iPaaS) offering is built on the leading player in this area – contact us today to find out more.

It does not make sense to rip out legacy, but working, data integration solutions and replace anew – very little tangible business value is going to come of that. Instead, the focus should be on leveraging these new technologies to solve your new data integration challenges. These challenges might include:

• Pulling data out of a cloud provider and streaming into Splunk for operational intelligence reporting or correlation with your internal data sets.
• Streaming data from one cloud provider to another, or several others.
• Extracting data from a number of databases within your internal enterprise and dropping the data into a data store like Hadoop.

Effective data intelligence can only come about by ensuring there is appropriate visibility and access to all your data sets, whether internal or in the cloud. The only way to do this, is to put in place an effective data integration strategy. Don’t neglect the need for data integration or your shiny new big data platform won’t give you visibility into anything.

© Discovered Intelligence Inc., 2013. Do More with your Big Data™. Unauthorised use and/or duplication of this material without express and written permission from this site’s owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Discovered Intelligence, with appropriate and specific direction (i.e. a linked URL) to this original content.