Maintaining Data Visibility across the Cloud and the Ground
The move to cloud provided services (cloud) has meant a significant shift in where data is generated and stored. No longer is all data generated within the internal boundaries of a company’s own network and data centre (ground). This presents a challenge when it comes to maintaining data visibility and intelligence gathering capabilities – especially from a security and risk perspective. In this post, we will examine several different high-level scenarios and the impact on data visibility of each.
The following scenarios follow a fictitious company who decide to adopt cloud services and the journey they take. Of course, not all companies will move in this way and one scenario might be applicable to one company vs. another. Additionally, the scenarios are overly simplified for illustrative purposes and thought. The reality is that decisions on the best approaches to maintaining data visibility will be a lot more complex and are usually fairly political in nature.
Forget the Cloud…
The company has excitedly started leveraging cloud services, but has neglected to think strategically about their data now being generated within these cloud services and the impact on data visibility. All of these services will have web applications, providing visibility to data specific to the service, but the company is getting a silo’d view and correlation with data from their other cloud services or internal data is not possible.
Pull it Down…
The company has realised the importance of their cloud data and gained full visibility by pulling it down from the various services into their internal big data intelligence platform, where the data is then correlated with other sources. This approach may work well initially, but as the company adopts more cloud services, a suitable data integration solution will likely be required. Ultimately, the volume of data being pulled will become unmanageable and the company will question the approach.
Distributed Centralization…
After battling with crazy data volumes and multiple data connections, the company has leveraged one of the many available cloud based big data platforms to log all data from their cloud providers in one place. Data from the cloud is logged in the cloud and data generated on the ground internally is logged internally. This approach seems efficient, allows for decent data visibility and probably offers significant cost savings. However, correlation between cloud and internal data has been lost, presenting challenges from a security point of view.
Hybrid Visibility…
By connecting their cloud data store with their ground data store, the company has been able to facilitate correlation, data visibility and analytics capability across all their data sets. This approach requires strategic thought, good planning and architecture, but is absolutely possible. As the company moves more into the cloud, this approach allows them to dynamically scale out the cloud data store, reduce their internal data store and still maintain central visibility across all their data sets.
Beam it Up…
Eventually, it becomes more efficient for the company to simply log all their data in the cloud. Their internal infrastructure is shrinking as they take advantage of the economies of scale and processing power that cloud services provide. Data visibility and correlation capabilities are maintained through their centralised data store in the cloud. They are also able to take advantage of the latest cloud services that can snap into their cloud based data store to perform data analytics and intelligence reporting and do things they could never have hoped to have been able to perform internally.
Forget the Ground…
Ultimately, the company no longer maintains any of their own internal infrastructure and everything is cloud based. A managed integration platform service, allows them to easily centralise all their data from the different services they leverage. They no longer need to operationally manage any infrastructure, leaving them to focus on advancing their data intelligence capabilities and insight.
Hopefully the above provides some food for thought, even if the scenarios are overly simplified. As you move to the cloud, it is important to strategically think about the impact this might have on your ability to maintain visibility across your data. When leveraging new cloud services, think about the data they being generated and stored – is this data important to your company? could you obtain further insight by correlating the data with other sources? can you even get data out of the service?
© Copyright Discovered Intelligence Inc., 2014. Do More with your Big Data™. Unauthorised use and/or duplication of this material without express and written permission from this site’s owner is strictly prohibited. Excerpts and links may be used, provided that full and clear credit is given to Discovered Intelligence, with appropriate and specific direction (i.e. a linked URL) to this original content.
