However, moving large data sets to and from cloud data centres will reveal the weak link in the "big data ecosystem" - a network that isn't built to handle such large quantities of data. As with all networking capacity problems, this issue cannot be sustainably addressed by simply installing or leasing larger data pipes - simply because of the bewildering array of new services and the inexorable rise in data.
Instead, processing big data effectively is better served using a virtual data centre architecture, where the physical walls of individual data centres are effectively broken down to connect multiple data centres as one logical entity. To put it another way, a "data centre without walls" is created, one that uses a high performing "cloud backbone" network to seamlessly connect a larger pool of resources shared between both enterprise and provider data centres.
By virtualising and pooling all data centre and network assets, organisations can allow flexible placement and migration of workloads according to changing needs. This is the only IT setup that can economically meet the infrastructure challenge posed by big data and allow this phenomenal trend to deliver on its promise.
Data centre without walls
Forrester has said that the vast majority of Asia-Pacific big data initiatives centre primarily on building big data repositories, and secondarily on providing users with more advanced data discovery and visualisation tools. But this doesn't leverage the "data centre without walls" to allow for the most cost-effective processing of big data to meet business objectives.
To make big data work for your organisation, such that computing power can easily be engaged to process data sets, you need a high-performance connection from your data centre to the cloud provider's data centre(s). But beyond a static connectivity service, which can be expensive to have enabled all the time, a "performance-on-demand" connectivity service can allow you to turn up the bandwidth when you need it and turn it back down when you don't.
So make sure you have an IT infrastructure that can not only leverage virtual resources, but is also built on performance-on-demand based connectivity so you pay only for what you use.
Karl Horne is chief technology officer, Asia Pacific, Ciena.
Sign up for CIO Asia eNewsletters.