The idea then, Barth says, is that Big Data platforms become one piece-albeit an important one-of a data ecosystem that is designed to constantly look for new insights into customers, markets, products and risks, while at the same time building upon what is already known. In other words, pursue the "new" while operating on the "known," a healthy, continuous improvement model.
Creating a Big Data Ecosystem
Think of it this way: Whether it comes from Big Data or traditional analytics, the important thing is to provide valuable answers. The value of an answer, Barth says, is based on its accuracy and the speed with which it can be delivered. To get an accurate, speedy answer, it's important to ask the right questions. And that's where Big Data comes in: It's about pursuing the "new."
"The art part of Big Data is associated with discovery and explanation," Barth says. "You're looking for something you can't quite articulate. There's a phase of analytics that's exploration and discovery, in which you're generating hypotheses. Then comes modeling and application. In my view, traditional BI tends to be really far down the road after you understand the underlying analytics and correlations that relate to an issue you're seeing. The phase in which you don't understand it, the discovery phase, is where Big Data is useful."
There are seven distinct steps to answering a complex business question, NewVantage says:
Traditionally, companies have spent 80 percent or more of their time on the third and fourth steps, NewVantage says. But Big Data solutions offer up new ways of approaching these steps.
First and foremost, because of the relatively low cost and high capacity of Big Data platforms, organizations can load all of the data from their source systems rather than choosing particular data for the question at hand.
"While this may seem wasteful, it eliminates two important delays: writing programs to select just the data needed, and going back to the source systems multiple times as new insights generate new questions that need new data," NewVantage says. "Building traditional data marts and data warehouses is extraordinarily complex and costly. The broad range of open source offerings coupled with flexible, scalable grid systems create an environment that not only drives down costs, but also offers the potential of decreasing query times exponentially."
For instance, Barth points to a large financial services firm that wanted to perform multi-channel pathing analysis of its customers to understand which elements led to a sale and which led to attrition. To do so, the firm needed to integrate six months of session data with other channel data. The first attempt, using traditional relational databases, took tens of thousands of lines of SQL code and the firm soon realized that it could only afford to access six days worth of data rather than six months. The firm abandoned the attempt after calculating that the effort would take weeks.
Sign up for CIO Asia eNewsletters.