Peddada, an MBA from the Lally School of Management & Technology, New York, has a two-year plan chalked out to adopt big data at SKS Microfinance. "We have a good MIS system. But I wanted to streamline it first and go to the next level of business intelligence, SQL services, analytics services, and finally get to the highest level of BI," he says. His aim is to use a number of information sources to be able to, say, lower the micro-finance company's risk. For example, he says, he'd like to use weather reports and news feeds to predict whether crop production in a certain region is good enough to ensure that customers from there would be able to pay their loans.
Another challenge is the cost of storing and archiving of all that data. Bahl says it's likely that organizations will collect and store more data than ever before. The key challenge they will face is how to manage that data, what to retain, and how to eventually dispose it.
At SKS Microfinance, Peddada plans to improve the company's storage capacities, invest in data warehousing and processing power to match his big data goals, over the next six to seven months.
But, what about data that's coming from outside organizations? That's a question Iyer at CRISIL had to ask himself. CRISIL generates 6TB of structured data and about 6GB of unstructured data everyday. It also uses Web crawlers to pick information from social media sites and various forums and platforms available on the Web. That's then run through parsers, which conducts syntactic analysis and gives data some structure for use and before the data's subsequently discarded. "Since this unstructured data is already available on the Internet, my database will only store which location the data was downloaded from," says Iyer. "It helps lower my storage cost," he says.
Big Data's Building Blocks
Some of big data's challenges, like ROI and storage, can be countered if CIOs chip at it long enough. However, there are others, say skeptics, which are out of a CIO's hands.
One of these is the uneven speed at which different technologies that make up big data are maturing at. According to Gartner's big data hype cycle, published July 2012, technologies like Web analytics and social media monitors will take less than two years to enter mainstream adoption, while technologies like in-memory data grids and Mapreduce will take between two and five years to mature. Other technology subsets of big data like the semantic Web and information valuation could take as long as 10 years to make in-roads into the enterprise.
In this shifting landscape of technologies, CIOs will be hard-pressed to find a scalable, flexible platform for their big data initiatives.
Sign up for CIO Asia eNewsletters.