Ramnath Iyer knows the business of information like few others. As the CTO of CRISIL India, the global ratings agency, he watches over financial and management data collected from over 15,000 medium and large enterprises. "Our database has more than 120 million data points," says Iyer, who started his career at CRISIL as an analyst.
Two years ago, CRISIL decided to help its analysts create more accurate ratings, faster. It did this by enabling analysts to tap into large pools of data--located outside CRISIL's walls--in a more organized fashion. If, for example, its analysts wanted to rate the performance of an auto manufacturer in Gurgaon, CRISIL's big data systems would help by crawling social media sites and blog posts for opinions about the company. It would also take data from sources as varied as government reports on the production of steel, online announcements of a new excise duty, weather reports affecting the production of rubber (to gauge the cost of tires) and news from Bloomberg or Reuters regarding a strike at one of the company's ancillary providers. Combining all that information and more gave analysts a more accurate picture of the company they were rating and the environment it operated in.
In a way, Iyer's team has built a system that's as smart as its human counterpart and has become a member of the team. That's allowed analysts to rate thrice the number of companies they could before. "The toughest challenge was constructing a system with in-built intelligence; a system that can think like an analyst, that can find and co-relate various pieces of information and throw results at analysts," says Iyer.
CRISIL's story is just the sort that big data vendors and analysts love to push. And you're their target.
But under scrutiny, the glossy veneer of big data's brochure-ware gives way to a whole set of challenges, doubts and questions that are typically associated with technologies in their hype cycle. The question is: Will big data stand the test of enterprise adoption? Or will go up in a big cloud of smoke?
The Big Identity Crisis
When big data first started creating buzz, it was perceived as a technology that could manage large volumes of data. Since then, however, newer definitions have evolved to include two new parameters: The different disparate sources that companies are gathering this data from, and the speed with which data is collected, stored and processed.
"We define big data as high volume, velocity and variety information assets that demand cost-effective, innovative forms of information processing for enhanced insights and decision making," says Sid Deshpande, senior research analyst at Gartner.
Sign up for CIO Asia eNewsletters.