The Harvard Business Review devoted its entire September issue to big data, rightfully focusing on Hadoop. Inside, the data scientist was anointed as the job of the future and experts dispensed advice on how to bring big data technologies to your company.
As someone who started his own business and grew it into a multi-million-dollar concern, I am a frequent reader of HBR. I don't have a business degree. I learned what I know from watching others and as a result of a life-long, debilitating, compulsive reading disorder (which I blame for my early-onset presbyopia, a disease of the eyes that means you're really old). It isn't common for HBR to be ahead of the curve on technology — and it isn't often I read about a relatively new open source project like Hadoop in its pages
According to Gartner, big data is expected to drive $34 billion of IT spending in 2013, with some predicting that number will more than triple by 2018. This year, amazingly, we also learned that the NSA is already a major user of big data technologies and has been for years. The government may not be able to deploy a website that allows us to buy health insurance, but when it comes to building a massive domestic surveillance network, that's done.
We also saw 10gen change its name to MongoDB, acquire a $1.2 billion valuation, and break the nine-figure funding mark that means you're probably going for IPO rather than acquisition. In the big data space, there was plenty of money to go around: $50 million to Hortonworks, $45 million to DataStax, $25 million to Couchbase, and so on.
In the last few years we've seen these technologies developed and deployed mainly by the "have tos" -- those companies with unprecidented data and usage problems, giants like Facebook, Yahoo, Twitter, or LinkedIn. While those companies may sit on more data and concurrent usage than some countries, they're often perceived as trendy or preoccupied with noncritical data -- hardly role models for mainstream businesses.
More recently, though, those of us in the trenches have seen even some of our more conservative clients stop kicking the tires and start adopting NoSQL technologies, especially MongoDB. In 2013, "log analysis" emerged as the killer app for Hadoop and the pilot project of the year, we're beginning to see more serious thought put into not just the solutions, but in developing the right questions to ask.
This phenomenon isn't localized, it's global. I write this from Brazil where a key-value store is being deployed to improve throughput in a large data project. Big data solutions may have begun by addressing problems in social media and entertainment, but they are spreading to education, health care, and finance.
Sign up for CIO Asia eNewsletters.