We are now dead-center in the middle of the second decade of the 21st century. When big data mania got rolling around five years ago, the consensus that the future went by the name Hadoop was shockingly pervasive. Growth in the Hadoop market since that time showed that this was no fad. The unrelenting hype has at least had some grounding in Hadoop's marketplace adoption and innovation.
Given that everybody pretty much agrees Hadoop is important, must we in the big data industry continue beating the drum for its proverbial "next big thing" status? Has Hadoop's inflection point long since passed -- and is its maturation point fast approaching? When a segment shows all the signs of maturation, it's time to tone down the marketing overkill. The Hadoop "next big thing" may now be as "big" as it'll ever get, in terms of its share of the big data analytics market (though the overall market may itself continue growing like the proverbial gangbusters).
To determine whether Hadoop has reached this point, let's review how far this segment has come and how it'll likely evolve going forward.
Startup activity is a clear sign of a growth market, and its decline is a strong signal of maturation. After a tremendous burst of startup formation in the early years of this decade, it would now appear that Hadoop platform, tool, and application vendors have settled into a familiar group of usual suspects. For example, every single vendor mentioned in this recent InformationWeek market overview was already in this space three to four years ago when I was Forrester's Hadoop analyst. That's one clear sign of a maturing market.
Another sign of Hadoop's maturation is the fact that the chief demand drivers are essentially constant from year to year, reflecting a niche that is continuing to scratch the same itch. Once again, the cited article rattles off survey response numbers showing that users adopt Hadoop principally for unstructured data analysis, predictive customer analytics, sentiment analysis, and so on. None of that is appreciably different from what I saw in my primary research into the then-embryonic Hadoop market in 2011.
Yet another sign of segment maturation is the fact that the industry tends to hammer on the same themes over and over, year after year, as befits a solution space that has found its functional sweet spot. For example, the big data blogosphere continues to tiresomely debate the already settled issue of whether SQL has a future in the Hadoop ecosystem. The answer is decidedly yes, as evidenced by the range of alternative SQL access/analysis options from every major vendor listed in the cited article.
Sign up for CIO Asia eNewsletters.