In their market overview for Hadoop distributions, Gartner analysts Nick Heudecker, Merv Adrian and Ankush Jain say: "These changes in the Hadoop ecosystem come at a time when information management megavendors IBM (which offers its own distribution), Microsoft, Oracle, SAP and Teradata have largely completed their integration with the salient elements of Hadoop.
"They are incorporating it into their broader portfolio of capabilities, such as event stream processing, analytics, database management systems (DBMS), data federation and integration, metadata management, security and governance. The Hadoop distributors are also adding these capabilities, either through partnerships or managed development efforts."
Computerworld UK will be doing a deeper dive on Hadoop and Spark vendors, including a comparison of each and their individual merits/issues in the coming months.
Hadoop vs Spark: Conclusion
Despite its relative maturity, compared to Spark, Hadoop still isn't delivering the kind of transformative results many vendors will claim. According to Gartner's market guide: "Through 2018, 70 percent of Hadoop deployments will fail to meet cost savings and revenue generation objectives due to skills and integration challenges."
The answer? "Match projects to specific business requirements and identify the existence and readiness of supported technology components suitable for them," says Gartner.
Spark, on the other hand, has the potential to be truly transformative for the right kind of companies with the relevant expertise. As Gartner puts it: "Apache Spark emerged as a force as potentially disruptive to Hadoop as Hadoop was to traditional database management systems."
Although some IT departments may feel compelled to pick between Hadoop and Spark, the fact is it's not quite a straight fight. They can be complementary technologies working in tandem, or depending on your data, one may be better suited than the other.
But one of the major advantages either way is that your organisation will be able to resist vendor lock-in, so with the right team, getting a proof of concept off the ground is easier than ever.
Sign up for CIO Asia eNewsletters.