If 2014 was the year that Apache Hadoop sparked the big data revolution, 2015 may be the year that Apache Spark supplants Hadoop with its superior capabilities for richer and more timely analysis.
"There is a strong industry consensus that Spark is the way to go," said Curt Monash, head of the IT analyst firm Monash Research.
"Next year, you will see a lot of [Hadoop] use cases that transcend Hadoop," said Ali Ghodsi, CEO and co-founder of Databricks, a company formed by a number of the creators of Spark that offers a hosted Spark service, as well as technical support for software distributors selling Spark packages.
Spark is an engine for analyzing data stored across a cluster of computers. Like Hadoop, Spark can be used to examine data sets that are too large to fit into a traditional data warehouse or a relational database. Also like Hadoop, Spark can work on unstructured data, such as event logs, that hasn't been formatted into database tables.
Spark, however, goes beyond what Hadoop can easily do, in that it can analyze streaming data as it is coming off the wire.
As such, it can serve as a faster replacement to the Hadoop MapReduce framework for data analysis. In the annual Daytona Gray Sort Challenge, which benchmarks the speed of data analysis systems, Spark easily trumped Hadoop MapReduce, and was able to sort through 100 terabytes of records within 23 minutes; It took Hadoop over three times as long to execute the same task, about 72 minutes.
Initially, real-time processing may not seem like a big distinction, however, such capabilities have been used to create entirely new lines of businesses.
"We've built our intellectual property around Spark," explained ClearStory Data CEO and co-founder Sharmila Shahani-Mulligan. ClearStory Data offers a new business intelligence service that allows teams to assemble a series of data visualizations into a narrative, as if they were a PowerPoint presentation. The data can come from many sources and can be updated as new data comes in.
"People want fast response times. They don't want to wait a day for an answer," Ghodsi said. For instance, Spark could be used to help digital advertisers decide what ad to serve to users based on their last few clicks, rather than on what sites they clicked on a few days or weeks prior. Spark's data processing speed is important, because while the amount of data we collect is growing rapidly, the advancement of computer processing power is tapering off.
Spark also offers a richer palate of ways to analyze data, Monash said. Hadoop's default analysis engine, MapReduce, is chiefly capable of executing one kind of problem, involving the filtering and sorting of data across different servers (the "map" portion of the job) and the summarizing of the results (the "reduce" side of the problem).
Sign up for CIO Asia eNewsletters.