On Tuesday, my company, Mammoth Data, releasedbenchmarks on Google Cloud Dataflow and Apache Spark. The benchmarks were primarily for batch use cases on Google's cloud infrastructure. Last year, Google contracted us to implement some use cases and extract user experience data points from people experienced in this field. As a follow-on, we did a benchmark for Google to see how its technology stacked up.
Benchmarks are often a black art of vendor-driven deception. I've never worked with a company more concerned with avoiding that. The benchmarks we released were constructed around Google Cloud Dataflow and Spark's batch processing capabilities. They don't address the more rapidly developing parts of both engines: the streaming portion.
We also wanted to avoid a "best SQL predicate pushdown" comparison. Because some queries don't distribute well, Spark and Google Cloud Dataflow push the SQL to the underlying datastore. Benchmarking that would largely be a database-tuning exercise and, in my opinion, not very productive.
What is Google Cloud Dataflow?
Google Cloud Dataflow is closely analogous to Apache Spark in terms of API and engine. Both are also directed acyclic graph-based (DAG) data processing engines. However, there are aspects of Dataflow that aren't directly comparable to Spark. Where Spark is strictly an API and engine with the supporting technologies, Google Cloud Dataflow is all that plus Google's underlying infrastructure and operational support. More comparable to Google Cloud Dataflow is the managed Spark service available as part of the Databricks platform.
Google Cloud Dataflow is a great choice for well-thought-out, production-ready jobs. However, the technology lacks read-eval-print loop (REPL) support, and it's bound to Google's cloud infrastructure. Apache Beam -- the API portion of Dataflow -- is an Apache Software Foundation incubation project. Beam supports Spark, Flink, and Google Cloud Dataflow as execution engines.
What the benchmarks say
In our testing, Google Cloud Dataflow was faster than Spark by a factor of five on smaller clusters and a factor of two on larger clusters. At the moment, Dataflow is limited to 1,024 cores; this is a hard limit set by Google that we expect will change in the near future. However, even most larger organizations are running smaller workloads. (Spark has been benchmarked at 8,000 cores.)
While you can toggle how many cores you want, Google's "autoscaling" will be a major boon to companies doing large batch jobs. Assigning too many cores may actually slow a job down on any of these engines. When running in the cloud you need to think a bit more about running jobs cost-effectively. The autoscaling feature allows you to do that. During the benchmark, autoscaling provided roughly the same result as when we manually picked the right number of cores for the job.
The great big data race
Sign up for CIO Asia eNewsletters.