While there has been a lot of hype around Spark, the survey found that 42 percent of organizations use Spark for educational purposes but have no real project using Spark as of yet. A third of respondents say Spark is primarily in development today, while 25 percent say they have deployed Spark in development and production.
"There's a lot of excitement around Spark, but very little real-life deployment," Aziza says.
"If you look at those planning on using Hadoop, most people go in thinking, 'I'm going to be using Spark as my primary engine.' But when you actually start using Hadoop, most people use Hive," Mariani adds. "You would never use Spark for an ETL pipeline. You're going to use Hive for that. But we would never use Hive for interactive queries; we'd use Spark or Impala for that."
It should be noted, however, that organizations that have deployed Spark in production were 85 percent more likely to achieve value.
When it comes to concerns around big data, accessibility, security and governance have become the fastest growing areas of concern year-over year, with worries related to governance growing the most at 21 percent.
Sign up for CIO Asia eNewsletters.