Big data technologies such as Hadoop, MapReduce, Pig and Hive give companies the ability to dig in at a level they simply cannot achieve with traditional security tools, he said.
Scott Crawford, an analyst at Enterprise Management Associates, said the fact that a vast majority of companies these days do not even know when they are breached highlights the need for a more data-driven approach to security.
"We have been plagued by much blindness when it comes to threat awareness," Crawford said. "Most people are taking weeks if not more to discover a breach. We are not seeing what we have to see."
The biggest limitation with current security tools is that they depend on alerting rules and triggers that are based on what is already known, Crawford said. "You have to build rules predicated on what is known," about a threat to detect the threat.
Most current tools do not fare very well when it comes to dealing with unknown threats. They do not support the sort of querying that is possible with big data analytics technologies, he said.
Despite the benefits, there are some major caveats associated with big data analytics in the security realm. The biggest has to do with the lack of professionals familiar with Hadoop and related technologies. While integrated products such as the one from RSA this week mask a lot of the underlying complexity, they still require a certain degree of knowledge of big data analytics.
The companies that will benefit from such tools are most likely going to be very forward looking ones with the skills and the resources needed to pore through and analyze big data sets, Crawford said. "This will become pervasive over time. As these technologies become more widely adopted they will become more commonly integrated into security tools."
Sign up for CIO Asia eNewsletters.