There are tool sets, some of them open-source, like Apache Hadoop, that can gather, share and analyze the constant rush of structured and unstructured data flowing through networks -- they offer speed and the ability to draw connections among seemingly random, unstructured sets of data.
And the ability to access and analyze all that data leads to intelligence. Kim Jones likes to talk about the differences among data, information and intelligence. One of his favorite examples is a seemingly random -- at first -- 10-digit number.
"Maybe it's just a number in excess of three billion," he said. "Maybe it's an overseas telephone number. Maybe it's a 10-digit barcode of something. Or maybe it breaks down to a U.S. telephone number, which in this case is what it is.
"If I add that to other pieces of information that may exist out there, such as the first three numbers -- 301 -- being the area code for Maryland and the fact that I used to live in Maryland back in the late '90s, you might be able to do some predictive analysis and extrapolate that this is my old phone number."
Bob Rudis, director of enterprise information security and risk management at Liberty Mutual, bristles at the buzzword "Big Data," preferring "large-scale, aggregated security analytics" instead, but said he does see organizations, "including the one I work for, embracing the potential of the advancements in security-oriented data analytics to help speed up and generally improve forensic investigations.
"Something that may have taken an organization a few hours or days to get intelligence on can take minutes with the right people, processes and technology."
Rudis said Liberty Mutual is also, "part of a regional, cross-sector group that is working to develop a way for member organizations to share their security-oriented data into one large system that would then be able to do very large-scale analytics across organizations for one purpose -- being able to share known attack indicators as well as see if there are already indicators on those networks."
Eddie Schwartz, CISO at RSA, said Big Data turns the traditional model of investigating and defending against attacks on a network "on its head by adding new content, context and analytic methods.
Schwartz said Big Data allows a, "predictive and proactive model," that by focusing on the entire operation of a business, including transactions, can identify or even anticipate attacks.
And insurance companies investigating an accident can now combine data from automobile sensors with weather readings and traffic data, to get a better understanding of the conditions surrounding a claim.
Simply having tools and data not enough
But those investigative advantages come with more demands and more risks.
Sign up for CIO Asia eNewsletters.