Credit: Shutterstock/ Computerworld
A lot of security processes failed during the breach of Target's systems during last year's holiday season, but one surprising revelation was that the retailer actually did receive security alerts about the malware in its system. Yet because the security team was bombarded with alerts -- estimated at hundreds per day -- it couldn't adequately prioritize them.
Both within the security field and in other areas, that's a problem that a lot of companies face in the big data era: The top priority is to dig out useful insights from the wealth of data that's coming at them from multiple sources. They're trying to get the right data to the right person at the right time; in fact, the goal is to deliver insights in near real time. The result is a discipline that many in IT are calling "fast data."
"We don't have a formal definition of it," says Chirag Dekate, an analyst at research firm IDC. "But it relates to any kind of big data workload that requires massive brainpower and complex algorithms to extract data from it. The need has arisen because of the urgency of analyzing data as the data is coming in."
Another IDC analyst, Steve Conway, explains the challenge this way: "You have to get rid of everything extraneous and do it quickly. Some of it needs to be in real time, like credit card fraud detection. Because if you don't do it in real time, you don't catch it." There's also an element of predictive analysis involved in fast data, because you can identify patterns with more recent -- and thus more accurate -- information.
Indeed, the need is so widespread that different approaches to fast data are emerging in different industries, from financial services to e-commerce. But the goal in every case is to take advantage of better insight sooner. Here's a look at the phenomenon known as fast data, the technology behind it and some of the companies that are putting it to work.
Symbiotic technology unleashed
New database technologies -- Hadoop, NoSQL, HBase and others -- are helping the fast data effort come to fruition by enabling greater flexibility in reading data. Oliver Downs, chief scientist and CTO at Globys, a Seattle-based developer of software that helps service providers analyze customer experience, says what those database technologies bring is "the ability to shape data in usable form when you read it rather than when you write it." The result: It's possible to analyze data sooner than it once was.
However, it's not just advancements in database technologies that are making fast data possible. "This is not a new type of application," says Tony Baer, principal analyst at Ovum.
Sign up for CIO Asia eNewsletters.