"We used to call it complex event processing,'" he adds. But that approach required proprietary software and expensive servers, which limited usage. In contrast, one of today's technologies, Hadoop, "is linearly scalable, and you can throw lots of hardware at it and use memory very effectively," he says. Roll into that the lower cost of flash memory, adds Baer, and "now we can process data very fast, and do more sophisticated processing than when you were bound by I/O."
That said, fast data involves some infrastructure considerations, according to analysts. Companies are more likely to use Linux than Windows for fast data projects, and they're more likely to use InfiniBand networking than traditional Ethernet because the former is faster. And those who want faster analysis might even use dedicated FPGA chips and network processors to watch network traffic, Dekate says.
On the back end, the machines used to understand and analyze workloads will need denser memory because they will need to focus on streaming data rather than stored data. Increasingly, organizations are using high-performance server clusters to accommodate fast data. As is to be expected, given their need for security, risk management and pricing accuracy, companies in the financial services industry have been among the first to embrace fast data. But fast data is taking hold in many other sectors of the economy.
Real-time risk assessment
Western Union is best known for something that most people under 50 have never seen: the telegram. Yet the company is now a $5.5 billion global payment services provider, and in 2013 moved some $82 billion in 242 million consumer-to-consumer transactions and 459 million business transactions. According to David Thompson, executive vice president and CIO at the Englewood, Colo.-based company, the data that Western Union collects from 174 kinds of data sources is increasing at the rate of 100TB per year.
Those data sources are both internal and external. Western Union has relationships with more than 1 million "front-line associates," which range from large companies to cash-dispensing bodegas. It collects data from approximately 500,000 retail agent locations, and also cross-checks information with government agencies that distribute lists of sanctions, politically exposed individuals (who might be subject to extortion threats) and other information.
For example, there's an element of business risk involved when someone in the U.S. wires a large amount of money to, say, India. Western Union has to determine whether there's enough cash in that location to complete the transaction, or whether the recipient should be sent to a different location.
That's a customer service issue, but there are also risks to watch out for, such as the potential for fraud. "If we become aware of potential fraud, such as elder abuse, or we suspect that a recipient is engaged in a scam, we can ask the sender in real time if they know the recipient personally," Thompson says.
Sign up for CIO Asia eNewsletters.