Automation was needed here, as well, and now IDT turned to another vendor, Hexadite.
Today, Hexadite receives the alert within a second after it comes in, and sends behavioral data to Palo Alto and other sandboxes for analysis.
A full behavioral alert is ready within 18 seconds, and other information is collected in the next 40 to 60 seconds, he said.
The entire alert investigation process now takes a total of one and a half minutes, and those alerts that turn out to be significant are funneled into the automated remediation process.
For user workstations, a confidence level of 95 percent or so knocks it off the network and sends it in for automatic reimaging. For production systems, that happens at a confidence level of around 30 percent, since it's easier to rebuild them quickly.
There are still occasions when real people need to get involved, Ben-Oni added.
"But what they're looking at right now is a clearer storyboard of what actually happened," he said. "They get the results of a full automated investigation on their screen."
Hexadite came in about six months ago, he said. It took about a week to get started with the first set of 20 to 30 machines, Ben-Oni said. The system was extended to cover the rest of the company's infrastructure in stages.
At the end of the day, automation was not a choice, but a necessity, he said.
"As a public organization, it's incumbent on me to do this," he said.
Not everyone is ready to go this far in automating their security response, however.
"It's not practical in a business setting," said Andy Woods, director of commercial cybersecurity at BAE Systems. The company provides outsourced incident response services.
The big risk, he said, is overreacting to false positives and trying to re-image too many desktops at once.
"It could take down your network," he said. "You could be performing a DDOS on yourself."
In addition, he said, attackers are always innovating and threat indicators change constantly.
It takes a trained analyst to tell whether a threat is real or not, and to adjust indicators as needed, he said.
"Most security professionals are wary of enabling automated 'active' responses that could cause an interruption the very services they're chartered to protect," said Mike Paquette, vice president of security products at Prelert, a security analytics company.
But many organizations are already using automation, such as to automatically block network traffic to known bad sites, or sandboxing networks to detect and block malicious executables.
"I predict that we'll see accelerating adoption of automated incident response over the coming years, guided by the combination of machine learning and human expertise," he said.
Sign up for CIO Asia eNewsletters.