"We've been able to reproduce results that took them months or years to get in a matter of minutes," he said.
Local and global training
In cybersecurity, regular updates are important for any kind of machine learning system because the landscape changes so rapidly.
Without regular updates, all systems become obsolete because humans are always coming up with new stuff. Employees start doing new things. Vendors change their applications. Customers change their shopping patterns. And, of course, hackers invent new malware specifically designed to bypass existing systems.
Meanwhile, there's a window of vulnerability until the next update comes out.
In particular, bad guys can buy copies of the security software and test their attacks against them until they find something that works.
"Then they can use that on all of that vendor's customers until the next update comes out," warned Mike Stute, chief scientist at managed networking company Masergy Communications.
One solution, he said, is to move away from the one-size-fits-all approach used by many security system vendors.
"You can work with local patterns, peer patterns, and industry-wide patterns, and update them at different rates," he said.
Masergy uses a certain number of global factors to look for the likelihood that something suspicious is happening, then combines it with unique local indicators.
A global system can only look at at a limited number of inputs, he said. "There is only so much space. I look for the features that occur most often."
The additional local focus allows the addition of many more inputs, he said. "In the local model, I don't have to compress them down to the smaller set of features."
That allows not only for uniqueness, but for much better accuracy, as well, he said.
The combined local and global approach is also the one used by Acuity Solutions, which makes the BluVector appliance that uses machine learning to detect cyber threats.
Based on an advanced research program for U.S. government agencies, the system starts out with years of good software and learns what benign code looks like.
"Our engine is good at looking at a piece of code and saying, this piece of code has the absence of features that you would expect to see in benign code," said Acuity CEO Kris Lovejoy.
But then it also incorporates new learning from individual customers.
"We have pre-trained our engine before we give it to our customer, and then from that point on, it's almost like the child has left the nest, and it will continue to learn within the customer's environment," she said.
The main engine also gets updates quarterly based on global data, but the unique, customer-specific data isn't shared across the system.
Sign up for CIO Asia eNewsletters.