One thing CISOs need to appreciate is not whether the hype is right or wrong. "Hype will exist until it is right or wrong," said Peraldi, "accept the hype and work through the challenges."
Joey Peloquin, director of cloud security operations for Citrix, said that the industry is at a point where vendors believe that they can't be taken seriously unless they are inserting the buzzwords of AI and machine learning.
"It really isn't my algorithm is better than yours, though. Machine learning and AI are a little different, and overhyping is doing a disservice," Peloquin said.
More to the point, it's important to examine the elements of a data science strategy and understand why we are interested in these tools. "We have an immense amount of information today that human beings can't analyze in a meaningful way," Peloquin said.
Intelligence is overhyped, potentially because of its sundry definitions across both the public and private sector. "At the end of the day, it's about intelligence. What data science is about is being able to leverage the huge amount of information we have, and to analyze it, enrich it, and make it actionable in a proactive instead of a reactive way," Peloquin said.
When developing a particular tool, data science informs which path to go down said Peloquin. "Vendors that are overhyping don't understand the problem we are trying to solve."
So, how do you know whether their algorithm is doing what it's supposed to do?
In order to make informed decisions, CSOs should ask vendors questions like, Do you have a Phd data scientist on staff? Who leads your team? Where are they from? What is their background and experience?
"They [CSOs] need to be smart enough to ask the vendor to ensure that their products are not just marketing speak. If they [the vendor] based all of their capabilities on the output of automated tools rather than experts in the field that can do targeted attacks, then I would argue that their solution is not as mature as they are claiming it to be," Peloquin said.
There are products that make the claim that they have blocked 100% of their attacks with no false negatives, but Peloquin said it's important to ask whether the pen testing has been done by an actual person who knows how to bypass controls and customize payloads.
Then, said Peloquin, go and research those individuals. "Don't take the responses from a vendor at face value."
Data science is changing the game, said Peloquin. "I'm excited about how it is enabling us in the security field to do so much more so that we are not wasting money on solutions that are not going to have a major impact."
Sign up for CIO Asia eNewsletters.