Then there is the sophistication of phishing messages to consider. I can purposefully manipulate the user response rate, if I choose. For example, if I want to show success in the program, I can create a very sophisticated message that uses inside information and is related to some timely event, and get a very high response rate. I would then follow it up with a more generic phishing message, such as a shipping message with poor grammar, and would get a very low rate.
The referenced article states that if phishing simulations get a 10 percent response rate, the effort is a success. As the previous paragraph highlights, a 10 percent response rate can mean little in actual effectiveness, depending upon the simulated phishing message used. However, even if you assume it is the most sophisticated simulated phishing message ever, that means that a significant number of people within an organization will still respond to the message.
More frequently, users begin to recognize the simulated phishing messages and do not respond, not because they are more aware of phishing concerns, but because they are aware of the simulations. Another common occurrence is that if one person detects a phishing message in an organization, they may then warn their coworkers about the message. The coworkers will then know to proactively delete the messages. In more than one simulation I was involved in, companies proactively warned employees that they will receive a simulated phishing message within a given time period for political reasons.
Phishing messages require technical failures to be successful
While security professionals seem to attribute responses to phishing messages as a demonstration of poor security awareness, it is actually a much more complicated issue. Again, there had to be a technical failure for messages to get to the user. More important, just because a user responds to a message, it does not mean that there should actually be a loss.
Click throughs to malicious websites can be blocked. Malware can be prevented from installing. So even if a user exercises poor awareness, there should not be a loss related to the user actions, as the attacker was prevented from achieving their intended goal.
What is phishing awareness success?
Given that reduced responses to simulated phishing messages is not a good measure of success, and even when there is success, you do not know if it equates to a loss, a different metric has to be utilized. To determine that, you have to understand the losses that occur due to phishing.
As real phishing messages typically intend to get people to either download malware or give up credentials, measures of related incidents should be used to determine the success of phishing reduction measures. You are looking for actual measures of phishing success.
Sign up for CIO Asia eNewsletters.