CBT and simulations aside, there are more tangible metrics to collect that will be much more useful in determining actual awareness, as well as how to determine how to optimize budgets.
Metrics that help optimizing budgets are perhaps the most useful to collect. An additional driver in collecting metrics is to determine which components of your awareness programs are actually being used and having an impact. An easy example is embedding analytics in emails and webpages. You can gauge if people are actually using the components that you are spending time and money to create. If they are not, you should quickly determine why they are not using those components and address it, or stop investing in those components.
Some components, like posters, coffee sleeves, and other physical (commonly paper) awareness materials, are harder to measure. In this case, placing QR codes on those materials can give an indication as to how many people are engaging with these components. If you hold events, it is relatively easy to count the number of people who attend the events.
These metrics indicate where your successes or failures are. Time and money can therefore be adjusted accordingly. Clearly, there needs to be some analysis applied to the metrics. For example, if your analytics on email newsletters are low, but you find that the newsletter is being printed and distributed, it is clearly being distributed beyond what analytics would otherwise indicate.
Now that you know you've gotten your employees attention, the next type of metrics is actually more important in demonstrating that you are achieving your awareness goals. Specifically, you want to ensure that you are changing behaviors. That is a very key distinction. You need to measure actual behaviors.
This specifically excludes measuring how many people take a particular training course and any results of the associated test scores. As most security practitioners can attest to, the fact that people say that they know that they should not give out their password does not mean that they will not give out their password when actually asked for it. You need to measure the desired behavior in practice, not knowledge.
For that reason, you need to determine how to measure root behaviors. We have performed extensive research to determine 17 root awareness behaviors and how to measure the associated behaviors. While some behavioral measurements are obvious, such as performing social engineering simulations to determine if a person does not divulge their password to a stranger, others such as mobile device security and phishing require more thought and are dependent upon the resources available to the organizations.
Additionally, metrics can frequently be misleading. For example, phishing simulations can produce numbers, however they only represent the metrics for susceptibility to the pretext being used. For example, if a phishing message involves a cat video, you can only generalize results to people who click on phishing messages containing cat videos. They do not represent the awareness related to phishing messages that appear to come from an executive stating that the recipient must review an attached document.
Sign up for CIO Asia eNewsletters.