Red Herring or the Reel Deal? Behavioral Insights in Security Awareness
Question 1: How do you get some positive reinforcement (as opposed to "you messed up") into your training?
I agree that positive reinforcement is an important part of training and especially important in this context as researchers suggest that running repeated phishing campaigns for training purposes can erode trust in an organization.
Options for positive reinforcement in training might include:
- Incorporating “Phishing IQ tests” into training. In these tests, employees are asked to separate phishing emails from regular emails in an inbox task. Taking this approach would allow you to vary the difficulty of the sample emails so that employees succeed on some problems, while possibly stumbling on others.
- Providing positive feedback when employees report a phishing email through the relevant channel.
- An extension of the reporting idea above might be to divide employees into teams and publish a leaderboard/award a prize (e.g. a donation to a charity of choice) to the team that identified phishing emails and reported them most often.
The above are simply initial ideas - it would be important to rigorously test their effectiveness before roll out.
Question 2: Do you recommend starting with an "easy" to get right phish campaign first? So people are successful (hopefully) initially.
In our project, we sent phishing emails to police officers to understand whether or not the training we’d delivered previously was effective. So, for us, it was important that our phishing emails closely mirrored the phishing emails that we were seeing in the real world.
As I mentioned in my answer above, some researchers suggest that repeatedly phishing your employees undermines organizational trust, so I am not sure I would recommend a phishing campaign as a form of training, whether or not it starts off easy. If you were training your employees using a phishing IQ test, as I suggest above, then I think there should be a range of email difficulties, mimicking the real world as closely as possible.
Question 3: Do you recommend sticking to 2 groups when testing, or would the data from 3 to 5 groups be just as useful? Just thinking about optimizing results.
The number of groups you can robustly test depends on:
- how many different types of training you want to test,
- the available sample size to test with (e.g. # of employees in the company) and
- how small an effect (e.g. % reduction in clicks on a phishing email) you’re interested in.
If your available trial sample is fixed (e.g. you’re working with a company with 2,000 employees), then there are tradeoffs between these considerations. I explore these in more detail at the end of this answer, if you’re interested.
As a very rough rule of thumb, we often say you should have at least 1,000 participants (e.g. employees) in each test group. As discussed in my presentation, it is important to have a comparison or “control” group to see what would have happened if you did nothing e.g. you did not provide phishing training to your employees. This means that if you want to test one type of training, you should, ideally, have at least 2,000 participants in your experiment -- 1,000 in the training group; 1,000 in the control group. If you want to test two types of training, you should have 3,000 participants -- 1,000 in training group 1; 1,000 in training group 2; 1,000 in control. However, it’s important to note that if you want to run a robust experiment, you should conduct statistical power calculations to determine exactly how many participants you need to detect a statistically significant effect.
Experimental tradeoffs: Adding additional test groups will reduce the number of participants in each group. For instance, with 2,000 people, testing 2 types of training would give you 1,000 participants per test group whereas 3 types would give you only 666 per test group. This reduction in the size of the test group reduces the “sensitivity” of the experiment, meaning that you might only be able to detect large differences between test groups e.g. a 20% reduction in clicks on a phishing email rather than a 10% reduction. If you ran an experiment where you found that you reduced clicks by 10% but your experiment was only sensitive enough to detect a 20% reduction, then you would not be able to say for sure whether your 10% reduction was the result of your training or of chance.
Question 4: Is it most effective to deliver training right after credentials are caught (embedded)?
Based on what we found in our experiment with the Metropolitan Police, I would say yes -- we found that embedded training was more effective than our training emails both 3 weeks and 3 months after training.
However, it’s important to note that this may not be the case with all training types in all organizations at all times. I think it would be worth testing embedded training against other forms of less timely training in future experiments.
Panel: Learning Diversity in Security Awareness
Kim Burton, Talya Gepner, Lauren Zink, Kelley Bray
Question 1: Any recommendations for a staff of 1 with little budget on how to build an effective awareness program?
Lauren: Use the people and resources you have internally. Even if they don’t work directly for you as part of your program they can still play a huge role in the program’s success. Work with HR, benefits, communications, legal, marketing, IT and see what kind of campaigns they are doing that you can work together on to include security elements. Also, tap into your SME experts within the security team. A lot of the team needs CPEs and developing and delivering training and communications can count as hours for them.
Question 2: Great to hear about a cybersecurity program manager role. Can you let me know if you have a technical background and any IT certifications?
Lauren: I started out teaching for years and was an adjunct at a local college where one of the other perks of teaching there was taking courses for free so I took courses in Homeland Security Information Technology. This afforded me the opportunity to take courses in business continuity, computer networking, information systems, etc.. Since this field piqued my interest I also took training and courses on my own to expand my knowledgebase on the technical side. Also, once I got my foot int the door to the cyber field I took every opportunity I could get to take training, go to conferences, and attend free webinars and courses to be more well versed in all things technical which even led to me taking on more technical roles in vulnerability management and incident response. At the time I took on these roles it was scary, but at the time it truly set me up for success to build better and more well-tailored security awareness programs.
Question 3: How do we incorporate cultural sensitivities into a good awareness program? What's acceptable in one region may be awkward or verboten elsewhere. Because of this, we tend to play it very "safe" with our content and programs.
Lauren: Build relationships with people in different regions of the company that you can consult with on your content in addition to already consulting with communications and marketing. When I worked at a company with 26,000 employees globally I commonly ran into the issue of things not translating well, both words and pictures. Having resources you can rely on to run things by is very helpful. Also, these resources can help provide some insight into the daily culture where they are located and can help drive some ideas for tailored content.
Question 4: Does a given program rely on a product solution only? If not, where are the critical attributes of the SAT program?
Lauren: I personally don’t believe that there is a product solution that can cover every aspect of a successful program. I prefer a hybrid approach of utilizing products, tools and resources to expand your program, not be the end all-be all. Plus, I have found that a product or solution that worked at one company for me may not be the best fit at another company.
Question 5: What are virtual programs we can do to continue awareness in the COVID situation?
Lauren: Host online speakers, both internal to your company and external. Maybe even set up a security week with a theme where speakers and topics are scheduled out and people can jump on as they are available. Host open online office hours where you make yourself and the security team available, could be formatted as an AMA or a coffee break or even a game theme, just do your best to get creative. Utilize different tools and platforms you have internally that everyone in your company can access and share different security content frequently and try to make it engaging so you can have the interaction that is missing from the office setting.
Closing Remarks: Cybersecurity Awareness, 2030
Question 1: Did you actually figure out how to use breathing techniques so you could get someone's password? That would be extremely interesting.
Yes, it's more about connecting with people and building trust. This can manifest in a few ways.
One would be getting someone to feel sameness with you and it makes them more available for the questions (much like the hypnosis tricks of creating consensus or the "yes" statements). The other is something that cold readers use. They will match your breath and then ask questions. The deviations will often show that you are off track. So you can do everything from understanding words / phrases that resonates with them all the way to the actual password.