Cybersecurity Alert Fatigue: An Analyst Perspective
The following is a new analyst’s perspective on the well-trodden concept of alert fatigue in cybersecurity. I chose this topic because I noticed a lack of content written about this phenomenon without the usual “the system is broken, and only I know how to fix it!” subtext. Also, of the top 10 articles returned by Googling “cyber security alert fatigue,” I found none that were written (or even quoted) cybersecurity analysts. You know — the people who actually handle the alerts. There is some great and important writing in those 10 articles, but it’s perspective is dominated by vendors, security journalists/researchers, and C-level executives.
Finally, I wanted to offer a perspective that is clear-eyed, honest, but free of negativity. As John Strand said in a recent Pay What You Can SOC fundamentals course (paraphrasing here), “If you want to work in this field, make sure your heart is in it. Because the world has enough falafel trucks run by former SOC analysts.” So, below I volunteer my one point of view on the alert fatigue topic, especially for the many, many talented people who will be transitioning into analyst positions in the coming months and years.
First, some background to get out of the way. Early in 2021, I embarked on a new career in cybersecurity. Following a 2020 dominated by a pandemic, job loss, unemployment, a cybersecurity bootcamp, and late nights spent studying in the hopes of becoming certified, I was ready for the awesomeness that I knew a career in cybersecurity would be.
And I wasn’t disappointed! As an analyst serving on a Managed Detection and Response (MDR) team, I operate on the front lines, applying my knowledge and existing skills to defend customer networks, respond to potential threats, hunt for malicious activity, and investigate suspicious traffic, all using shiny, state-of-the art tools. My team is professional, supportive, and pushes me to do my best and realize my potential. I learn something new every day, which is a main driver for why I wanted to switch to cybersecurity in the first place. It is a thrill to read a CISA bulletin or Threatpost article over breakfast, only to be hunting for the exact same indicators of compromise at work an hour later.
In many ways, my new job is a dream come true. To anyone trying to break into cybersecurity who may be reading this, who may be wondering if it’s worth the late nights of studying, or if they have what it takes: it is and you do.
It is also the most mentally taxing job I have ever worked, bar none.
Alerts, threats, detections, incidents: the nomenclature may change with the vendor, but the message is the same, and it is relentless: Ahem. Your attention is needed. Now. Look over here! PAY ATTENTION TO ME!
As an analyst, alerts are a fact of life. So are false positives. Particularly in an environment where your attention is spread across multiple customer environments, performing this work in a shift setting (particularly a 10 hour shift) incurs a mental debt that will come due. Consider the phrase, “pay attention.” A lot of people who use this phrase focus on the word attention. As a cybersecurity analyst, you will become keenly aware of the reality that attention is paid from a limited reserve which diminishes the more it pays out.
My suggestion? Have a plan in place for what to do after your shift ahead of time, and do not expect to maintain peak mental performance. I go for a run, take a nap, or perform a largely mindless task like vacuuming or grocery shopping at a supermarket, the layout of which I am extremely familiar with. Critically, I do not attempt to decide what to cook! That would require far too much mental effort. Know when that well is dry, and have a plan (dare I say, runbook?) for what to do until the rains come again.
Besides the sheer mental effort, another challenge that caught me somewhat off guard was a set of twin and often-conflicting priorities that permeate the work of a cybersecurity analyst:
- Priority 1: Stay on top of your alert queue. Don’t let them pile up. Keep within the SLA. Respond promptly.
- Priority 2: Provide quality analysis. Don’t jump to conclusions — perform in-depth investigations. Never cut corners. Do not write off an alert that turns out to be (or lead to) a breach.
Human brains are not Kubernetes clusters.
We can’t add additional nodes to a cluster or spawn additional process threads when things get busy. The best we can do is be efficient. For instance, we can use password managers to handle credential recall efficiently without sacrificing (much) security. We can write scripts to automate repetitive tasks. And we can create shared resources and configurations that help us get up and running faster. But the truth is, it’s going to get rough, and the challenge requires a certain degree of mental and emotional resilience to withstand.
To avoid getting overwhelmed when I fall behind, I use a phrase like, “It’s not me, it’s my environment. I will stick to the plan and get this done.” This reminds me that I am enough. I will get faster and more efficient. But today I need to follow procedures to prioritize alerts, do my best to cover my bases, investigate thoroughly, and ask for help when I need it. I recommend reading that last item again. Asking for help is not a sign of weakness, failure, or incompetence. It should be encouraged.
The leadership of organizations that stigmatize asking questions or requesting assistance should take immediate corrective action, or risk having threat actors do it for them.
Alerts don’t give a damn about your shift schedule.
The last “real life” alert fatigue challenge to mention is this: alerts don’t give a damn about your shift schedule. You could be 15 minutes from the end of your shift, and handling your 30th alert of the same type of the day when a true positive emerges. I have been in a lot of workplaces where late afternoons get pretty lax: coffee breaks get longer; meetings go overtime and veer off on tangents. Cybersecurity analyst is not a role that tolerates this.
Frankly, it sucks when a break or shift ending evaporates because an alert demands a thorough investigation and follow-up. But as cybersecurity professionals, we signed up to get the job done. And thus far, I have not found that this occasional occurrence has dented my enthusiasm for the work.
In closing, I offer two suggestions to those who manage cybersecurity analysts (or anyone who handles alerts).
First, ̶ ̶c̶h̶e̶c̶k̶ ̶o̶u̶t̶ ̶m̶y̶ ̶A̶I̶ ̶p̶l̶a̶t̶f̶o̶r̶m̶ ̶-̶ ̶i̶t̶ ̶w̶i̶l̶l̶ ̶r̶e̶d̶u̶c̶e̶ ̶a̶l̶e̶r̶t̶ ̶f̶a̶t̶i̶g̶u̶e̶ ̶b̶y̶ ̶9̶0̶ ̶p̶e̶r̶c̶e̶n̶t̶!̶ consider how to load balance alerts among your teams. Network load balancers are an awesome technology that can be applied to the problem of alert fatigue — not to solve it, but perhaps to mitigate it. Are all analysts pulling from the same queue(s)? Perhaps your team could benefit from a round-robin approach, or one that takes into account current capacity, affinity for certain types of alerts, or some other scheme that makes sense for your staff.
Second, consider hiring more individuals transitioning from the service industry or other time-sensitive and customer-focused roles. I have never worked in the service industry, but my impression is that success as a waiter, bartender, barista, deli clerk, cook, or any number of other demanding and customer-facing positions would qualify an individual well for the types of challenges described above.
Oh, and get a second monitor. You’re gonna need it!
