Alert fatigue

What is alert fatigue?

Alert fatigue is a type of burnout that exists when on-call engineers are getting paged to deal with incidents too often. Sometimes alert fatigue can be so severe that teams become desensitized to incidents and, as a result, may fail to properly respond to incidents.

Although alert fatigue is commonly associated with the healthcare industry, on-call engineers also face a seemingly never-ending barrage of notifications across multiple platforms, all demanding their attention. Even the actual sounds that indicate an alert can lead to alert fatigue: continuous noise from operating complex systems keeps our brains on high alert, which releases stress hormones. 

Warning signs of alert fatigue 

Symptoms of alert fatigue appear before full burnout. Warning signs include:

• Ignoring recurring alerts 

• Feeling angry when incidents arise 

• Finding it hard to keep track of notifications during incidents 

• Receiving alerts every day 

• Losing the desire to resolve incidents, whether on your own or with a team

Why is it important to acknowledge alert fatigue?

Continuous messages contribute to cognitive stress that is difficult for engineers to overcome. In Overload and Boredom, O.E. Klapp says, “...a large amount and high rate of information act like noise when they reach overload: a rate too high for the receiver to process efficiently without distraction, stress, increasing errors and other costs making information poorer."

Although warning signs might be easy to ignore at first, ongoing effects of alert fatigue include: 

• Worsening outages due to task overload

• Desensitization

• Employee burnout

• An increased possibility of more significant errors

• A negative financial impact on the entire company

How do you minimize alert fatigue?

While it’s impossible to prevent 100% of all incidents, it is possible to reduce the possibility of alert fatigue by creating a supportive culture around incidents.

Incident response teams should: 

• Create alerts based on customer impact

• Split alert triage responsibilities across a broader range of roles, including non-engineers

• Strengthen your incident management plan to help streamline high-stress situations

The more decisions we have to make or tasks we must remember to do while responding to an incident, the more likely we are to make a mistake. Some teams may benefit from investing in automation tools to remove rote tasks during incident response. Products like FireHydrant automate assembly, communication, and documentation, shortening the lists of on-call engineers’ responsibilities during incidents.