Learn from 50,000 incidents with the first Incident Benchmark Report

Conference bridge or chat channel? How many responders do I need in an incident? How important is a service catalog? What’s the average response time by severity? Although we’ve been managing incidents for as long as we’ve been building software, incident management as a dedicated practice is still in early stages — there’s no benchmarking data, no gauge for what’s “normal.” But after analyzing data from 50,000 incidents, we now have a start.

Today we released the Incident Benchmark Report, the largest analysis of the entire incident lifecycle, from declaration to retrospective. Using anonymized data from 50,000 incidents declared, managed and resolved on the FireHydrant platform between 2019 and 2022, the report reveals insights into the when, what, who, and how behind incidents and also highlights behaviors that correlate to faster response times.

Key discoveries from the report

The Incident Benchmark Report lays clear a number of averages when it comes to incidents, including incidents per month by company size, when incidents occur most often, and how incidents break down by severity level. These averages give anyone working on incidents a starting point in tracking their own incident data

But the real gold in the report is the discovery of behaviors that correlate to a faster mean time to resolve (MTTR). It’s worth mentioning before we go forward that we know there are mixed reviews on using MTTR as a measurement. We chose to look at it here because, simply put, there are limitations to how we can measure the success of incident response efforts currently. It’s important to start somewhere, so we started here. 

For example, we found that: 

• Incidents with clearly assigned roles had a 42% decrease in MTTR compared to those with no roles attached.

• Incidents with services attached to them had a 36% decrease in MTTR compared to those with no services attached. 

• The use of a conference bridge has no positive impact on MTTR. 

By understanding the behaviors that correlate to faster recovery time, incident management practitioners have a framework for evaluating their own programs and making improvements in how they manage incidents and how they invest learnings back into their systems. 

Growing trends for 2023

We also saw a few unmistakable patterns from the past year when it came to incident management. Many companies made the decision to up level their practices in 2022, and we expect that to continue through 2023 across four main areas:

More lower-severity incidents: As incident management is becoming a practice about not just more quickly resolving incidents but also learning from them, more teams are using low-severity incidents as investigation and training opportunities. 

More services attached to incidents: This is a reflection of a rise in the “you build it, you own it” mentality that bodes well for incident management. The faster you can get the right service owners and subject matter experts in the room, the faster you can resolve.

More incident retrospectives: As more teams realize the value of incidents as learning opportunities, there’s a growing commitment to converting unexpected downtime into more reliable systems of the future.

More use of external communication during incidents: When you’re known for handling your tough moments well, you build trust among your customers. And based on the trends we saw, more teams are leaning into that principle.

Learn more

The way we manage incidents varies greatly by company, influenced by team size, culture, maturity, and much more. But in order to improve, we all need a place to start, something to compare to. 

We hope you find inspiration in the Incident Benchmark Report and that it helps you take the next step in improving how you manage incidents — no matter where you are today. Read the full report.