How Backblaze Slashed Mitigation Time by 91% with FireHydrant

Serving over 500,000 customers, Backblaze is a pioneer in cloud storage and backup solutions, known for its simplicity, affordability, and reliability. As their customer base grew, so did the need for a scalable, efficient incident management strategy.

To streamline their response coordination, improve on-call processes, and improve resolution times, Backblaze implemented FireHydrant’s Incident Management and Signals solutions. The result? A 91% reduction in mean time to mitigation (MTTM) — a game-changing improvement in reliability and operational efficiency.

“FireHydrant has dramatically improved our time to respond, which in turn has dramatically improved our time to mitigation, which means we solve problems faster," said Sam Burke, SRE and Incident Manager at Backblaze. "We’ve ultimately made our infrastructure even more reliable."

Scaling Incident Management: The Challenges of Dynamic Environments

As infrastructure expands to meet the increasing demands of rapid growth, incident management teams face challenges:

• Lack of clear ownership means teams scramble to find the right responders
• Without an on-call rotation, teams get off-hours interruptions
• Miscommunication can result in inefficient, all-hands-on-deck incidents

“In a fast-paced environment, it’s easy for miscommunication to occur, especially when multiple incidents could happen simultaneously requiring all hands on deck and making it difficult to organize,” said Sam. “We were looking for a tool that could help us automate organization, lower the load placed on an incident manager, and track everything happening in an incident, including documentation.”

Beyond engineering, Backblaze also needed a way to bring in key stakeholders — such as legal and customer success — easily and affordably.

Signals: The Game-Changer for On-Call Management

To improve operations, Backblaze turned to FireHydrant Signals to automate on-call scheduling and incident escalations. The impact was immediate:

• Optimized response processes, ensuring the right teams engaged immediately.
• Eliminated the cost constraints of per-seat pricing by switching from a legacy on-call solution to Signals’ pay for what you use model.
• Enabled organization-wide incident response — bringing engineering, legal, and customer success into the loop without additional costs.
• Consolidated incident response and on-call management into a single platform, reducing complexity.

By making on-call more efficient and scalable, Backblaze ensured that when an incident occurred, the right people responded immediately — without unnecessary noise or delays.

Empowering Engineers: The Shift to an On-Call Culture

A pivotal shift in Backblaze’s incident management strategy was bringing engineers into the on-call process. This move had significant benefits:

• Clear service ownership: Every engineer knew their role in incident response
• Reduced response times: No more scrambling to find the right person
• Stronger reliability metrics: More accurate SLOs

“You absolutely need to have engineering on call,” said Sam. “We’ve started to see a lot of positive metrics and results from it. We have more accurate SLOs, and our time to mitigate has improved.”

By combining Signals with FireHydrant’s Service Catalog, Backblaze established automatic ownership mapping. Engineers were paged only when their specific service was impacted, eliminating confusion and unnecessary escalations.

“There’s no more ‘Who owns this?’ or ‘Who needs to be on this call now?’ — something we struggled with in the past,” said Sam. “Either everyone was on the call, or we couldn’t get the specific person we needed. Now, it’s clear who should respond.”

This cross-team collaboration also led to better knowledge-sharing, breaking down silos and helping engineers gain deeper insights into Backblaze’s infrastructure.

Centralized Alerting: Cutting Through the Noise

With multiple monitoring and observability tools in place, Backblaze needed a way to prioritize alerts and reduce noise. FireHydrant’s Signals became the central hub for alert distribution.

• Prioritized notifications: Ensuring critical alerts reached the right team
• Reduced alert fatigue: Signals automatically grouped redundant alerts, minimizing distractions
• More efficient workflows: Teams focused on actionable notifications rather than sifting through noise

“I went from receiving two or three pages a day to getting paged only twice in an entire week," said Sam. "The way that Signals handles displaying a notification tree makes it really easy to figure out why an alert didn’t get responded to.”

By streamlining notifications, Backblaze prevented alert fatigue, allowing engineers to respond only when necessary — without getting overwhelmed.

Automated Incident Coordination: Faster Progression from Alert to Resolution

Beyond on-call management, FireHydrant automated incident coordination, ensuring incidents moved from alert to resolution as quickly as possible.

• Slack Integration: Incident channels spin up automatically, tagging responders and pulling in relevant details
• Google Meet Auto-Generation: High-severity incidents trigger immediate coordination calls — no scrambling to set up meetings
• Runbooks for More Services: Teams follow predefined workflows, ensuring a consistent, structured response
• Automated Ticketing in Jira: Follow-ups are easier to track, with tickets generated directly from Slack and centralized in FireHydrant

“We automatically page the appropriate engineers, which dramatically speeds up our response time,” said Sam.

By replacing manual coordination with automation, Backblaze transformed their incident management process into a well-orchestrated, fast-moving operation.

The Results: Faster Response, Stronger Reliability, and Scalable Growth

FireHydrant fundamentally changed how Backblaze handles incidents, delivering significant improvements:

• 91% reduction in mean time to mitigation (MTTM)
• Fewer incidents and more accurate SLOs
• Clear ownership and faster response times
• Reduced incident fatigue across teams
• Scalable, efficient incident management — ready for future growth

By adopting FireHydrant Signals and Incident Management, Backblaze has built a scalable, future-proof incident response system, ensuring their cloud storage services remain fast, reliable, and resilient — even as they continue to grow.