Service Status Update: June 11, 2025

Well, that was a fun one! We’ve pulled from our incident transcripts from our incident bridge during the GCP outage that occurred earlier today to provide our public summary below.

TL;DR: Google Cloud had an outage that impacted FireHydrant, but most critical services remained usable – including Signals.

Edited June 16, 2025: We have updated our incident report to include Google's official incident report here and to update the root cause of the incident to be an issue with Google's Service Control. Previous versions of the incident report named Google's IAM service as the primary cause of the incident.

Summary

On June 11, 2025, at 18:05 UTC, a global outage occurred with Google Cloud's Service Control leading to several dependent services to also be heavily degraded. 

The outage disrupted critical services for FireHydrant leading to latency on the web UI and disruptions in integrations like Slack. AI functionality such as incident summaries were also affected due to our 3rd party providers also relying on Google Cloud.

The good news: Signals delivery experienced some minor latency – but there was no loss of data and alerts were sent to devices as expected.

What Happened

• 18:05 UTC: We noticed internally that our environment was experiencing latency and began investigating the cause of the issue with an internally declared incident.

• 18:46 UTC: Google Cloud reported an IAM issue preventing authenticated calls, which disrupted access to critical services such as Cloud SQL, Cloud Storage, and networking.

• 18:52 UTC: We continued monitoring the situation and investigated contingency plans.

• 19:30 UTC: Signals functionality was verified to be operational, with runbooks and other automations continuing to operate without interruption.

• 19:30 UTC: Google Cloud confirmed partial recovery in most regions, except US Central 1. Some Firehydrant services began recovering intermittently.

• 20:16 UTC: Google Cloud reported ongoing mitigations, with most services recovering globally.

• 20:30 UTC: Firehydrant confirmed all affected systems had returned to normal operations but continued to monitor the situation. 

• 20:45 UTC: All systems returned to operational.

Customer Impact

• Web UI: Users experienced latency and loading issues due to disruptions in Google Cloud Storage and IAM. This was dependent on the location a user was attempting to access these systems as well.

• Slack Integrations: Commands like /fh new faced delays or timeouts due to upstream provider issues.

• Incident Summaries: Delays occurred in generating summaries because of backend disruptions to upstream providers.

• Signals: All alerts were delivered with some minor latency. Email notifications experienced the most significant delays.

Commitment to Reliability

We understand that you rely on our services for critical operations. We are committed to maintaining the highest levels of service reliability and transparency. The measures we're implementing will help ensure we meet the standards you expect from us.

We encourage you to subscribe to our Status Page for timely communication and updates on incidents. For any additional questions, please send an email to support@firehydrant.com