Managing Incident Response with Slack

This article describes how to best utilize FireHydrant's Slack integration. If you haven't already set up the integration, read Integrating FireHydrant with Slack for details.

At its core, the FireHydrant integration with Slack is designed to be a low-friction way of opening an incident, compiling notes and messages automatically, and mobilizing team members quickly. This article describes how to get started with FireHydrant commands in Slack.

Declaring an Incident

Declaring a new incident is simple! To get started:

1. Directly from Slack, run:

   /firehydrant new

2. Add the details you want to specify (the only required field is the incident Name ).

3. After filling in the details for the incident, click Open. FireHydrant automatically creates a new Slack channel for the incident. You can customize this automation step by creating or editing existing Runbooks.

Using Incident Channels

The FireHydrant Slack integration acts as your scribe, recording all of the chatter and attachments pasted into the channel for a specific incident. For example, if you add an image of a graph in the channel, FireHydrant stores that image in your timeline automatically.

Automatic channel notifications

FireHydrant enables you to configure channels that automatically receive incident notifications. You can do this by adding a "Notify Channel" step to your Runbook for a given incident.

Assigning Incident roles and teams

You can configure as many incident roles in FireHydrant as you'd like. Incident roles are valuable for quickly delegating responsibilities to responders during an incident.

To assign a user a role during an incident, simply run:

/firehydrant assign role

A dialogue box opens where you can select an existing FireHydrant user and assign them a role.

All of the roles you create in FireHydrant for your organization are available through this command.

While assigning a role to a user is great if you know exactly who to bring in and with what role, most of the time you won't have a great idea of which person on a separate team is the current on-call, whether they have their manager automatically added to incidents that impacts their team, or if they have a shadow rotation going on to train new commanders. In this case you can use teams in FireHydrant. Teams are collections of roles, people, and on-call schedules from your alerting provider. With configured teams you can ensure you're bringing in the right people by running 

/firehydrant assign role

Have no fear, we'll show you a preview of who you'll be assigning before you assign them.

When you assign roles to users, they’ll receive a direct message in Slack (assuming their FireHydrant account is linked with Slack) saying they’ve been assigned the role. You can add team members within the FH UI by going to Organizations > Members.

Evaluating impact and running incident communications

From inside a dedicated incident response channel (e.g., #incident-343), you can add impact to the incident, meaning you can specify the services, environments, and/or functionalities impacted by the incident, by running

/firehydrant edit

After impact has been determined and you have the correct people responding to the incident the next step is to evaluate communications external to the incident responders. Whether that it internal stakeholders or the general public, timely communications ensure that you are building trust with your customers. To post an update run:

/firehydrant update

From here you can update the incident milestone, incident impact, and a status update message. When you have active incidents on your status pages, you can post these updates to those status pages.

Adding tasks and action items

You can add tasks during an incident-- to create a ticket that needs to be addressed during the incident. You can also assign a task to a specific user. To create a new task, run:

/firehydrant add task

Tasks can be managed by running

/firehydrant tasks

Which will show you a full project management modal dedicated to tasks.

This modal allows you to filter on multiple facets

• Assigned user
• Task state
• Task List
• Unassigned tasks

Additionally, you can quickly assign and change the state of tasks from this modal as well.

You can add action items during an incident-- to create a follow-up ticket that needs to be addressed in the future. You can also assign an action item to a specific user. To open the action item dialog box, run:

/firehydrant add action-item

Note: If you're using our JIRA integration, adding an action item automatically creates a ticket in JIRA.

Editing and Deleting messages

When messages are edited or deleted in Slack, those changes will be reflected in the FireHydrant incident timeline as well. Once a retrospective for an incident has been completed, this functionality is disabled.

Resolving incidents

Once you’ve fixed that pesky incident, mark it as resolved in the incident channel by running:

/firehydrant resolve

This automatically marks the incident as resolved.