Managing Incident Response with Slack

This article describes how to best utilize FireHydrant's Slack integration. If you haven't already set up the integration, read Integrating FireHydrant with Slack for details.

At its core, the FireHydrant integration with Slack is designed to be a low-friction way of opening an incident, compiling notes and messages automatically, and mobilizing team members quickly.

This article describes how to conduct an incident end-to-end within Slack.

Declaring an Incident

You can declare an incident running the new command or from converting an existing message/thread from your Slack instance.

Alternatively, you can also skip manual declaration and automatically start incidents from your alerts using Alert Routing.

Using the new command

1. In any channel or DM in Slack, run:

    /firehydrant new
   

   You can also use our other command aliases:

    /fh new
    /incident new
   

2. This opens a modal where you can fill in details about the Incident.

   • This modal is configurable. Learn about customizing the modal.

3. After filling in the details for the incident, click Open. FireHydrant will automatically attach any Runbooks and execute steps within them that you've configured.

From a message or thread

1. From any channel or thread, click the ellipses in Slack next to a message. There will be an option to open a new incident as well as add the message to an existing incident.

2. Click Open a new Incident. This pops open the same declaration modal as you've configured, but this time, FireHydrant automatically inserts a link to the message in the description of the incident. Fill in information as needed and declare the incident.

Using Incident Channels

If you've configured a runbook to create an Incident channel, then the FireHydrant Slack integration acts as your scribe.

Everything that happens in the incident, from actions taken by users, to any chatter and attachments pasted into the channel, are all tracked in the incident timeline.

Contextually, when you are in an incident channel, the list of available commands expands, giving you capability to manage an incident end-to-end from within the channel.

We generally recommend handling as many automations as possible with Runbooks. However, you have the capability to take actions manually within the incident channel and executing commands or using the contextual action buttons.

Assigning Incident roles and teams

You can configure as many incident roles in FireHydrant as you'd like. Incident roles are valuable for quickly delegating responsibilities to responders during an incident.

To assign a role during an incident, simply run:

/firehydrant assign role

Alternatively, you can assign a team, which will pull in all people and assign default roles (if configured), or pull in whoever is on-call from a schedule. When assigning a team, you will see all the individuals in the team and what role they will be assigned, if relevant.

/firehydrant assign team

When you assign roles to users or pull in teams, they’ll receive a direct message in Slack (assuming their FireHydrant account is linked with Slack) saying they’ve been assigned the role.

Adding tasks and follow-ups

You can add tasks ad-hoc during an incident and also assign them to users. To do this, run:

/firehydrant add task

Alternatively, if you've predefined task lists, then you can also assign the list to specific users as well if they haven't already been added to the incident:

/firehydrant add task-list

Once tasks have been created or added to the incident, you can then view them and manage them by running:

/firehydrant tasks [@slack_handle | unassigned | all]

If you run only /fh tasks without any arguments, this will show you any tasks assigned to you. Alternatively, providing additional arguments for @slack_handle will show you tasks assigned to a specific user, unassigned for unassigned tasks, or all for the list of all available tasks regardless of assignment.

Additionally, you can quickly assign and change the state of tasks from this modal as well.

FireHydrant differentiates between Tasks, which are during-incident tasks, and Follow-Ups, which are post-incident items that were identified as important follow-up work but not key to resolving the incident itself. You can do this by running:

/firehydrant add follow-up

Follow-ups allow you some additional options like setting the priority and project, which links to external ticketing projects you've configured (e.g. Jira).

Currently, if you want to view the list of Follow-Ups from Slack, you will need to run:

/firehydrant action-items

This is the result of a historical decision to categorize both Tasks and Follow-ups as "action items," but Action Items will eventually be deprecated in favor of explicit distinction between the two types of items.

Note: Running /fh action-items will show you both Tasks and Follow-ups, but Follow-ups linked with tickets in external projects like Jira will have URLs pointing to them whereas Tasks will not.

Updating the incident

During the incident, you'll often want to make various updates. Different commands will allow you to change different things about the incident.

Update incident details

This includes things like the Name, Description, Customer Impact, Impacted Infrastructure, Severity, and Priority.

/firehydrant edit

Update incident impact

We have a command specifically for updating the Milestone, Impacted Infrastructure, and adding a note/update to the incident.

/firehydrant update

If status page(s) is/are attached, then you will also see checkboxes here for propagating messages out to these status pages, allowing you to update any external users/customers in one fluid motion.

Post an incident note

We also have commands to specifically post an incident note but without any other changes.

/firehydrant add note
/firehydrant post

Editing and Deleting messages

When messages are edited or deleted in Slack, those changes will be reflected in the FireHydrant incident timeline as well. Once a retrospective for an incident has been completed, this functionality is disabled.

Resolving incidents

Once you’ve fixed that pesky incident, mark it as resolved in the incident channel by running:

/firehydrant resolve

This automatically marks the incident as resolved.