Search Results

Start for free

Runbook recipes

The following are recommended runbook architectures for the categories outlined in Runbook Best Practices:

In the runbook recipes steps are formatted as Step Category > Step Description.  Unless otherwise noted, the recommended default step execution rules are: Always execute and Execute automatically.  You may wish to modify this according to your business rules.

Note: You will need to have the desired integration configured before you can add an integration-driven runbook step.  Runbook conditions are available to paid FireHydrant plans only. If you're on a free plan, you can specify to execute those steps manually or choose not to include them.

Default Runbook

Execution rule:  _ Always attach _

SEV-1 (or equivalent) Incident Runbook

Execution rule:  Execute when Current Severity is SEV-1 (or equivalent) 

  • Remind responders to provide high frequency internal communication updates. For example:

  • Assign the appropriate incident roles.

  • Publish incident status updates to the internal status page.

    • Step: FireHydrant Status Page > Publish to status page  
      • Conditions & Scheduling: Automatically execute the step when the Current Milestone is [insert key milestones] (e.g. Acknowledged, Investigating, Mitigated, etc.)

  • Publish the incident and send updates to the external status page. (e.g. Statuspage.io)

  • Attach a task list for each critical assigned role.

  • Notify other relevant Slack channels with periodic incident status updates.

    • Prerequisite: The FireHydrant app must be added to the Slack channels for FireHydrant to post your message. Please make sure to /invite @FireHydrant to any channels you list in the configuration.
    • Step: Slack > Notify channel with a custom message
      • Add relevant team channels (e.g. #support)
      • Conditions & Scheduling
        • Current milestone is one of: Started, Detected, Acknowledged, Investigating, Identified, Mitigated
        • Execute automatically every 30 minutes

  •  

SEV-2 (or equivalent) Incident Runbook

Execution rule:  Execute when Current Severity is SEV-2 (or equivalent) 

  • Remind responders to provide medium frequency internal communication updates. For example:

  • Publish incident status updates to the internal status page.

    • Step: FireHydrant Status Page > Publish to status page
      • Conditions and Scheduling: Automatically execute the step when the Current Milestone is [insert key milestones] (e.g. Acknowledged, Investigating, Mitigated, etc.)

  • Attach a general task list for the incident.

  • Notify other relevant Slack channels with periodic incident status updates until the incident is resolved.

    • Prerequisite: The FireHydrant app must be added to the Slack channels for FireHydrant to post your message. Please make sure to/invite @FireHydrantto any channels you list in the configuration.
    • Step: Slack > Notify channel with a custom message
      • Add relevant team channels (e.g. #support)
      • Condition and scheduling:
        • E** xecute automatically**
        • Repeat  every 60 minutes
        • When the C_urrent milestone is one of: Started, Detected, Acknowledged, Investigating, Identified, Mitigated_

  •  

Advanced Runbooks

Service-based Runbook

  • Add a team
    • Step: FireHydrant > Assign A Team
  • Page out to an alerting provider
    • Prerequisite: Alerting provider configured
    • As appropriate for your provider setup:
      • Step: Opsgenie > Create OpsGenie Incident
      • Step: PagerDuty > Create PagerDuty Incident
      • Step: VictorOps > Create VictorOps Incident
  • Notify incident channel of available documentation
  • Tasklist with specific steps to be taken to restore the service based on condition
  • Run a custom script
    • Step: FireHydrant > Script
      • Add your description and custom script to the runbook step configuration.

Team-based Runbook

  • Notify a team’s home channel of engagement

    • Step: Slack > Notify channel
      • Comma Separated Channels: [#team-channel]

  • Add a Role

    • Step: FireHydrant > Assign a Role
      • Add the role and user to the runbook step configuration.

  • Email Manager

    • Firehydrant > Send an email notification
      • Add the manager's email address, subject, and message to the runbook step configuration.

  • Reminder to team’s home channel for long-running incidents

    • Step: Slack > Notify incident channel with a custom message
      • Your Message: [Friendly reminder to post an update on the incident status page(s)]
      • Action button: Post an update
      • Comma Separated Channels: [#team-channel]
    • Conditions and Scheduling:
      • E** xecute automatically**
      • Repeat every 60 minutes
      • When the Time since incident opened: greater than 5 hours

  • Tasklist

Customer Facing Issue Runbook

  • Let the customer support team know an incident has been opened.
  • Send reminders to update customer-facing information while the incident is open.
    • Slack > Notify incident channel with a custom message
      • Your Message: [Friendly reminder to post an update to the external status page.]
      • Action button: Post an update
      • Condition and scheduling:
        • Execute automatically
        • Repeat every 60 minutes
        • When the Current milestone is one of: Started, Detected, Acknowledged, Investigating, Identified, Mitigated ted** ed** ted** g, Identified, Mitigated** ted**

** ted** ted**

Last updated on 3/28/2023