Integrating with PagerDuty

Our PagerDuty integration enables various functions to improve the efficiency of your incident management. You can:

• Automatically create FireHydrant incidents from PagerDuty alerts
• Notify different channels based on incoming alert content
• Page out to teams and service in PagerDuty from FireHydrant
• Automatically pull in on-call responders from PagerDuty's escalation policies to incident channels

Installation

First, you need a PagerDuty API key.

1. Go to your PagerDuty dashboard and navigate to the page where you can set up API keys. (This will be something like Integrations > API Access Keys ).

2. On the API Access page, click Create New API Key and provide a description for the API key.
3. Click Create Key.
4. Copy the API Key that PagerDuty generates. Then go to FireHydrant's integrations page.
5. Click the PagerDuty integration card, then click Setup PagerDuty.
6. Enter the PagerDuty API key that you copied into the API Token field.
7. For Domain, we default to showing https://api.pagerduty.com, which is the domain for PagerDuty's US service region. If you are a company based out of the EU and using PagerDuty's EU service region, then you will need to modify that value to https://api.eu.pagerduty.com. For more information, visit PagerDuty's documentation.
8. Click Authorize Application.

Setup Outgoing Webhook

Note: If you plan to link or import services from PagerDuty, do not configure this outgoing webhook and skip this section. When you link or import PagerDuty Services to FireHydrant services, we automatically create service-scoped hooks in PagerDuty.

After configuring the connection, you will lastly need to configure an outgoing webhook from PagerDuty so FireHydrant can both sync incident states as well as receive webhooks on alerts.

1. In the integration settings page for PagerDuty on FireHydrant, copy the webhook address provided.
2. In PagerDuty, Head to Integrations > Generic Webhooks and click "+ New Webhook."
3. In Webhook URL paste in the value you copied from FireHydrant.
4. For Scope Type choose "Account".
5. For Event Subscription, you can uncheck all the service.* events, leaving only the incident updates.
6. Click "Add Webhook".

To test this, go ahead and create an incident in FireHydrant. This will, with default rules in FireHydrant, log an alert processing message.

Linking PagerDuty Services to FireHydrant Services

After PagerDuty is installed, you also have the capability to link and/or import PagerDuty services into FireHydrant's services. This enables multiple capabilities, including:

• From Slack, directly paging out to teams or finding who's on call for a particular service
• Automatically paging owners of a particular service when it's marked impacted in a FireHydrant incident

How to link/import services

1. Head over to Service Catalog > Services and click the Add services dropdown and Import from third party.
2. Click the Import button next to PagerDuty.
3. From here, you can import all of your services, or you can load all of the available services in PagerDuty and selectively pick which ones you'd like to import or link.

4. Tick the checkbox next to services you want to import/link, and then in the dropdown box, you can specify two choices:
   1. Create New - Will create a new entry in the FireHydrant service catalog with the same name as the PagerDuty service. This will also automatically link the new FireHydrant service and the PagerDuty Service.
   2. Link to Existing - If you already have services in FireHydrant, you can choose which one you'd like to link this PagerDuty service to.

When you link services between FireHydrant and PagerDuty, FireHydrant automatically creates Webhooks in PagerDuty for each linked service.

Paging Users and Escalation Policies

When using the /fh page command in Slack, you can send pages to any user or esclation policy in your Pager Duty account. For more information, see Page & Lookup On-Call.

Paging Teams via Services

Via Slack, FireHydrant gives you the capability to directly page the teams that respond to a service through PagerDuty. To make use of this, there are two prerequisites:

1. You must complete the section/steps above to link PagerDuty services to FireHydrant services - FireHydrant relies on this link to find the correct on-call/escalation schedule for a particular service.
2. You must fill in the Default User Email field in the integration settings. This is because PagerDuty requires a user as part of each request's header to log who the request is on behalf of.

We recommend using a service account in PagerDuty that you create to represent FireHydrant as a whole.

Once you complete the above steps, you should be able to run /fh page service [NAME] within Slack, or pull up a list of services to page with just /fh page service.

For more information, see Page & Lookup On-Call.

Next Steps

Now that you've configured the PagerDuty integration, you can take the next steps to get the full power of FireHydrant + PagerDuty:

• Configure teams to pull in users from on-call schedules.
• Configuring Alert Routing to automatically open incidents or post notifications from inbound alerts.
• Automatically create PagerDuty incidents via Runbook step

PagerDuty Alert Routing

Once your PagerDuty instance is configured, you can setup Alert Routes to take action on your alerts based on the data included in the alert. You can automatically open new incidents, send alerts to any Slack channel, log an alert in FireHydrant, or simply ignore it. To learn more, read about Alert Routes.

Parameter Mappings

Here is the table of routable parameters on FireHydrant and the corresponding key/value from the inbound PagerDuty webhook(s). The $ refers to the webhook body content as a JSON object.

An explanation of PagerDuty's Webhook content can be found in their docs here.

The following table is a mapping of our overall Alert Routing mapping object - these parameters are common across all Alert Routing/Monitoring integrations.