Integrating with Splunk On-Call (VictorOps)

With our Splunk On-Call integration, you can easily link your incidents in Splunk On-Call (formerly VictorOps) to your incidents in FireHydrant. This is a great way to coordinate your incident response, from right within FireHydrant, after an initial alert comes in from Splunk On-Call.

Installing the Splunk On-Call Integration

To get started:

1. Click Integrations in the FireHydrant left nav.
2. On the VictorOps integration tile, click Setup. (This tile is still labeled "VictorOps" in the FireHydrant UI, although the product was recently renamed "Splunk On-Call.")
3. Enter your Splunk On-Call (VictorOps) Organization name.
4. Enter your Splunk On-Call API ID and an API Key. You can find and generate these on the Splunk On-Call Integrations page under the API tab. If you don’t see them, reach out to your organization’s administrator.
5. Enter your REST Endpoint Token.
   • To find your REST Endpoint Token, go to Splunk On-Call's Integrations page and locate the REST integration. Under URL to notify , your routing key is included in the destination URL, between /alert/ and /$routing_key . If the REST endpoint integration has not been enabled, click the blue Enable button to generate your endpoint destination URL. (For details, see the Splunk On-Call documentation.)
6. Click Update Configuration. FireHydrant creates a new installed integration for VictorOps on your Connected & Available integrations list.

Using Alert Routes with Splunk On-Call (VictorOps)

Once your Splunk On-Call (VictorOps) instance is configured, you can setup Alert Routes to take action on your alerts based on the data included in the alert. You can automatically open new incidents, send alerts to any Slack channel, log an alert in FireHydrant, or simply ignore it. To learn more, read about Alert Routes.

Synching an incident state from Splunk On-Call

After you have configuring your integration, you need to set up an outgoing Splunk On-Call (VictorOps) webhook, so FireHydrant can synchronize incident states from Splunk On-Call.

1. Return to the configuration page by clicking the VictorOps tile on the Integrations page. This opens your VictorOps configuration details page, where a webhook URL has been generated for you.
2. Copy the webhook address provided.
3. Go to your VictorOps account at https://portal.victorops.com.
4. Click Integrations > Outgoing Webhooks.
5. Click Add Webhook and enter in the URL provided in FireHydrant. Leave the payload empty. (Splunk On-Call sends a default payload to FireHydrant.)
6. Select the Any-Incident value from the event dropdown.
7. Click Save on the new webhook.

Note: Splunk On-Call Outgoing Webhooks are an Enterprise service level feature. To view or modify them, you must have administrative credentials.

Importing and linking routing keys to new or existing services

With Splunk On-Call (VictorOps), you submit an alert to a Routing Key. That Routing Key dictates one or more escalation policies that will be alerted. Using Routing Keys is the preferred way to create and route alerts to the right team or escalation policy in Splunk On-Call. FireHydrant allows you to import Routing Keys and link them to specific services so that the proper teams are paged whenever services are impacted by an incident. If services you have linked to a Routing Key are impacted during an incident, FireHydrant directs an alert to the linked Routing Key—which in turn alerts the team members associated with its targeted escalation policy.
Refer to Splunk On-Call's documentation for details on managing alert Routing Keys.

To configure the link and import process:

1. In the FireHydrant left nav, go to Service Catalog > Services. Click Link and import.
2. From the Select an import source dropdown menu, select VictorOps.
   
3. Select the routing keys you wish to import and link them to existing services (or create new services).

Creating a Splunk On-Call incident from a Runbook step

To create a Runbook step to kick off a Splunk On-Call incident:

1. In the FireHydrant left nav, go to Runbooks.
2. Find and click the name of the Runbook where you want to add the Splunk On-Call step.
3. In the Available Steps section, scroll down to the VictorOps step and click Add.
   Under the Details tab in this section, the first two fields display the templated text that will be generated with the incident.
   
4. The third field in this section allows you to specify an Additional Routing Key.
   When you create an incident with an impacted service, FireHydrant will page routing keys linked to all impacted services by default. The Additional Routing Key field lets you specify another routing key that should always be paged, even if it is not linked to an impacted service.
5. As described in the section above, VictorOps lets you submit an alert to a specific Routing Key. Once alerted, the Routing Key will route the alert to designated escalation policies.
   The last field in the Runbook step configuration section asks if you want to alert a default escalation policy when no impacted services are present and no additional Routing Key is specified. Selecting Yes will cause FireHydrant to submit an alert to VictorOps without a routing key. That alert is then directed to whatever escalation policy is set as the default in VictorOps.