Runbook step: Creating a Splunk On-Call (VictorOps) Incident

In addition to routing alerts from Splunk On-Call and taking action on them in FireHydrant, our platform also allows creating Splunk On-Call incidents from FireHydrant incidents automatically via Runbooks.

Configuration

1. In the FireHydrant left nav, go to Runbooks.
2. Find and click the name of the Runbook where you want to add the Splunk On-Call step.
3. Click on Edit runbook and then scroll down and click on '+ Add Step'.
4. On the left, search for 'victorops' and click on Create VictorOps Incident.

5. Fill in the details for this Runbook step.
   • The first box is the name of the Incident. Liquid templating is supported.
   • The second box will be the description, or more specifically the state_message of the Incident in Splunk On-Call. Liquid templating is supported.
   • Additional Routing Key:
     • When you create an incident with an impacted service, FireHydrant will page routing keys linked to all impacted services by default. This Additional Routing Key field lets you specify another routing key that should always be paged, even if it is not linked to an impacted service.
   • Alert default escalation policy?:
     • Specifies if you want to alert a default escalation policy when no impacted services are present and no additional Routing Key is specified in the box above. Selecting Yes will cause FireHydrant to submit an alert to VictorOps without a routing key. That alert is then directed to whatever escalation policy is set as the default in VictorOps.