Use Alert Routing to Make Sense of your Alerts

Alert routes allow you to create conditional rules for alerts that are being sent to FireHydrant from providers like PagerDuty, OpsGenie, and VictorOps. These rules allow you to automatically open incidents, send notifications to Slack channels, log alerts in FireHydrant, or simply ignore an alert. You can use powerful conditional statements that take advantage of the data from your alerting provider.

Adding Alert Routes

To get started with alert routes, navigate to the integration page in FireHydrant for your alerting provider. Alongside the configuration of your alerting provider, you’ll find a tab for Alert Routes as well as Alert Logs.

Image of Tabs

In the Alert Routes tab, you’ll find a default route that sends all your alerts to the channel that was configured when you set up your Slack integration (this channel can be referenced via a liquid variable: ``). Default routes are the fallback rule that will be executed if none of the other routes have been run.

Image of default rule

Conditions and Actions

Adding a new Alert Route requires you to add a Condition and an Action. A condition allows you to set the logic that controls whether or not an alert route is run. Once the conditions on a route have been matched, FireHydrant will stop evaluating the other rules and we will perform the action associated with the route. The conditions can be configured from any of the data that has come through from the alerting provider.

There are some common fields from all providers such as Summary, Priority, and Impacted Infrastructure (your services, functionalities and environments in the Service Catalog), but any additional fields from your provider can also be pulled in from the webhook’s request body. Conditionals can be chained together with either an “OR” or “AND” condition for the whole route.

Image of Conditional

For each route, you can choose between four actions:

Automatically declare an Incident in FireHydrant
Send an alert to a specific Slack Channel
Create a log for this integration
Ignore the alert.

Array of Images from the Actions Template

When automatically opening an incident, you can specify the content of various fields on the incident either with regular text (and in some cases Markdown) or with liquid templates, which references the data available from the alert. When sending alerts to a Slack Channel, you can create a custom template for the Alert title that is sent and we’ll include a link to the original alert in the message. You can specify a channel by name (#backend-team) or with the liquid variable referencing the slack integration or the alert if it has identifiable information in it ({{ slack_connection.alert_channel }} or #{{alert.team_slack_channel}}). Here's an example alert being sent to a Slack channel.

When logging an alert to FireHydrant, you can select the level of the log you’d like to provider (Info, Warning, Error, etc.) and you can also specify the message that is included in the log. You can access any logged alerts in the Alert Logs tab of your integration.

Alert Logs

To see the logs from any alerts that have been routed to a log, navigate to the Alert Logs tab in your integration page in FireHydrant.

Image of Alert Logs UI

Here, you browse a list of alerts which is initially filtered to “Warn” level and above. Select another log level to see all alerts at that log level and above. To view any further details about the log, click “View Context”. You may find additional details including error messages and more.

Image of a Log Context pane