AWS CloudWatch

This AWS CloudWatch integration guide for Signals will walk you through how to send webhook events from AWS CloudWatch to Signals. We'll use CloudWatch alarms, Simple Notificaiton Service (SNS), and a Lambda function to customize and send the webhook to FireHydrant. Anytime CloudWatch triggers an alarm and it is sent as an event to FireHydrant, we’ll evaluate the event payload to see if it matches a rule one of your teams set up. If a rule matches, we’ll alert the team. Learn more about Alert Rules here.

🚧

IAM Roles and Permissions

This guide assumes that you have appropriate roles and permissions between the services mentioned.

Configuring SNS Topic

1. In you AWS management console, navigate to the Simple Notification Service.
2. Create a topic (make sure that the type is set to Standard). This topic is where you will send notifications from your cloud watch alarm. If you already have a topic configured for your CloudWatch alarms, you can reuse that topic and subscribe your Lambda function to that from the next step.

Configuring Lambda to Send Webhook

1. In your AWS management console, navigate to the Lambda dahsboard.
2. Create a new Lambda function. The example here assume using Node.js, but you can send the webhook using any language available in Lambda.
3. Use the following as a starting point for sending an event to Signals. To get your webhook URL, log in to FireHydrant and navigate to the Signals Webhooks page. Copy the URL for the Generic Webhook and use that in the function below for webhookUrl.
4. After deploying your Lambda function, use the Diagram view in the Function Overview to add a new Trigger. Select SNS as the source, and then select the topic that you created in Configuring SNS topic. Now your Lambda will trigger anytime an event is sent to that topic.

//index.mjs
export const handler = async (event) => {
  const webhookUrl = "https://signals.firehydrant.com/v1/process/{your_unique_key}";
  const message = JSON.parse(event.Records[0].Sns.Message);
  return fetch(webhookUrl, {
    method: "POST", 
    body: JSON.stringify({
      summary: event.Records[0].Sns.Subject,
      body: `${message.NewStateValue} - ${message.NewStateReason} - ${message.AlarmName} - ${message.AlarmDescription} - ${message.Region}`,
      idempotency_key: message.AlarmArn,
      status: message.NewStateValue == "ALARM" ? 0 : 1, 
      tags: event.resources ? event.resources : []
    })
  });
};

To customize the body of your webhook event, use any of the data available from the SNS message. Read the Lambda docs to better understand what is available on the message object.

Configuring CloudWatch Alarm

1. In your AWS Management console, navigate to the CloudWatch dashboard.
2. Create a new Alarm, and configure any metrics for services that you'd like to track. On the "Configure Actions" page of the setup, choose an Alarm state trigger for "In alarm".
   1. To auto-resolve flappy alerts, add a second notification with "OK select for the Alarm state trigger. Use the same SNS topic since the Lambda function above is checking and setting the status in Signals.
3. Select the existing SNS topic that we created in the first section and complete the setup. You can add an Alarm name and description that you can use to send specific data to Signals in your Lambda.
4. Review the details of your Alarm and click "Create Alarm."

Testing CloudWatch Alarm

1. Either from your CLI or from the AWS cloud shell, run the following command: aws cloudwatch set-alarm-state --alarm-name "{YOUR ALARM NAME}" --state-reason "Testing the Amazon Cloudwatch alarm" --state-value ALARM.
2. In FireHydrant, navigate to Signals > Events Logs to see your new incoming event.
3. To test auto-resolving the alert, change the command above to use --state-value OK. You will need to have an Alert Rule configured to see Alerts created and resolved with this.