Send email alerts with the Sensu Go Email Handler

Sensu event handlers are actions the Sensu backend executes on events. This guide explains how to use the Sensu Go Email Handler dynamic runtime asset to send notification emails.

When you are using Sensu in production, events will come from a check or metric you configure. For this guide, you will create an ad hoc event that you can trigger manually to test your email handler.

To follow this guide, you’ll need to install the Sensu backend, have at least one Sensu agent running on Linux, and install and configure sensuctl.

Your backend will execute an email handler that sends notifications to the email address you specify. You’ll also add an event filter to make sure you only receive a notification when your event represents a status change.

Add the email handler dynamic runtime asset

Dynamic runtime assets are shareable, reusable packages that help you deploy Sensu plugins. In this guide, you’ll use the Sensu Go Email Handler dynamic runtime asset to power an email handler.

Use the following sensuctl example to register the Sensu Go Email Handler dynamic runtime asset:

sensuctl asset add sensu/sensu-email-handler -r email-handler
no version specified, using latest: 0.6.0
fetching bonsai asset: sensu/sensu-email-handler:0.6.0
added asset: sensu/sensu-email-handler:0.6.0

You have successfully added the Sensu asset resource, but the asset will not get downloaded until
it's invoked by another Sensu resource (ex. check). To add this runtime asset to the appropriate
resource, populate the "runtime_assets" field with ["email-handler"].

The -r (rename) flag allows you to specify a shorter name for the dynamic runtime asset (in this case, email-handler).

You can also download the latest dynamic runtime asset definition for your platform from Bonsai and register the asset with sensuctl create --file filename.yml.

To confirm that the handler dynamic runtime asset was added correctly, run:

sensuctl asset list

You should see the email-handler dynamic runtime asset in the list. For a detailed list of everything related to the asset that Sensu added automatically, run:

sensuctl asset info email-handler

The dynamic runtime asset includes the sensu-email-handler command, which you will use when you create the email handler definition later in this guide.

NOTE: Sensu does not download and install dynamic runtime asset builds onto the system until they are needed for command execution. Read the asset reference for more information about dynamic runtime asset builds.

Add an event filter

Event filters allow you to fine-tune how your events are handled and reduce alert fatigue. In this guide, your event filter will send notifications only when your event’s state changes (for example, for any change between 0 OK, 1 warning, and 2 critical).

Here’s an overview of how the state_change_only filter will work:

  • If your event status changes from 0 to 1, you will receive one email notification for the change to warning status.
  • If your event status stays at 1 for the next hour, you will not receive repeated email notifications during that hour.
  • If your event status changes to 2 after 1 hour at 1, you will receive one email notification for the change from warning to critical status.
  • If your event status fluctuates between 0, 1, and 2 for the next hour, you will receive one email notification each time the status changes.

To create the event filter, run:

cat << EOF | sensuctl create
---
type: EventFilter
api_version: core/v2
metadata:
  annotations: null
  labels: null
  name: state_change_only
  namespace: default
spec:
  action: allow
  expressions:
  - event.check.occurrences == 1
  runtime_assets: []
EOF

Create the email handler definition

After you add an event filter, create the email handler definition to specify the email address where the sensu/sensu-email-handler dynamic runtime asset will send notifications. In the handler definition’s command value, you’ll need to change a few things.

Copy this text into a text editor:

cat << EOF | sensuctl create
---
api_version: core/v2
type: Handler
metadata:
  namespace: default
  name: email
spec:
  type: pipe
  command: sensu-email-handler -f YOUR-SENDER@example.com -t YOUR-RECIPIENT@example.com -s YOUR-SMTP-SERVER.example.com
    -u USERNAME -p PASSWORD
  timeout: 10
  filters:
  - is_incident
  - not_silenced
  - state_change_only
  runtime_assets:
  - email-handler
EOF

Then, replace the following text:

  • YOUR-SENDER@example.com: Replace with the email address you want to use to send email alerts.
  • YOUR-RECIPIENT@example.com: Replace with the email address you want to receive email alerts.
  • YOUR-SMTP-SERVER.example.com: Replace with the hostname of your SMTP server.
  • USERNAME: Replace with your SMTP username, typically your email address.
  • PASSWORD: Replace with your SMTP password, typically the same as your email password.

NOTE: To use Gmail or G Suite as your SMTP server, follow Google’s instructions to send email via SMTP. If you have enabled 2-step verification on your Google account, use an app password instead of your login password. If you have not enabled 2-step verification, you may need to adjust your app access settings to follow the example in this guide.

You probably noticed that the handler definition includes two other filters besides state_change_only: is_incident and not_silenced. These two filters are included in every Sensu backend installation, so you don’t have to create them.

After you add your email, server, username, and password values, run your updated code to create the email handler definition.

Now your handler and event filter are set up!

The Sensu Go Email Handler dynamic runtime asset makes it possible to add a template that provides context for your email notifications. The email template functionality uses tokens to populate the values provided by the event, and you can use HTML to format the email.

Before your handler can send alerts to your email, you need an event that generates the alerts. In the final step, you will create an ad hoc event that you can trigger manually.

Create and trigger an ad hoc event

To create an ad hoc event, first use sensuctl env to set up environment variables. The environment variables will provide the required credentials for the Sensu API:

eval $(sensuctl env)

Verify that the SENSU_ACCESS_TOKEN environment variable is set by echoing its value:

echo $SENSU_ACCESS_TOKEN
efPxbRciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1NzkwMzY5NjQsImp0aSI6ImJiMmY0ODY4ZTJhZWEyMDhhMTExOTllMGZkNzkzMDc0Iiwic3ViIjoiYWRtaW4iLCJncm91cHMiOlsiY2x1c3Rlci1hZG1pbnMiLCJzeXN0ZW06dXNlcnMiXSwicHJvdmlkZXIiOnsicHJvdmlkZXJfaWQiOiJiYXNpYyIsInByb3ZpZGVyX3R5cGUiOiIiLCJ1c2VyX2lkIjoiYWRtaW4ifX0.6XmuvblCN743R2maF4yErS3K3sOVczsCBsjib9TenUU

With the environment variables set, you can use the Sensu API to create your ad hoc observability event. This event outputs the message “Everything is OK.” when it occurs:

curl -sS -H 'Content-Type: application/json' \
-H "Authorization: Bearer $SENSU_ACCESS_TOKEN" \
-d '{
  "entity": {
    "entity_class": "proxy",
    "metadata": {
      "name": "server01",
      "namespace": "default"
    }
  },
  "check": {
    "metadata": {
      "name": "server-health"
    },
    "output": "Everything is OK.",
    "status": 0,
    "interval": 60
  }
}' \
http://localhost:8080/api/core/v2/namespaces/default/events

As configured, the event status is 0 (OK). Now it’s time to trigger an event and see the results!

To generate a status change event, use the update event endpoint to create a 1 (warning) event. Run:

curl -sS -X PUT \
-H "Authorization: Bearer $SENSU_ACCESS_TOKEN" \
-H 'Content-Type: application/json' \
-d '{
  "entity": {
    "entity_class": "proxy",
    "metadata": {
      "name": "server01",
      "namespace": "default"
    }
  },
  "check": {
    "metadata": {
      "name": "server-health"
    },
    "output": "This is a warning.",
    "status": 1,
    "interval": 60,
    "handlers": ["email"]
  }
}' \
http://localhost:8080/api/core/v2/namespaces/default/events/server01/server-health

NOTE: If you see an invalid credentials error, refresh your token. Run eval $(sensuctl env).

Check your email — you should see a message from Sensu!

Create another event with status set to 0. Run:

curl -sS -X PUT \
-H "Authorization: Bearer $SENSU_ACCESS_TOKEN" \
-H 'Content-Type: application/json' \
-d '{
  "entity": {
    "entity_class": "proxy",
    "metadata": {
      "name": "server01",
      "namespace": "default"
    }
  },
  "check": {
    "metadata": {
      "name": "server-health"
    },
    "output": "Everything is OK.",
    "status": 0,
    "interval": 60,
    "handlers": ["email"]
  }
}' \
http://localhost:8080/api/core/v2/namespaces/default/events/server01/server-health

You should receive another email because the event status changed to 0 (OK).

Next steps

Now that you know how to apply a handler to a check and take action on events:

You can also follow our Up and running with Sensu Go interactive tutorial to set up the Sensu Go email handler and test a similar workflow with the addition of a Sensu agent for producing events using scheduled checks.