Send PagerDuty alerts with Sensu

Sensu checks are commands the Sensu agent executes that generate observability data in a status or metric event. Sensu handlers define the actions the Sensu backend executes on the events. Follow this guide to create a check that looks for a specific file and a handler that sends an alert to PagerDuty if the file is not found.

One quick note before you begin: you’ll need your PagerDuty API integration key to set up the handler in this guide.

Install and configure Sensu Go

Follow the RHEL/CentOS install instructions to install and configure the Sensu backend, the Sensu agent, and sensuctl.

Find your entity name:

sensuctl entity list

The ID in the response is the name of your entity.

Replace ENTITY_NAME with the name of your entity in the sensuctl command below. Then run the command to add the system subscription to your entity:

sensuctl entity update ENTITY_NAME

• For Entity Class, press enter.
• For Subscriptions, type system and press enter.

Confirm both Sensu services are running:

systemctl status sensu-backend && systemctl status sensu-agent

Add the file_exists check

Before Sensu can send alerts to your PagerDuty account, you’ll need a check to generate events. In this guide, you will use a file_exists check, which is included in Sensu’s Nagios Foundation dynamic runtime asset.

Before you can add the check, you need to add the Nagios Foundation asset to your Sensu configuration:

sensuctl asset add ncr-devops-platform/nagiosfoundation

To confirm that the asset was added to your Sensu backend, run:

sensuctl asset info ncr-devops-platform/nagiosfoundation

You should see === ncr-devops-platform/nagiosfoundation followed by a list of available builds for the asset.

Now that you’ve added the Nagios Foundation dynamic runtime asset, you can add its file_exists check to your Sensu backend. Use sensuctl to add the check:

sensuctl check create file_exists \
--command "check_file_exists --pattern /tmp/my-file.txt" \
--subscriptions system \
--handlers pagerduty \
--interval 10 \
--runtime-assets ncr-devops-platform/nagiosfoundation

To confirm that the check was added to your Sensu backend and view the check definition in YAML or JSON format, run:

sensuctl check info file_exists --format yaml

sensuctl check info file_exists --format wrapped-json

The response will list the complete check resource definition:

---
type: CheckConfig
api_version: core/v2
metadata:
  created_by: admin
  name: file_exists
  namespace: default
spec:
  check_hooks: null
  command: check_file_exists --pattern /tmp/my-file.txt
  env_vars: null
  handlers:
  - pagerduty
  high_flap_threshold: 0
  interval: 10
  low_flap_threshold: 0
  output_metric_format: ""
  output_metric_handlers: null
  proxy_entity_name: ""
  publish: true
  round_robin: false
  runtime_assets:
  - ncr-devops-platform/nagiosfoundation
  secrets: null
  stdin: false
  subdue: null
  subscriptions:
  - system
  timeout: 0
  ttl: 0

{
  "type": "CheckConfig",
  "api_version": "core/v2",
  "metadata": {
    "created_by": "admin",
    "name": "file_exists",
    "namespace": "default"
  },
  "spec": {
    "check_hooks": null,
    "command": "check_file_exists --pattern /tmp/my-file.txt",
    "env_vars": null,
    "handlers": [
      "pagerduty"
    ],
    "high_flap_threshold": 0,
    "interval": 10,
    "low_flap_threshold": 0,
    "output_metric_format": "",
    "output_metric_handlers": null,
    "proxy_entity_name": "",
    "publish": true,
    "round_robin": false,
    "runtime_assets": [
      "ncr-devops-platform/nagiosfoundation"
    ],
    "secrets": null,
    "stdin": false,
    "subdue": null,
    "subscriptions": [
      "system"
    ],
    "timeout": 0,
    "ttl": 0
  }
}

The check command includes the path for the file that the check will look for on your system, /tmp/my-file.txt. For this guide, you’ll add /tmp/my-file.txt as a temporary file:

touch /tmp/my-file.txt

In production, you could replace /tmp/my-file.txt with the file path or globbing pattern for any existing file or files on your system.

Well done! In the next step, you’ll configure your workflow for sending Sensu alerts to your PagerDuty account.

Add the PagerDuty handler

The Sensu PagerDuty Handler dynamic runtime asset includes the scripts you will need to send events to PagerDuty.

To add the PagerDuty handler asset, run:

sensuctl asset add sensu/sensu-pagerduty-handler

Once again, it’s a good idea to confirm that the asset was added to your Sensu backend. Run:

sensuctl asset info sensu/sensu-pagerduty-handler

The response will list the available builds for the PagerDuty handler dynamic runtime asset.

Now that you’ve added the Sensu PagerDuty Handler dynamic runtime asset, you can create a handler that uses the asset to send non-OK events to PagerDuty. This requires you to update the handler command by adding your PagerDuty API integration key.

In the following command, replace YOUR_PAGERDUTY_KEY with your PagerDuty API integration key. Then run the updated command:

sensuctl handler create pagerduty \
--type pipe \
--filters is_incident \
--runtime-assets sensu/sensu-pagerduty-handler \
--command "sensu-pagerduty-handler -t YOUR_PAGERDUTY_KEY"

Make sure that your handler was added by retrieving the complete handler definition in YAML or JSON format:

sensuctl handler info pagerduty --format yaml

sensuctl handler info pagerduty --format wrapped-json

The response will list the complete check resource definition:

---
type: Handler
api_version: core/v2
metadata:
  created_by: admin
  name: pagerduty
  namespace: default
spec:
  command: sensu-pagerduty-handler -t YOUR_PAGERDUTY_KEY
  env_vars: null
  filters:
  - is_incident
  handlers: null
  runtime_assets:
  - sensu/sensu-pagerduty-handler
  secrets: null
  timeout: 0
  type: pipe

{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "created_by": "admin",
    "name": "pagerduty",
    "namespace": "default"
  },
  "spec": {
    "command": "sensu-pagerduty-handler -t YOUR_PAGERDUTY_TOKEN",
    "env_vars": null,
    "filters": [
      "is_incident"
    ],
    "handlers": null,
    "runtime_assets": [
      "sensu/sensu-pagerduty-handler"
    ],
    "secrets": null,
    "timeout": 0,
    "type": "pipe"
  }
}

Trigger an event

With your file_exists check and PagerDuty handler workflow in place, you can remove a file to cause Sensu to send a non-OK event.

Remove the file /tmp/my-file.txt:

rm /tmp/my-file.txt

This will make sure the file is not there for Sensu to find the next time the file_exists check runs. After about 10 seconds, Sensu will detect that my-file.txt is missing and reflect that in an event.

To view the event with sensuctl, run:

sensuctl event list

The response should show that the file removal resulted in a CRITICAL (2) event:

     Entity         Check                                      Output                                   Status   Silenced             Timestamp                             UUID                  
 ────────────── ───────────── ──────────────────────────────────────────────────────────────────────── ──────── ────────── ─────────────────────────────── ────────────────────────────────────── 
  host01         file_exists   CheckFileExists CRITICAL - 0 files matched pattern /tmp/my-file.txt           2   false      2021-03-15 19:28:21 +0000 UTC   1b4266ae-7200-4728-a0n4-2f50f7a56613

Open the Sensu web UI to see the events the file_exists check is generating. Visit http://127.0.0.1:3000, and log in as the admin user (created during initialization when you installed the Sensu backend). The failing check’s events will be listed on the Events page.

Observe the alert in PagerDuty

After Sensu detects the non-OK event, the handler you set up will send the alert to PagerDuty. Log in to your PagerDuty account to see an event similar to this one:

Resolve the alert in PagerDuty

To complete your workflow, restore the file that you removed so Sensu sends a resolution to PagerDuty:

touch /tmp/my-file.txt

In your PagerDuty account, the alert should now be listed under the “Resolved” tab.

To view the resolved event with sensuctl, run:

sensuctl event list

The response should show that the file is restored, with an OK (0) status:

     Entity         Check                                      Output                                   Status   Silenced             Timestamp                             UUID                  
 ────────────── ───────────── ──────────────────────────────────────────────────────────────────────── ──────── ────────── ─────────────────────────────── ────────────────────────────────────── 
  host01         file_exists   CheckFileExists OK - 1 files matched pattern /tmp/my-file.txt                 0   false      2021-03-15 19:58:31 +0000 UTC   73f382fe-4290-48e8-955a-6efd54347b43

The web UI Events page will also show that the file_check check is passing.

Next steps

You should now have a working set-up with a file_exists check and a handler that sends alerts to your PagerDuty account. To share and reuse the check and handler like code, save them to files and start building a monitoring as code repository.

You can customize your PagerDuty handler with configuration options like severity mapping, PagerDuty team-based routing and authentication via check and agent annotations, and event-based templating. Learn more about the Sensu PagerDuty integration and our curated, configurable quick-start template for incident management to integrate Sensu with your existing PagerDuty workflows.