Send PagerDuty alerts with Sensu
Sensu checks are commands the Sensu agent executes that generate observability data in a status or metric event. Sensu handlers define the actions the Sensu backend executes on the events. Follow this guide to create a check that looks for a specific file and a handler that sends an alert to PagerDuty if the file is not found.
One quick note before you begin: you’ll need your PagerDuty API integration key to set up the handler in this guide.
Install and configure Sensu Go
Follow the RHEL/CentOS install instructions to install and configure the Sensu backend, the Sensu agent, and sensuctl.
Find your entity name:
sensuctl entity list
The ID
in the response is the name of your entity.
Replace ENTITY_NAME
with the name of your entity in the sensuctl command below.
Then run the command to add the system
subscription to your entity:
sensuctl entity update ENTITY_NAME
- For
Entity Class
, press enter. - For
Subscriptions
, typesystem
and press enter.
Confirm both Sensu services are running:
systemctl status sensu-backend && systemctl status sensu-agent
Add the file_exists check
Before Sensu can send alerts to your PagerDuty account, you’ll need a check to generate events.
In this guide, you will use a file_exists
check, which is included in Sensu’s Nagios Foundation dynamic runtime asset.
Before you can add the check, you need to add the Nagios Foundation asset to your Sensu configuration:
sensuctl asset add ncr-devops-platform/nagiosfoundation
To confirm that the asset was added to your Sensu backend, run:
sensuctl asset info ncr-devops-platform/nagiosfoundation
You should see === ncr-devops-platform/nagiosfoundation
followed by a list of available builds for the asset.
Now that you’ve added the Nagios Foundation dynamic runtime asset, you can add its file_exists
check to your Sensu backend.
Use sensuctl to add the check:
sensuctl check create file_exists \
--command "check_file_exists --pattern /tmp/my-file.txt" \
--subscriptions system \
--handlers pagerduty \
--interval 10 \
--runtime-assets ncr-devops-platform/nagiosfoundation
To confirm that the check was added to your Sensu backend and view the check definition in YAML or JSON format, run:
sensuctl check info file_exists --format yaml
sensuctl check info file_exists --format wrapped-json
The response will list the complete check resource definition:
---
type: CheckConfig
api_version: core/v2
metadata:
created_by: admin
name: file_exists
namespace: default
spec:
check_hooks: null
command: check_file_exists --pattern /tmp/my-file.txt
env_vars: null
handlers:
- pagerduty
high_flap_threshold: 0
interval: 10
low_flap_threshold: 0
output_metric_format: ""
output_metric_handlers: null
proxy_entity_name: ""
publish: true
round_robin: false
runtime_assets:
- ncr-devops-platform/nagiosfoundation
secrets: null
stdin: false
subdue: null
subscriptions:
- system
timeout: 0
ttl: 0
{
"type": "CheckConfig",
"api_version": "core/v2",
"metadata": {
"created_by": "admin",
"name": "file_exists",
"namespace": "default"
},
"spec": {
"check_hooks": null,
"command": "check_file_exists --pattern /tmp/my-file.txt",
"env_vars": null,
"handlers": [
"pagerduty"
],
"high_flap_threshold": 0,
"interval": 10,
"low_flap_threshold": 0,
"output_metric_format": "",
"output_metric_handlers": null,
"proxy_entity_name": "",
"publish": true,
"round_robin": false,
"runtime_assets": [
"ncr-devops-platform/nagiosfoundation"
],
"secrets": null,
"stdin": false,
"subdue": null,
"subscriptions": [
"system"
],
"timeout": 0,
"ttl": 0
}
}
The check command includes the path for the file that the check will look for on your system, /tmp/my-file.txt
.
For this guide, you’ll add /tmp/my-file.txt
as a temporary file:
touch /tmp/my-file.txt
In production, you could replace /tmp/my-file.txt
with the file path or globbing pattern for any existing file or files on your system.
Well done! In the next step, you’ll configure your workflow for sending Sensu alerts to your PagerDuty account.
Add the PagerDuty handler
The Sensu PagerDuty Handler dynamic runtime asset includes the scripts you will need to send events to PagerDuty.
To add the PagerDuty handler asset, run:
sensuctl asset add sensu/sensu-pagerduty-handler
Once again, it’s a good idea to confirm that the asset was added to your Sensu backend. Run:
sensuctl asset info sensu/sensu-pagerduty-handler
The response will list the available builds for the PagerDuty handler dynamic runtime asset.
Now that you’ve added the Sensu PagerDuty Handler dynamic runtime asset, you can create a handler that uses the asset to send non-OK events to PagerDuty. This requires you to update the handler command by adding your PagerDuty API integration key.
In the following command, replace YOUR_PAGERDUTY_KEY
with your PagerDuty API integration key.
Then run the updated command:
sensuctl handler create pagerduty \
--type pipe \
--filters is_incident \
--runtime-assets sensu/sensu-pagerduty-handler \
--command "sensu-pagerduty-handler -t YOUR_PAGERDUTY_KEY"
NOTE: For checks whose handlers use the Sensu PagerDuty Handler dynamic runtime asset (like the one you’ve created in this guide), you can use an alternative method for authenticating and routing alerts based on PagerDuty teams.
To use this option, list the teams’ PagerDuty API integration keys in the handler definition as environment variables or secrets or in the /etc/default/sensu-backend
configuration file as environment variables.
Then, add check or agent annotations to specify which PagerDuty teams should receive alerts based on check events.
Sensu will look up the key in the handler definition or backend configuration file that corresponds to the team name in the check annotation to authenticate and send alerts.
Make sure that your handler was added by retrieving the complete handler definition in YAML or JSON format:
sensuctl handler info pagerduty --format yaml
sensuctl handler info pagerduty --format wrapped-json
The response will list the complete check resource definition:
---
type: Handler
api_version: core/v2
metadata:
created_by: admin
name: pagerduty
namespace: default
spec:
command: sensu-pagerduty-handler -t YOUR_PAGERDUTY_KEY
env_vars: null
filters:
- is_incident
handlers: null
runtime_assets:
- sensu/sensu-pagerduty-handler
secrets: null
timeout: 0
type: pipe
{
"type": "Handler",
"api_version": "core/v2",
"metadata": {
"created_by": "admin",
"name": "pagerduty",
"namespace": "default"
},
"spec": {
"command": "sensu-pagerduty-handler -t YOUR_PAGERDUTY_TOKEN",
"env_vars": null,
"filters": [
"is_incident"
],
"handlers": null,
"runtime_assets": [
"sensu/sensu-pagerduty-handler"
],
"secrets": null,
"timeout": 0,
"type": "pipe"
}
}
Trigger an event
With your file_exists
check and PagerDuty handler workflow in place, you can remove a file to cause Sensu to send a non-OK event.
Remove the file /tmp/my-file.txt
:
rm /tmp/my-file.txt
This will make sure the file is not there for Sensu to find the next time the file_exists
check runs.
After about 10 seconds, Sensu will detect that my-file.txt
is missing and reflect that in an event.
To view the event with sensuctl, run:
sensuctl event list
The response should show that the file removal resulted in a CRITICAL (2) event:
Entity Check Output Status Silenced Timestamp UUID
────────────── ───────────── ──────────────────────────────────────────────────────────────────────── ──────── ────────── ─────────────────────────────── ──────────────────────────────────────
host01 file_exists CheckFileExists CRITICAL - 0 files matched pattern /tmp/my-file.txt 2 false 2021-03-15 19:28:21 +0000 UTC 1b4266ae-7200-4728-a0n4-2f50f7a56613
Open the Sensu web UI to see the events the file_exists
check is generating.
Visit http://127.0.0.1:3000, and log in as the admin user (created during initialization when you installed the Sensu backend).
The failing check’s events will be listed on the Events page.
Observe the alert in PagerDuty
After Sensu detects the non-OK event, the handler you set up will send the alert to PagerDuty. Log in to your PagerDuty account to see an event similar to this one:

Resolve the alert in PagerDuty
To complete your workflow, restore the file that you removed so Sensu sends a resolution to PagerDuty:
touch /tmp/my-file.txt
In your PagerDuty account, the alert should now be listed under the “Resolved” tab.
To view the resolved event with sensuctl, run:
sensuctl event list
The response should show that the file is restored, with an OK (0) status:
Entity Check Output Status Silenced Timestamp UUID
────────────── ───────────── ──────────────────────────────────────────────────────────────────────── ──────── ────────── ─────────────────────────────── ──────────────────────────────────────
host01 file_exists CheckFileExists OK - 1 files matched pattern /tmp/my-file.txt 0 false 2021-03-15 19:58:31 +0000 UTC 73f382fe-4290-48e8-955a-6efd54347b43
The web UI Events page will also show that the file_check
check is passing.
Next steps
You should now have a working set-up with a file_exists
check and a handler that sends alerts to your PagerDuty account.
To share and reuse the check and handler like code, save them to files and start building a monitoring as code repository.
You can customize your PagerDuty handler with configuration options like severity mapping, PagerDuty team-based routing and authentication via check and agent annotations, and event-based templating. Learn more about the Sensu PagerDuty integration and our curated, configurable quick-start template for incident management to integrate Sensu with your existing PagerDuty workflows.