Plan maintenance windows with silencing

As the Sensu backend processes check results, the server executes event handlers to send alerts to personnel or otherwise relay observation data (events) to external services. Sensu’s built-in silencing, along with the built-in not_silenced filter, provides a way to suppress execution of event handlers on an ad hoc basis.

Use silencing to prevent handlers configured with the not_silenced filter from being triggered based on the check name in a check result or the subscriptions associated with the entity that published the check result. Sensu’s silencing capability allows operators to quiet incoming alerts while coordinating a response or during planned maintenance windows.

Sensu silencing makes it possible to:

• Silence all checks on a specific entity
• Silence a specific check on a specific entity
• Silence all checks on entities with a specific subscription
• Silence a specific check on entities with a specific subscription
• Silence a specific check on every entity

Suppose you want to plan a maintenance window. In this example, you’ll create a silenced entry for a specific entity named i-424242 and its check, check-http, to prevent alerts as you restart and redeploy the services associated with this entity.

Create the silenced entry

To begin, create a silenced entry that will silence the check check-http on the entity i-424242 for a planned maintenance window that starts at 01:00 on Sunday and ends 1 hour later. Your username will be added automatically as the creator of the silenced entry:

sensuctl silenced create \
--subscription 'entity:i-424242' \
--check 'check-http' \
--begin '2021-03-14 01:00:00 -04:00' \
--expire 3600 \
--reason 'Server upgrade'

This command creates the following silenced resource definition:

type: Silenced
api_version: core/v2
metadata:
  created_by: admin
  name: entity:i-424242:check-http
  namespace: default
spec:
  begin: 1615698000
  check: check-http
  creator: admin
  expire: 3600
  expire_at: 1615701600
  expire_on_resolve: false
  reason: Server upgrade
  subscription: entity:i-424242

{
  "type": "Silenced",
  "api_version": "core/v2",
  "metadata": {
    "created_by": "admin",
    "name": "entity:i-424242:check-http",
    "namespace": "default"
  },
  "spec": {
    "begin": 1615698000,
    "check": "check-http",
    "creator": "admin",
    "expire": 3600,
    "expire_at": 1615701600,
    "expire_on_resolve": false,
    "reason": "Server upgrade",
    "subscription": "entity:i-424242"
  }
}

See the sensuctl documentation for the supported time formats for the begin flag.

Validate the silenced entry

Use sensuctl to verify that the silenced entry against the entity i-424242 was created properly:

sensuctl silenced info 'entity:i-424242:check-http' --format yaml

sensuctl silenced info 'entity:i-424242:check-http' --format wrapped-json

The response will list the silenced resource definition.

After the silenced entry starts to take effect, events that are silenced will be marked as such in the response:

   Entity         Check        Output       Status     Silenced          Timestamp
──────────────   ─────────    ─────────   ──────────── ────────── ───────────────────────────────
   i-424242      check-http                    0          true     2021-03-14 13:22:16 -0400 EDT

Next steps

Read the silencing reference for in-depth documentation about silenced entries.