Event filter reference

Sensu executes event filters during the filter stage of the observability pipeline.

Sensu event filters are applied when you configure event handlers to use one or more filters. Before executing a handler, the Sensu backend will apply any event filters configured for the handler to the observation data in events. If the filters do not remove the event, the handler will be executed.

The filter analysis performs these steps:

• When the Sensu backend is processing an event, it checks for the definition of a handler (or handlers). Before executing each handler, the Sensu server first applies any configured filters for the handler.
• If multiple filters are configured for a handler, they are executed sequentially.
• Filter expressions are compared with event data.

Event filters can be inclusive (only matching events are handled) or exclusive (matching events are not handled).

As soon as a filter removes an event, no further analysis is performed and the event handler will not be executed.

Event filter example (minimum required attributes)

This example shows the minimum required attributes for an event filter resource:

type: EventFilter
api_version: core/v2
metadata:
  name: filter_minimum
  namespace: default
spec:
  action: allow
  expressions:
  - event.check.occurrences == 1

{
  "type": "EventFilter",
  "api_version": "core/v2",
  "metadata": {
    "name": "filter_minimum",
    "namespace": "default"
  },
  "spec": {
    "action": "allow",
    "expressions": [
      "event.check.occurrences == 1"
    ]
  }
}

Inclusive and exclusive event filters

Event filters can be inclusive ("action": "allow"; replaces "negate": false in Sensu Core) or exclusive ("action": "deny"; replaces "negate": true in Sensu Core). Configuring a handler to use multiple inclusive event filters is the equivalent of using an AND query operator (only handle events if they match the inclusive filter: x AND y AND z). Configuring a handler to use multiple exclusive event filters is the equivalent of using an OR operator (only handle events if they don’t match x OR y OR z).

In inclusive filtering, by setting the event filter definition attribute "action": "allow", only events that match the defined filter expressions are handled.

In exclusive filtering, by setting the event filter definition attribute "action": "deny", events are only handled if they do not match the defined filter expressions.

Filter expression comparison

Event filter expressions are compared directly with their event data counterparts. For inclusive event filter definitions ("action": "allow"), matching expressions will result in the filter returning a true value. For exclusive event filter definitions ("action": "deny"), matching expressions will result in the filter returning a false value, and the event will not pass through the filter. Event filters that return a true value will continue to be processed via additional filters (if defined), mutators (if defined), and handlers.

Filter expression evaluation

When more complex conditional logic is needed than direct filter expression comparison, Sensu event filters provide support for expression evaluation using Otto. Otto is an ECMAScript 5 (JavaScript) virtual machine that evaluates JavaScript expressions provided in an event filter. There are some caveats to using Otto: not all of the regular expressions (regex) specified in ECMAScript 5 will work. See the Otto README for more details.

Use Go regex syntax to create event filter expressions that combine any available event, check, or entity attributes with match(<regex>).

For example, this event filter allows handling for events whose event.check.name ends with metrics:

---
type: EventFilter
api_version: core/v2
metadata:
  name: metrics-checks-only
  namespace: default
spec:
  action: allow
  expressions:
  - event.check.name.match(/metrics$/)

{
   "type": "EventFilter",
   "api_version": "core/v2",
   "metadata": {
      "name": "metrics-checks-only",
      "namespace": "default"
   },
   "spec": {
      "action": "allow",
      "expressions": [
         "event.check.name.match(/metrics$/)"
      ]
   }
}

Here’s another example that uses regex matching for event entity labels. This event filter allows handling for events created by entities with the region label us-west-1, us-west-2, or us-west-3:

---
type: EventFilter
api_version: core/v2
metadata:
  name: us-west-events
  namespace: default
spec:
  action: allow
  expressions:
  - event.entity.labels.region.match(/us-west-\b[1-3]\b/)

{
   "type": "EventFilter",
   "api_version": "core/v2",
   "metadata": {
      "name": "us-west-events",
      "namespace": "default"
   },
   "spec": {
      "action": "allow",
      "expressions": [
         "event.entity.labels.region.match(/us-west-\b[1-3]\b/)"
      ]
   }
}

Filter dynamic runtime assets

Sensu event filters can have dynamic runtime assets that are included in their execution context. When valid dynamic runtime assets are associated with an event filter, Sensu evaluates any files it finds that have a .js extension before executing the filter. The result of evaluating the scripts is cached for a given asset set for the sake of performance. For an example of how to implement an event filter as an asset, read Reduce alert fatigue.

Built-in event filters

Sensu includes built-in event filters to help you customize event pipelines for metrics and alerts. To start using built-in event filters, read Send Slack alerts and Plan maintenance.

Built-in filter: is_incident

The is_incident event filter is included in every installation of the Sensu backend. You can use the is_incident filter to allow only high-priority events through a Sensu pipeline. For example, you can use the is_incident filter to reduce noise when sending notifications to Slack. When applied to a handler, the is_incident filter allows warning ("status": 1), critical ("status": 2), other (unknown or custom status), and resolution events to be processed.

To use the is_incident event filter, include is_incident in the handler configuration filters array:

type: Handler
api_version: core/v2
metadata:
  name: slack
  namespace: default
spec:
  command: sensu-slack-handler --channel '#monitoring'
  env_vars:
  - SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX
  filters:
  - is_incident
  handlers: []
  runtime_assets: []
  timeout: 0
  type: pipe

{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "name": "slack",
    "namespace": "default"
  },
  "spec": {
    "command": "sensu-slack-handler --channel '#monitoring'",
    "env_vars": [
      "SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"
    ],
    "filters": [
      "is_incident"
    ],
    "handlers": [],
    "runtime_assets": [],
    "timeout": 0,
    "type": "pipe"
  }
}

The is_incident event filter applies the following filtering logic:

status	allow	discard
0
1
2
other (unknown or custom status)
resolution event such as 1 –> 0 or 3 –> 0

Built-in filter: not_silenced

Sensu silencing lets you suppress execution of event handlers on an on-demand basis so you can quiet incoming alerts and plan maintenance.

To allow silencing for an event handler, add not_silenced to the handler configuration filters array:

type: Handler
api_version: core/v2
metadata:
  name: slack
  namespace: default
spec:
  command: sensu-slack-handler --channel '#monitoring'
  env_vars:
  - SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX
  filters:
  - is_incident
  - not_silenced
  handlers: []
  runtime_assets: []
  timeout: 0
  type: pipe

{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "name": "slack",
    "namespace": "default"
  },
  "spec": {
    "command": "sensu-slack-handler --channel '#monitoring'",
    "env_vars": [
      "SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"
    ],
    "filters": [
      "is_incident",
      "not_silenced"
    ],
    "handlers": [],
    "runtime_assets": [],
    "timeout": 0,
    "type": "pipe"
  }
}

When applied to a handler configuration, the not_silenced event filter silences events that include the silenced attribute. The handler in the example above uses both the not_silenced and is_incident event filters, preventing low-priority and silenced events from being sent to Slack.

Built-in filter: has_metrics

The has_metrics event filter is included in every installation of the Sensu backend. When applied to a handler, the has_metrics filter allows only events that contain Sensu metrics to be processed. You can use the has_metrics filter to prevent handlers that require metrics from failing in case of an error in metric collection.

To use the has_metrics event filter, include has_metrics in the handler configuration filters array:

type: Handler
api_version: core/v2
metadata:
  name: influx-db
  namespace: default
spec:
  command: sensu-influxdb-handler -d sensu
  env_vars:
  - INFLUXDB_ADDR=http://influxdb.default.svc.cluster.local:8086
  - INFLUXDB_USER=sensu
  - INFLUXDB_PASSWORD=password
  filters:
  - has_metrics
  handlers: []
  runtime_assets: []
  timeout: 0
  type: pipe

{
  "type": "Handler",
  "api_version": "core/v2",
  "metadata": {
    "name": "influx-db",
    "namespace": "default"
  },
  "spec": {
    "command": "sensu-influxdb-handler -d sensu",
    "env_vars": [
      "INFLUXDB_ADDR=http://influxdb.default.svc.cluster.local:8086",
      "INFLUXDB_USER=sensu",
      "INFLUXDB_PASSWORD=password"
    ],
    "filters": [
      "has_metrics"
    ],
    "handlers": [],
    "runtime_assets": [],
    "timeout": 0,
    "type": "pipe"
  }
}

When applied to a handler configuration, the has_metrics event filter allows only events that include a metrics scope.

Build event filter expressions with Sensu query expressions

You can write custom event filter expressions as Sensu query expressions using the event data attributes described in this section. For more information about event attributes, see the event reference.

Syntax quick reference

Event attributes available to filters

Check attributes available to filters

Entity attributes available to filters

Build event filter expressions with JavaScript execution functions

In addition to Sensu query expressions, Sensu includes several built-in JavaScript functions for event filter execution:

• sensu.FetchEvent
• sensu.CheckStatus
• sensu.ListEvents

Use these functions to query your event stores for other events in the same namespace.

For example, to handle only events for the server01 entity and the disk check, use the sensu.FetchEvent function in your event filter expressions:

"expressions": [
  '(function () { var diskEvent = sensu.FetchEvent("server01", "disk"); if (diskEvent == nil) { return false; } return diskEvent.check.status == 0; })()'
]

sensu.EventStatus

The sensu.EventStatus function takes zero or more checks as arguments. It returns an array of status codes for the events associated with the specified checks.

If you do not specify any checks, the function always returns an empty array.

You can refer to the checks as strings:

sensu.EventStatus("database", "disk")

If you pass the check names as strings, Sensu assumes that the entities are the same as those in the events being filtered.

You can also refer to the checks in objects that include both the entity and check name. For example:

sensu.EventStatus({entity: "server01", check: "disk"}, {entity: "server01", check: "database"})

In both cases, if no event matches the specified entities and checks, Sensu will raise an error.

sensu.FetchEvent

The sensu.FetchEvent function loads the Sensu event that corresponds to the specified entity and check names.

The format is sensu.FetchEvent(entity, check). For example:

sensu.FetchEvent("server01", "disk")

You can only load events from the same namespace as the event being filtered. The returned object uses the same format as responses for the events API.

If an event does not exist for the specified entity and check names, Sensu will raise an error.

sensu.ListEvents

The sensu.ListEvents function returns an array of all events in the same namespace as the event being filtered.

For example:

sensu.ListEvents()

The events in the returned array use the same format as responses for the events API.

Event filter specification

Top-level attributes

type: EventFilter

{
  "type": "EventFilter"
}

api_version: core/v2

{
  "api_version": "core/v2"
}

metadata:
  name: filter-weekdays-only
  namespace: default
  created_by: admin
  labels:
    region: us-west-1
  annotations:
    slack-channel: "#monitoring"

{
  "metadata": {
    "name": "filter-weekdays-only",
    "namespace": "default",
    "created_by": "admin",
    "labels": {
      "region": "us-west-1"
    },
    "annotations": {
      "slack-channel": "#monitoring"
    }
  }
}

spec:
  action: allow
  expressions:
  - event.entity.namespace == 'production'
  runtime_assets: []

{
  "spec": {
    "action": "allow",
    "expressions": [
      "event.entity.namespace == 'production'"
    ],
    "runtime_assets": []
  }
}

Metadata attributes

name: filter-weekdays-only

{
  "name": "filter-weekdays-only"
}

namespace: production

{
  "namespace": "production"
}

created_by: admin

{
  "created_by": "admin"
}

labels:
  environment: development
  region: us-west-2

{
  "labels": {
    "environment": "development",
    "region": "us-west-2"
  }
}

annotations:
  managed-by: ops
  playbook: www.example.url

{
  "annotations": {
    "managed-by": "ops",
    "playbook": "www.example.url"
  }
}

Spec attributes

action: allow

{
  "action": "allow"
}

expressions:
- event.check.team == 'ops'

{
  "expressions": [
    "event.check.team == 'ops'"
  ]
}

runtime_assets:
- underscore

{
  "runtime_assets": [
    "underscore"
  ]
}

Use JavaScript libraries with Sensu filters

You can include JavaScript libraries in their event filter execution context with dynamic runtime assets. For instance, if you package underscore.js into a Sensu asset, you can use functions from the underscore library for filter expressions:

type: EventFilter
api_version: core/v2
metadata:
  annotations: null
  labels: null
  name: deny_if_failure_in_history
  namespace: default
spec:
  action: deny
  expressions:
  - _.reduce(event.check.history, function(memo, h) { return (memo || h.status !=
    0); })
  runtime_assets:
  - underscore

{
  "type": "EventFilter",
  "api_version": "core/v2",
  "metadata": {
    "name": "deny_if_failure_in_history",
    "namespace": "default",
    "labels": null,
    "annotations": null
  },
  "spec": {
    "action": "deny",
    "expressions": [
      "_.reduce(event.check.history, function(memo, h) { return (memo || h.status != 0); })"
    ],
    "runtime_assets": ["underscore"]
  }
}

Filter for production events

The following event filter allows handling for only events with a custom entity label "environment": "production":

type: EventFilter
api_version: core/v2
metadata:
  name: production_filter
  namespace: default
spec:
  action: allow
  expressions:
  - event.entity.labels['environment'] == 'production'

{
  "type": "EventFilter",
  "api_version": "core/v2",
  "metadata": {
    "name": "production_filter",
    "namespace": "default"
  },
  "spec": {
    "action": "allow",
    "expressions": [
      "event.entity.labels['environment'] == 'production'"
    ]
  }
}

Filter for non-production events

The following event filter discards events with a custom entity label "environment": "production", allowing handling only for events without an environment label or events with environment set to something other than production.

type: EventFilter
api_version: core/v2
metadata:
  name: not_production
  namespace: default
spec:
  action: deny
  expressions:
  - event.entity.labels['environment'] == 'production'

{
  "type": "EventFilter",
  "api_version": "core/v2",
  "metadata": {
    "name": "not_production",
    "namespace": "default"
  },
  "spec": {
    "action": "deny",
    "expressions": [
      "event.entity.labels['environment'] == 'production'"
    ]
  }
}

Filter for state change only

This example demonstrates how to use the state_change_only inclusive event filter to reproduce the behavior of a monitoring system that alerts only on state change:

type: EventFilter
api_version: core/v2
metadata:
  annotations: null
  labels: null
  name: state_change_only
  namespace: default
spec:
  action: allow
  expressions:
  - event.check.occurrences == 1
  runtime_assets: []

{
  "type": "EventFilter",
  "api_version": "core/v2",
  "metadata": {
    "name": "state_change_only",
    "namespace": "default",
    "labels": null,
    "annotations": null
  },
  "spec": {
    "action": "allow",
    "expressions": [
      "event.check.occurrences == 1"
    ],
    "runtime_assets": []
  }
}

Filter for repeated events

In this example, the filter_interval_60_hourly event filter will match event data with a check interval of 60 seconds AND an occurrences value of 1 (the first occurrence) OR any occurrences value that is evenly divisible by 60 via a modulo operator calculation (calculating the remainder after dividing occurrences by 60):

type: EventFilter
api_version: core/v2
metadata:
  annotations: null
  labels: null
  name: filter_interval_60_hourly
  namespace: default
spec:
  action: allow
  expressions:
  - event.check.interval == 60
  - event.check.occurrences == 1 || event.check.occurrences % 60 == 0
  runtime_assets: []

{
  "type": "EventFilter",
  "api_version": "core/v2",
  "metadata": {
    "name": "filter_interval_60_hourly",
    "namespace": "default",
    "labels": null,
    "annotations": null
  },
  "spec": {
    "action": "allow",
    "expressions": [
      "event.check.interval == 60",
      "event.check.occurrences == 1 || event.check.occurrences % 60 == 0"
    ],
    "runtime_assets": []
  }
}

This example will apply the same logic as the previous example but for checks with a 30-second interval:

type: EventFilter
api_version: core/v2
metadata:
  annotations: null
  labels: null
  name: filter_interval_30_hourly
  namespace: default
spec:
  action: allow
  expressions:
  - event.check.interval == 30
  - event.check.occurrences == 1 || event.check.occurrences % 120 == 0
  runtime_assets: []

{
  "type": "EventFilter",
  "api_version": "core/v2",
  "metadata": {
    "name": "filter_interval_30_hourly",
    "namespace": "default",
    "labels": null,
    "annotations": null
  },
  "spec": {
    "action": "allow",
    "expressions": [
      "event.check.interval == 30",
      "event.check.occurrences == 1 || event.check.occurrences % 120 == 0"
    ],
    "runtime_assets": []
  }
}

Filter to reduce alert fatigue for keepalive events

This example keepalive_timeouts event filter will match event data with an occurrences value of 1 OR any occurrences value that matches 15 minutes via a modulo operator calculation. This limits keepalive timeout event alerts to the first occurrence and every 15 minutes thereafter.

This example uses conditional JavaScript logic to check for an entity-level annotation, keepalive_alert_minutes, and if it exists, parses the annotation value as an integer. If the annotation does not exist, the event filter uses 15 minutes for the alert cadence.

type: EventFilter
api_version: core/v2
metadata:
  annotations: null
  labels: null
  name: keepalive_timeouts
  namespace: default
spec:
  action: allow
  expressions:
   - is_incident
   - event.check.occurrences == 1 || event.check.occurrences % parseInt( 60 * ( 'keepalive_alert_minutes' in event.entity.annotations ? parseInt(event.entity.annotations.keepalive_alert_minutes): 15) / event.check.timeout ) == 0
  runtime_assets: []

{
  "type": "EventFilter",
  "api_version": "core/v2",
  "metadata": {
    "name": "keepalive_timeouts",
    "namespace": "default",
    "labels": null,
    "annotations": null
  },
  "spec": {
    "action": "allow",
    "expressions": [
      "is_incident",
      "event.check.occurrences == 1 || event.check.occurrences % parseInt( 60 * ( 'keepalive_alert_minutes' in event.entity.annotations ? parseInt(event.entity.annotations.keepalive_alert_minutes): 15) / event.check.timeout ) == 0"
    ],
    "runtime_assets": []
  }
}

Filter for events during office hours only

This event filter evaluates the event timestamp to determine if the event occurred between 9 AM and 5 PM UTC on a weekday. Remember that action is equal to allow, so this is an inclusive event filter. If evaluation returns false, the event will not be handled.

type: EventFilter
api_version: core/v2
metadata:
  annotations: null
  labels: null
  name: nine_to_fiver
  namespace: default
spec:
  action: allow
  expressions:
  - weekday(event.timestamp) >= 1 && weekday(event.timestamp) <= 5
  - hour(event.timestamp) >= 9 && hour(event.timestamp) <= 17
  runtime_assets: []

{
  "type": "EventFilter",
  "api_version": "core/v2",
  "metadata": {
    "name": "nine_to_fiver",
    "namespace": "default",
    "labels": null,
    "annotations": null
  },
  "spec": {
    "action": "allow",
    "expressions": [
      "weekday(event.timestamp) >= 1 && weekday(event.timestamp) <= 5",
      "hour(event.timestamp) >= 9 && hour(event.timestamp) <= 17"
    ],
    "runtime_assets": []
  }
}

Add filter expressions that use the minute and second custom functions for more granular control. For example, if office hours are 8:30 AM to 5:30 PM:

type: EventFilter
api_version: core/v2
metadata:
  annotations: null
  labels: null
  name: 830_to_530
  namespace: default
spec:
  action: allow
  expressions:
  - weekday(event.timestamp) >= 1 && weekday(event.timestamp) <= 5
  - hour(event.timestamp) >= 8 && minute(event.timestamp) >= 30
  - hour(event.timestamp) <= 17 && minute(event.timestamp) <= 30
  runtime_assets: []

{
  "type": "EventFilter",
  "api_version": "core/v2",
  "metadata": {
    "name": "830_to_530",
    "namespace": "default",
    "labels": null,
    "annotations": null
  },
  "spec": {
    "action": "allow",
    "expressions": [
      "weekday(event.timestamp) >= 1 && weekday(event.timestamp) <= 5",
      "hour(event.timestamp) >= 8 && minute(event.timestamp) >= 30",
      "hour(event.timestamp) <= 17 && minute(event.timestamp) <= 30"
    ],
    "runtime_assets": []
  }
}

Filter for events not processed within 30 seconds

This event filter evaluates the event timestamp to determine if the event was created more than 30 seconds since the current time. In other words, this filter sets a 30-second time budget for event processing so you can identify and handle events that aren’t processed within 30 seconds.

type: EventFilter
api_version: core/v2
metadata:
  annotations: null
  labels: null
  name: budget_30
  namespace: default
spec:
  action: allow
  expressions:
  - seconds_since(event.timestamp) > 30
  runtime_assets: []

{
  "type": "EventFilter",
  "api_version": "core/v2",
  "metadata": {
    "name": "budget_30",
    "namespace": "default",
    "labels": null,
    "annotations": null
  },
  "spec": {
    "action": "allow",
    "expressions": [
      "seconds_since(event.timestamp) > 30"
    ],
    "runtime_assets": []
  }
}

Disable alerts without a silence

This filter allows you to disable alerts without creating silences.

Add the filter name to the filters array for any handler you want to control. To disable alerts, change the filter’s action attribute value from allow to deny.

---
type: EventFilter
api_version: core/v2
metadata:
  name: emergency_alert_control
spec:
  action: allow
  expressions:
  - event.has_check

{
  "type": "EventFilter",
  "api_version": "core/v2",
  "metadata": {
    "name": "emergency_alert_control"
  },
  "spec": {
    "action": "allow",
    "expressions": [
      "event.has_check"
    ]
  }
}