Event filter reference
Sensu executes event filters during the filter stage of the observability pipeline.
Sensu event filters are applied when you configure event handlers to use one or more filters. Before executing a handler, the Sensu backend will apply any event filters configured for the handler to the observation data in events. If the filters do not remove the event, the handler will be executed.
The filter analysis performs these steps:
- When the Sensu backend is processing an event, it checks for the definition of a
handler
(orhandlers
). Before executing each handler, the Sensu server first applies any configuredfilters
for the handler. - If multiple
filters
are configured for a handler, they are executed sequentially. - Filter
expressions
are compared with event data.
Event filters can be inclusive (only matching events are handled) or exclusive (matching events are not handled).
As soon as a filter removes an event, no further analysis is performed and the event handler will not be executed.
NOTE: Filters specified in a handler set definition have no effect. Filters must be specified in individual handler definitions.
Inclusive and exclusive event filters
Event filters can be inclusive ("action": "allow"
; replaces "negate": false
in Sensu Core) or exclusive ("action": "deny"
; replaces "negate": true
in Sensu Core).
Configuring a handler to use multiple inclusive event filters is the equivalent of using an AND
query operator (only handle events if they match the inclusive filter: x AND y AND z
).
Configuring a handler to use multiple exclusive event filters is the equivalent of using an OR
operator (only handle events if they don’t match x OR y OR z
).
In inclusive filtering, by setting the event filter definition attribute "action": "allow"
, only events that match the defined filter expressions are handled.
In exclusive filtering, by setting the event filter definition attribute "action": "deny"
, events are only handled if they do not match the defined filter expressions.
Filter expression comparison
Event filter expressions are compared directly with their event data counterparts.
For inclusive event filter definitions ("action": "allow"
), matching expressions will result in the filter returning a true
value.
For exclusive event filter definitions ("action": "deny"
), matching expressions will result in the filter returning a false
value, and the event will not pass through the filter.
Event filters that return a true
value will continue to be processed via additional filters (if defined), mutators (if defined), and handlers.
Filter expression evaluation
When more complex conditional logic is needed than direct filter expression comparison, Sensu event filters provide support for expression evaluation using Otto. Otto is an ECMAScript 5 (JavaScript) virtual machine that evaluates JavaScript expressions provided in an event filter. There are some caveats to using Otto: not all of the regular expressions specified in ECMAScript 5 will work. See the Otto README for more details.
Filter dynamic runtime assets
Sensu event filters can have dynamic runtime assets that are included in their execution context. When valid dynamic runtime assets are associated with an event filter, Sensu evaluates any files it finds that have a “.js” extension before executing the filter. The result of evaluating the scripts is cached for a given asset set for the sake of performance. For an example of how to implement an event filter as an asset, see Reduce alert fatigue.
Built-in event filters
Sensu includes built-in event filters to help you customize event pipelines for metrics and alerts. To start using built-in event filters, see Send Slack alerts and Plan maintenance.
NOTE: Sensu Go does not include the built-in occurrence-based event filter in Sensu Core 1.x, but you can replicate its functionality with the repeated events filter definition.
Built-in filter: is_incident
The is_incident event filter is included in every installation of the Sensu backend.
You can use the is_incident filter to allow only high-priority events through a Sensu pipeline.
For example, you can use the is_incident filter to reduce noise when sending notifications to Slack.
When applied to a handler, the is_incident filter allows warning ("status": 1
), critical ("status": 2
), other (unknown or custom status), and resolution events to be processed.
To use the is_incident event filter, include is_incident
in the handler configuration filters
array:
type: Handler
api_version: core/v2
metadata:
name: slack
namespace: default
spec:
command: sensu-slack-handler --channel '#monitoring'
env_vars:
- SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX
filters:
- is_incident
handlers: []
runtime_assets: []
timeout: 0
type: pipe
{
"type": "Handler",
"api_version": "core/v2",
"metadata": {
"name": "slack",
"namespace": "default"
},
"spec": {
"command": "sensu-slack-handler --channel '#monitoring'",
"env_vars": [
"SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"
],
"filters": [
"is_incident"
],
"handlers": [],
"runtime_assets": [],
"timeout": 0,
"type": "pipe"
}
}
The is_incident event filter applies the following filtering logic:
status | allow | discard | ||||
---|---|---|---|---|---|---|
0 | ![]() |
|||||
1 | ![]() |
|||||
2 | ![]() |
|||||
other (unknown or custom status) | ![]() |
|||||
resolution event such as 1 –> 0 or 3 –> 0 |
![]() |
Built-in filter: not_silenced
Sensu silencing lets you suppress execution of event handlers on an on-demand basis so you can quiet incoming alerts and plan maintenance.
To allow silencing for an event handler, add not_silenced
to the handler configuration filters
array:
type: Handler
api_version: core/v2
metadata:
name: slack
namespace: default
spec:
command: sensu-slack-handler --channel '#monitoring'
env_vars:
- SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX
filters:
- is_incident
- not_silenced
handlers: []
runtime_assets: []
timeout: 0
type: pipe
{
"type": "Handler",
"api_version": "core/v2",
"metadata": {
"name": "slack",
"namespace": "default"
},
"spec": {
"command": "sensu-slack-handler --channel '#monitoring'",
"env_vars": [
"SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"
],
"filters": [
"is_incident",
"not_silenced"
],
"handlers": [],
"runtime_assets": [],
"timeout": 0,
"type": "pipe"
}
}
When applied to a handler configuration, the not_silenced event filter silences events that include the silenced
attribute.
The handler in the example above uses both the not_silenced and is_incident event filters, preventing low-priority and silenced events from being sent to Slack.
Built-in filter: has_metrics
The has_metrics event filter is included in every installation of the Sensu backend. When applied to a handler, the has_metrics filter allows only events that contain Sensu metrics to be processed. You can use the has_metrics filter to prevent handlers that require metrics from failing in case of an error in metric collection.
To use the has_metrics event filter, include has_metrics
in the handler configuration filters
array:
type: Handler
api_version: core/v2
metadata:
name: influx-db
namespace: default
spec:
command: sensu-influxdb-handler -d sensu
env_vars:
- INFLUXDB_ADDR=http://influxdb.default.svc.cluster.local:8086
- INFLUXDB_USER=sensu
- INFLUXDB_PASSWORD=password
filters:
- has_metrics
handlers: []
runtime_assets: []
timeout: 0
type: pipe
{
"type": "Handler",
"api_version": "core/v2",
"metadata": {
"name": "influx-db",
"namespace": "default"
},
"spec": {
"command": "sensu-influxdb-handler -d sensu",
"env_vars": [
"INFLUXDB_ADDR=http://influxdb.default.svc.cluster.local:8086",
"INFLUXDB_USER=sensu",
"INFLUXDB_PASSWORD=password"
],
"filters": [
"has_metrics"
],
"handlers": [],
"runtime_assets": [],
"timeout": 0,
"type": "pipe"
}
}
When applied to a handler configuration, the has_metrics event filter allows only events that include a metrics
scope.
Build event filter expressions with Sensu query expressions
You can write custom event filter expressions as Sensu query expressions using the event data attributes described in this section. For more information about event attributes, see the event reference.
Syntax quick reference
operator | description |
---|---|
=== / !== |
Identity operator / Nonidentity operator |
== / != |
Equality operator / Inequality operator |
&& / || |
Logical AND / Logical OR |
< / > |
Less than / Greater than |
<= / >= |
Less than or equal to / Greater than or equal to |
Event attributes available to filters
attribute | type | description |
---|---|---|
event.has_check |
Boolean | Returns true if the event contains check data |
event.has_metrics |
Boolean | Returns true if the event contains metrics |
event.is_incident |
Boolean | Returns true for critical alerts (status 2 ), warnings (status 1 ), and resolution events (status 0 transitioning from status 1 or 2 ) |
event.is_resolution |
Boolean | Returns true if the event status is OK (0 ) and the previous event was of a non-zero status |
event.is_silenced |
Boolean | Returns true if the event matches an active silencing entry |
event.timestamp |
integer | Time that the event occurred in seconds since the Unix epoch |
Check attributes available to filters
attribute | type | description |
---|---|---|
event.check.annotations |
map | Custom annotations applied to the check |
event.check.command |
string | The command executed by the check |
event.check.cron |
string | Check execution schedule using cron syntax |
event.check.discard_output |
Boolean | Whether the check is configured to discard check output from event data |
event.check.duration |
float | Command execution time in seconds |
event.check.env_vars |
array | Environment variables used with command execution |
event.check.executed |
integer | Time that the check was executed in seconds since the Unix epoch |
event.check.handlers |
array | Sensu event handlers assigned to the check |
event.check.high_flap_threshold |
integer | The check’s flap detection high threshold in percent state change |
event.check.history |
array | Check status history for the last 21 check executions |
event.check.hooks |
array | Check hook execution data |
event.check.interval |
integer | The check execution frequency in seconds |
event.check.issued |
integer | Time that the check request was issued in seconds since the Unix epoch |
event.check.labels |
map | Custom labels applied to the check |
event.check.last_ok |
integer | The last time that the check returned an OK status (0 ) in seconds since the Unix epoch |
event.check.low_flap_threshold |
integer | The check’s flap detection low threshold in percent state change |
event.check.max_output_size |
integer | Maximum size of stored check outputs in bytes |
event.check.name |
string | Check name |
event.check.occurrences |
integer | The number of preceding events with the same status as the current event |
event.check.occurrences_watermark |
integer | For resolution events, the number of preceding events with a non-OK status |
event.check.output |
string | The output from the execution of the check command |
event.check.output_metric_format |
string | The metric format generated by the check command: nagios_perfdata , graphite_plaintext , influxdb_line , opentsdb_line , or prometheus_text |
event.check.output_metric_handlers |
array | Sensu metric handlers assigned to the check |
event.check.proxy_entity_name |
string | The entity name, used to create a proxy entity for an external resource |
event.check.proxy_requests |
map | Proxy request configuration |
event.check.publish |
Boolean | Whether the check is scheduled automatically |
event.check.round_robin |
Boolean | Whether the check is configured to be executed in a round-robin style |
event.check.runtime_assets |
array | Sensu dynamic runtime assets used by the check |
event.check.state |
string | The state of the check: passing (status 0 ), failing (status other than 0 ), or flapping |
event.check.status |
integer | Exit status code produced by the check: 0 (OK), 1 (warning), 2 (critical), or other status (unknown or custom status) |
event.check.stdin |
Boolean | Whether the Sensu agent writes JSON-serialized entity and check data to the command process’ STDIN |
event.check.subscriptions |
array | Subscriptions that the check belongs to |
event.check.timeout |
integer | The check execution duration timeout in seconds |
event.check.total_state_change |
integer | The total state change percentage for the check’s history |
event.check.ttl |
integer | The time-to-live (TTL) until the event is considered stale, in seconds |
event.metrics.handlers |
array | Sensu metric handlers assigned to the check |
event.metrics.points |
array | Metric data points including a name, timestamp, value, and tags |
Entity attributes available to filters
attribute | type | description |
---|---|---|
event.entity.annotations |
map | Custom annotations assigned to the entity |
event.entity.deregister |
Boolean | Whether the agent entity should be removed when it stops sending keepalive messages |
event.entity.deregistration |
map | A map that contains a handler name for use when an entity is deregistered |
event.entity.entity_class |
string | The entity type: usually agent or proxy |
event.entity.labels |
map | Custom labels assigned to the entity |
event.entity.last_seen |
integer | Timestamp the entity was last seen in seconds since the Unix epoch |
event.entity.name |
string | Entity name |
event.entity.redact |
array | List of items to redact from log messages |
event.entity.subscriptions |
array | List of subscriptions assigned to the entity |
event.entity.system |
map | Information about the entity’s system |
event.entity.system.arch |
string | The entity’s system architecture |
event.entity.system.hostname |
string | The entity’s hostname |
event.entity.system.network |
map | The entity’s network interface list |
event.entity.system.os |
string | The entity’s operating system |
event.entity.system.platform |
string | The entity’s operating system distribution |
event.entity.system.platform_family |
string | The entity’s operating system family |
event.entity.system.platform_version |
string | The entity’s operating system version |
event.entity.user |
string | Sensu RBAC username used by the agent entity |
Build event filter expressions with JavaScript execution functions
COMMERCIAL FEATURE: Access built-in JavaScript event filter execution functions in the packaged Sensu Go distribution. For more information, see Get started with commercial features.
In addition to Sensu query expressions, Sensu includes several built-in JavaScript functions for event filter execution:
sensu.FetchEvent
sensu.CheckStatus
sensu.ListEvents
Use these functions to query your event stores for other events in the same namespace.
For example, to handle only events for the server01
entity and the disk
check, use the sensu.FetchEvent
function in your event filter expressions:
"expressions": [
'(function () { var diskEvent = sensu.FetchEvent("server01", "disk"); if (diskEvent == nil) { return false; } return diskEvent.check.status == 0; })()'
]
sensu.EventStatus
The sensu.EventStatus
function takes zero or more checks as arguments.
It returns an array of status codes for the events associated with the specified checks.
If you do not specify any checks, the function always returns an empty array.
You can refer to the checks as strings:
sensu.EventStatus("database", "disk")
If you pass the check names as strings, Sensu assumes that the entities are the same as those in the events being filtered.
You can also refer to the checks in objects that include both the entity and check name. For example:
sensu.EventStatus({entity: "server01", check: "disk"}, {entity: "server01", check: "database"})
In both cases, if no event matches the specified entities and checks, Sensu will raise an error.
sensu.FetchEvent
The sensu.FetchEvent
function loads the Sensu event that corresponds to the specified entity and check names.
The format is sensu.FetchEvent(entity, check)
.
For example:
sensu.FetchEvent("server01", "disk")
You can only load events from the same namespace as the event being filtered. The returned object uses the same format as responses for the events API.
If an event does not exist for the specified entity and check names, Sensu will raise an error.
sensu.ListEvents
The sensu.ListEvents
function returns an array of all events in the same namespace as the event being filtered.
NOTE: If you have many events in the namespace, this function may require a substantial amount of time to return them.
For example:
sensu.ListEvents()
The events in the returned array use the same format as responses for the events API.
Event filter specification
Top-level attributes
type | |
---|---|
description | Top-level attribute that specifies the sensuctl create resource type. Event filters should always be type EventFilter . |
required | Required for filter definitions in wrapped-json or yaml format for use with sensuctl create . |
type | String |
example |
|
api_version | |
---|---|
description | Top-level attribute that specifies the Sensu API group and version. For event filters in this version of Sensu, this attribute should always be core/v2 . |
required | Required for filter definitions in wrapped-json or yaml format for use with sensuctl create . |
type | String |
example |
|
metadata | |
---|---|
description | Top-level collection of metadata about the event filter, including name , namespace , and created_by as well as custom labels and annotations . The metadata map is always at the top level of the filter definition. This means that in wrapped-json and yaml formats, the metadata scope occurs outside the spec scope. See metadata attributes for details. |
required | Required for filter definitions in wrapped-json or yaml format for use with sensuctl create . |
type | Map of key-value pairs |
example |
|
spec | |
---|---|
description | Top-level map that includes the event filter spec attributes. |
required | Required for filter definitions in wrapped-json or yaml format for use with sensuctl create . |
type | Map of key-value pairs |
example |
|
Metadata attributes
name | |
---|---|
description | Unique string used to identify the event filter. Filter names cannot contain special characters or spaces (validated with Go regex \A[\w\.\-]+\z ). Each filter must have a unique name within its namespace. |
required | true |
type | String |
example |
|
namespace | |
---|---|
description | Sensu RBAC namespace that the event filter belongs to. |
required | false |
type | String |
default | default |
example |
|
created_by | |
---|---|
description | Username of the Sensu user who created the filter or last updated the filter. Sensu automatically populates the created_by field when the filter is created or updated. |
required | false |
type | String |
example |
|
labels | |
---|---|
description | Custom attributes to include with event data that you can use for response and web UI view filtering. If you include labels in your event data, you can filter API responses, sensuctl responses, and web UI views based on them. In other words, labels allow you to create meaningful groupings for your data. Limit labels to metadata you need to use for response filtering. For complex, non-identifying metadata that you will not need to use in response filtering, use annotations rather than labels. |
required | false |
type | Map of key-value pairs. Keys can contain only letters, numbers, and underscores and must start with a letter. Values can be any valid UTF-8 string. |
default | null |
example |
|
annotations | |
---|---|
description | Non-identifying metadata to include with event data that you can access with event filters. You can use annotations to add data that’s meaningful to people or external tools that interact with Sensu. In contrast to labels, you cannot use annotations in API response filtering, sensuctl response filtering, or web UI views. |
required | false |
type | Map of key-value pairs. Keys and values can be any valid UTF-8 string. |
default | null |
example |
|
Spec attributes
action | |
---|---|
description | Action to take with the event if the event filter expressions match. See Inclusive and exclusive event filters for more information. |
required | true |
type | String |
allowed values | allow , deny |
example |
|
expressions | |
---|---|
description | Event filter expressions to be compared with event data. You can reference event metadata without including the metadata scope (for example, event.entity.namespace ). |
required | true |
type | Array |
example |
|
runtime_assets | |
---|---|
description | Dynamic runtime assets to apply to the event filter’s execution context. JavaScript files in the lib directory of the dynamic runtime asset will be evaluated. |
required | false |
type | Array of string |
default | [] |
example |
|
Event filter examples
Minimum required filter attributes
type: EventFilter
api_version: core/v2
metadata:
name: filter_minimum
namespace: default
spec:
action: allow
expressions:
- event.check.occurrences == 1
{
"type": "EventFilter",
"api_version": "core/v2",
"metadata": {
"name": "filter_minimum",
"namespace": "default"
},
"spec": {
"action": "allow",
"expressions": [
"event.check.occurrences == 1"
]
}
}
Handle production events
The following event filter allows handling for only events with a custom entity label "environment": "production"
:
type: EventFilter
api_version: core/v2
metadata:
name: production_filter
namespace: default
spec:
action: allow
expressions:
- event.entity.labels['environment'] == 'production'
{
"type": "EventFilter",
"api_version": "core/v2",
"metadata": {
"name": "production_filter",
"namespace": "default"
},
"spec": {
"action": "allow",
"expressions": [
"event.entity.labels['environment'] == 'production'"
]
}
}
Handle non-production events
The following event filter discards events with a custom entity label "environment": "production"
, allowing handling only for events without an environment
label or events with environment
set to something other than production
.
NOTE: action
is deny
, so this is an exclusive event filter.
If evaluation returns false, the event is handled.
type: EventFilter
api_version: core/v2
metadata:
name: not_production
namespace: default
spec:
action: deny
expressions:
- event.entity.labels['environment'] == 'production'
{
"type": "EventFilter",
"api_version": "core/v2",
"metadata": {
"name": "not_production",
"namespace": "default"
},
"spec": {
"action": "deny",
"expressions": [
"event.entity.labels['environment'] == 'production'"
]
}
}
Handle state change only
This example demonstrates how to use the state_change_only
inclusive event filter to reproduce the behavior of a monitoring system that alerts only on state change:
type: EventFilter
api_version: core/v2
metadata:
annotations: null
labels: null
name: state_change_only
namespace: default
spec:
action: allow
expressions:
- event.check.occurrences == 1
runtime_assets: []
{
"type": "EventFilter",
"api_version": "core/v2",
"metadata": {
"name": "state_change_only",
"namespace": "default",
"labels": null,
"annotations": null
},
"spec": {
"action": "allow",
"expressions": [
"event.check.occurrences == 1"
],
"runtime_assets": []
}
}
Handle repeated events
In this example, the filter_interval_60_hourly
event filter will match event data with a check interval
of 60
seconds AND an occurrences
value of 1
(the first occurrence) OR any occurrences
value that is evenly divisible by 60 via a modulo operator calculation (calculating the remainder after dividing occurrences
by 60):
type: EventFilter
api_version: core/v2
metadata:
annotations: null
labels: null
name: filter_interval_60_hourly
namespace: default
spec:
action: allow
expressions:
- event.check.interval == 60
- event.check.occurrences == 1 || event.check.occurrences % 60 == 0
runtime_assets: []
{
"type": "EventFilter",
"api_version": "core/v2",
"metadata": {
"name": "filter_interval_60_hourly",
"namespace": "default",
"labels": null,
"annotations": null
},
"spec": {
"action": "allow",
"expressions": [
"event.check.interval == 60",
"event.check.occurrences == 1 || event.check.occurrences % 60 == 0"
],
"runtime_assets": []
}
}
This example will apply the same logic as the previous example but for checks with a 30-second interval
:
type: EventFilter
api_version: core/v2
metadata:
annotations: null
labels: null
name: filter_interval_30_hourly
namespace: default
spec:
action: allow
expressions:
- event.check.interval == 30
- event.check.occurrences == 1 || event.check.occurrences % 120 == 0
runtime_assets: []
{
"type": "EventFilter",
"api_version": "core/v2",
"metadata": {
"name": "filter_interval_30_hourly",
"namespace": "default",
"labels": null,
"annotations": null
},
"spec": {
"action": "allow",
"expressions": [
"event.check.interval == 30",
"event.check.occurrences == 1 || event.check.occurrences % 120 == 0"
],
"runtime_assets": []
}
}
Handle events during office hours only
This event filter evaluates the event timestamp to determine if the event occurred between 9 AM and 5 PM UTC on a weekday.
Remember that action
is equal to allow
, so this is an inclusive event filter.
If evaluation returns false, the event will not be handled.
type: EventFilter
api_version: core/v2
metadata:
annotations: null
labels: null
name: nine_to_fiver
namespace: default
spec:
action: allow
expressions:
- weekday(event.timestamp) >= 1 && weekday(event.timestamp) <= 5
- hour(event.timestamp) >= 9 && hour(event.timestamp) <= 17
runtime_assets: []
{
"type": "EventFilter",
"api_version": "core/v2",
"metadata": {
"name": "nine_to_fiver",
"namespace": "default",
"labels": null,
"annotations": null
},
"spec": {
"action": "allow",
"expressions": [
"weekday(event.timestamp) >= 1 && weekday(event.timestamp) <= 5",
"hour(event.timestamp) >= 9 && hour(event.timestamp) <= 17"
],
"runtime_assets": []
}
}
Use JavaScript libraries with Sensu filters
You can include JavaScript libraries in their event filter execution context with dynamic runtime assets. For instance, if you package underscore.js into a Sensu asset, you can use functions from the underscore library for filter expressions:
type: EventFilter
api_version: core/v2
metadata:
annotations: null
labels: null
name: deny_if_failure_in_history
namespace: default
spec:
action: deny
expressions:
- _.reduce(event.check.history, function(memo, h) { return (memo || h.status !=
0); })
runtime_assets:
- underscore
{
"type": "EventFilter",
"api_version": "core/v2",
"metadata": {
"name": "deny_if_failure_in_history",
"namespace": "default",
"labels": null,
"annotations": null
},
"spec": {
"action": "deny",
"expressions": [
"_.reduce(event.check.history, function(memo, h) { return (memo || h.status != 0); })"
],
"runtime_assets": ["underscore"]
}
}