Datastore

Sensu stores the most recent event for each entity and check pair using either an etcd (default) or PostgreSQL database. You can access event data with the Sensu web UI Events page, sensuctl event commands, and the events API. For longer retention of event data, integrate Sensu with a time series database like InfluxDB or a searchable index like ElasticSearch or Splunk.

Use default event storage

By default, Sensu uses its embedded etcd database to store configuration and event data. This embedded database allows you to get started with Sensu without deploying a complete, scalable architecture.

Sensu can be configured to disable the embedded etcd database and use one or more external etcd nodes for configuration and event storage instead.

As your deployment grows beyond the proof-of-concept stage, review Deployment architecture for Sensu for more information about deployment considerations and recommendations for a production-ready Sensu deployment.

Sensu requires at least etcd 3.3.2 and is tested against releases in the 3.3.x series. etcd versions 3.4.0 and later are not supported.

Scale event storage

COMMERCIAL FEATURE: Access enterprise-scale event storage in the packaged Sensu Go distribution. For more information, see Get started with commercial features.

Sensu supports using an external PostgreSQL instance for event storage in place of etcd. PostgreSQL can handle significantly higher volumes of Sensu events, which allows you to scale Sensu beyond etcd’s 8-GB limit.

When configured with a PostgreSQL event store, Sensu connects to PostgreSQL to store and retrieve event data in place of etcd. Etcd continues to store Sensu entity and configuration data. You can access event data stored in PostgreSQL using the same Sensu web UI, API, and sensuctl processes as etcd-stored events.

Requirements

Sensu supports PostgreSQL 9.5 and later, including Amazon Relational Database Service (Amazon RDS) when configured with the PostgreSQL engine. See the PostgreSQL docs to install and configure PostgreSQL.

Configuration

At the time when you enable the PostgreSQL event store, event data cuts over from etcd to PostgreSQL. This results in a loss of recent event history. No restarts or Sensu backend configuration changes are required to enable the PostgreSQL event store.

When you successfully enable PostgreSQL as the Sensu Go event store, the Sensu backend log will include a message like this:

Mar 10 17:44:45 sensu-centos sensu-backend[1365]: {"component":"store-providers","level":"warning","msg":"switched event store to postgres","time":"2020-03-10T17:44:45Z"}

After you install and configure PostgreSQL, configure Sensu by creating a PostgresConfig resource. See Datastore specification for more information.

type: PostgresConfig
api_version: store/v1
metadata:
  name: my-postgres
spec:
  dsn: "postgresql://user:secret@host:port/dbname"
  max_conn_lifetime: 5m
  max_idle_conns: 2
  pool_size: 20
{
  "type": "PostgresConfig",
  "api_version": "store/v1",
  "metadata": {
    "name": "my-postgres"
  },
  "spec": {
    "dsn": "postgresql://user:secret@host:port/dbname",
    "max_conn_lifetime": "5m",
    "max_idle_conns": 2,
    "pool_size": 20
  }
}

With the PostgresConfig resource definition saved to a file (for example, postgres.yml), use sensuctl, configured as the admin user, to activate the PostgreSQL event store.

sensuctl create --file postgres.yml

To update your Sensu PostgreSQL configuration, repeat the sensuctl create process. You can expect to see PostgreSQL status updates in the Sensu backend logs at the warn log level and PostgreSQL error messages in the Sensu backend logs at the error log level.

Disable the PostgreSQL event store

To disable the PostgreSQL event store, use sensuctl delete with your PostgresConfig resource definition:

sensuctl delete --file postgres.yml

The Sensu backend log will include a message to record that you successfully disabled PostgreSQL as the Sensu Go event store:

Mar 10 17:35:04 sensu-centos sensu-backend[1365]: {"component":"store-providers","level":"warning","msg":"switched event store to etcd","time":"2020-03-10T17:35:04Z"}

When you disable the PostgreSQL event store, event data cuts over from PostgreSQL to etcd, which results in a loss of recent event history. No restarts or Sensu backend configuration changes are required to disable the PostgreSQL event store.

Datastore specification

Top-level attributes

type
description Top-level attribute that specifies the sensuctl create resource type. PostgreSQL datastore configs should always be type PostgresConfig.
required true
type String
example 
type: PostgresConfig
api_version
description Top-level attribute that specifies the Sensu API group and version. For PostgreSQL datastore configs, the api_version should be store/v1.
required true
type String
example 
api_version: store/v1
metadata
description Top-level scope that contains the PostgreSQL datastore name and created_by field.
required true
type Map of key-value pairs
example 
metadata:
  name: my-postgres
  created_by: admin
spec
description Top-level map that includes the PostgreSQL datastore config spec attributes.
required true
type Map of key-value pairs
example 
spec:
  dsn: "postgresql://user:secret@host:port/dbname"
  max_conn_lifetime: 5m
  max_idle_conns: 2
  pool_size: 20

Metadata attributes

name
description PostgreSQL datastore name used internally by Sensu.
required true
type String
example 
name: my-postgres
created_by
description Username of the Sensu user who created the datastore or last updated the datastore. Sensu automatically populates the created_by field when the datastore is created or updated.
required false
type String
example 
created_by: admin

Spec attributes

dsn
description Data source names. Specified as a URL or PostgreSQL connection string. The Sensu backend uses the golang pq library, which supports a subset of the PostgreSQL libpq connection string parameters.
required true
type String
example 
dsn: "postgresql://user:secret@host:port/dbname"
max_conn_lifetime
description Maximum time a connection can persist before being destroyed. Specify values with a numeral and a letter indicator: s to indicate seconds, m to indicate minutes, and h to indicate hours. For example, 1m, 2h, and 2h1m3s are valid.
required false
type String
example 
max_conn_lifetime: 5m
max_idle_conns
description Maximum number of number of idle connections to retain.
required false
default 2
type Integer
example 
max_idle_conns: 2
pool_size
description Maximum number of connections to hold in the PostgreSQL connection pool. We recommend 20 for most instances.
required false
default 0 (unlimited)
type Integer
example 
pool_size: 20