Use secrets management in Sensu

COMMERCIAL FEATURE: Access the Env and VaultProvider secrets provider datatypes in the packaged Sensu Go distribution. For more information, see Get started with commercial features.

Sensu’s secrets management allows you to avoid exposing secrets like usernames, passwords, and access keys in your Sensu configuration. In this guide, you’ll learn how to use Sensu’s built-in secrets provider, Env, or HashiCorp Vault as your external secrets provider and authenticate without exposing your secrets. You’ll set up your PagerDuty Integration Key as a secret and create a PagerDuty handler definition that requires the secret. Your Sensu backend can then execute the handler with any check.

To follow this guide, you’ll need to install the Sensu backend, have at least one Sensu agent running, and install and configure sensuctl.

Secrets are configured via secrets resources. A secret resource definition refers to the secrets provider (Env or VaultProvider) and an ID (the named secret to fetch from the secrets provider).

This guide only covers the handler use case, but you can use secrets management in handler, mutator, and check execution. When a check configuration references a secret, the Sensu backend will only transmit the check’s execution requests to agents that are connected via mutually authenticated transport layer security (mTLS)-encrypted websockets. Read more about enabling mTLS.

The secret included in your Sensu handler will be exposed to Sensu services at runtime as an environment variable. Sensu only exposes secrets to Sensu services like environment variables and automatically redacts secrets from all logs, the API, and the web UI.

Retrieve your PagerDuty Integration Key

The example in this guide uses the PagerDuty Integration Key as a secret and a PagerDuty handler definition that requires the secret.

Here’s how to find your Integration Key in PagerDuty so you can set it up as your secret:

  1. Log in to your PagerDuty account.
  2. In the Configuration drop-down menu, select Services.
  3. Click your Sensu service.
  4. Click the Integrations tab. The Integration Key is listed in the second column.
PagerDuty Integration Key location

PagerDuty Integration Key location

Make a note of your Integration Key — you’ll need it to create your backend environment variable or HashiCorp Vault secret.

Use Env for secrets management

The Sensu Go commercial distribution includes a built-in secrets provider, Env, that exposes secrets from environment variables on your Sensu backend nodes. The Env secrets provider is automatically created with an empty spec when you start your Sensu backend.

Create your backend environment variable

To use the built-in Env secrets provider, you will add your secret as a backend environment variable.

First, make sure you have created the files you need to store backend environment variables.

Then, run the following code, replacing INTEGRATION_KEY with your PagerDuty Integration Key:

$ echo 'SENSU_PAGERDUTY_KEY=INTEGRATION_KEY' | sudo tee -a /etc/default/sensu-backend
$ sudo systemctl restart sensu-backend
$ echo 'SENSU_PAGERDUTY_KEY=INTEGRATION_KEY' | sudo tee -a /etc/sysconfig/sensu-backend
$ sudo systemctl restart sensu-backend

This configures the SENSU_PAGERDUTY_KEY environment variable to your PagerDuty Integration Key in the context of the sensu-backend process.

Create your Env secret

Now you’ll use sensuctl create to create your secret. This code creates a secret named pagerduty_key that refers to the environment variable ID SENSU_PAGERDUTY_KEY. Run:

cat << EOF | sensuctl create
---
type: Secret
api_version: secrets/v1
metadata:
  name: pagerduty_key
  namespace: default
spec:
  id: SENSU_PAGERDUTY_KEY
  provider: env
EOF

You can securely pass your PagerDuty Integration Key in Sensu checks, handlers, and mutators by referring to the pagerduty_key secret. Skip to the add a handler section, where you’ll use your pagerduty_key secret in your handler definition.

Use HashiCorp Vault for secrets management

This section explains how to use HashiCorp Vault as your external secrets provider to authenticate via the HashiCorp Vault integration’s token auth method or TLS certificate auth method.

NOTE: You must set up HashiCorp Vault to use VaultProvider secrets management in production. The examples in this guide use the Vault dev server, which is useful for learning and experimenting. The Vault dev server gives you access to a preconfigured, running Vault server with in-memory storage that you can use right away. Follow the HashiCorp Learn curriculum when you are ready to set up a production server in Vault.

In addition, this guide uses the Vault KV secrets engine. Using the Vault KV secrets engine with the Vault dev server requires v2 connections. For this reason, in the VaultProvider spec in these examples, the client version value is v2.

Configure your Vault authentication method (token or TLS)

If you use HashiCorp Vault as your external secrets provider, you can authenticate via the HashiCorp Vault integration’s token or transport layer security (TLS) certificate authentication method.

Vault token authentication

Follow the steps in this section to use HashiCorp Vault as your external secrets provider to authenticate with the HashiCorp Vault integration’s token auth method.

Retrieve your Vault root token

NOTE: The examples in this guide use the Root Token for the the Vault dev server, which gives you access to a preconfigured, running Vault server with in-memory storage that you can use right away. Follow the HashiCorp Learn curriculum when you are ready to set up a production server in Vault.

To retrieve your Vault root token:

  1. Download and install the Vault edition for your operating system.
  2. Open a terminal window and run vault server -dev.

The command output includes a Root Token line. Find this line in your command output and copy the Root Token value. You will use it next to create your Vault secrets provider.

HashiCorp Vault Root Token location

HashiCorp Vault Root Token location

Leave the Vault dev server running. Because you aren’t using TLS, you will need to set VAULT_ADDR=http://127.0.0.1:8200 in your shell environment.

Create your Vault secrets provider

NOTE: In Vault’s dev server, TLS is not enabled, so you won’t be able to use certificate-based authentication.

Use sensuctl create to create your secrets provider, vault. In the code below, replace ROOT_TOKEN with the Root Token value for your Vault dev server. Then, run:

cat << EOF | sensuctl create
---
type: VaultProvider
api_version: secrets/v1
metadata:
  name: vault
spec:
  client:
    address: http://localhost:8200
    token: ROOT_TOKEN
    version: v2
    tls: null
    max_retries: 2
    timeout: 20s
    rate_limiter:
      limit: 10
      burst: 100
EOF

To continue, skip ahead to create your Vault secret.

Vault TLS certificate authentication

This section explains how use HashiCorp Vault as your external secrets provider to authenticate with the HashiCorp Vault integration’s TLS certificate auth method.

NOTE: You will need to set up HashiCorp Vault in production to use TLS certificate-based authentication. In Vault’s dev server, TLS is not enabled. Follow the HashiCorp Learn curriculum when you are ready to set up a production server in Vault.

First, in your Vault, enable and configure certificate authentication. For example, your Vault might be configured for certificate authentication like this:

vault write auth/cert/certs/sensu-backend \
    display_name=sensu-backend \
    policies=sensu-backend-policy \
    certificate=@sensu-backend-vault.pem \
    ttl=3600

Second, configure your VaultProvider in Sensu:

---
type: VaultProvider
api_version: secrets/v1
metadata:
  name: vault
spec:
  client:
    address: https://vault.example.com:8200
    version: v2
    tls:
      ca_cert: /path/to/your/ca.pem
      client_cert: /etc/sensu/ssl/sensu-backend-vault.pem
      client_key: /etc/sensu/ssl/sensu-backend-vault-key.pem
      cname: sensu-backend.example.com
    max_retries: 2
    timeout: 20s
    rate_limiter:
      limit: 10
      burst: 100

The certificate you specify for tls.client_cert should be the same certificate you configured in your Vault for certificate authentication.

Next, create your Vault secret.

Create your Vault secret

First, retrieve your PagerDuty Integration Key (the secret you will set up in Vault).

Next, open a new terminal and run vault kv put secret/pagerduty key=INTEGRATION_KEY. Replace INTEGRATION_KEY with your PagerDuty Integration Key. This writes your secret into Vault.

In this example, the name of the secret is pagerduty. The pagerduty secret contains a key, and you specified that the key value is your PagerDuty Integration Key. The id value for your secret will be secret/pagerduty#key.

NOTE: The id value for secrets that target a HashiCorp Vault must start with the name of the secret’s path in Vault. The Vault dev server is preconfigured with the secret keyspace already set up, and Sensu requires the secret/ path for the id value.

Run vault kv get secret/pagerduty to see the secret you just set up.

Use sensuctl create to create your vault secret:

cat << EOF | sensuctl create
---
type: Secret
api_version: secrets/v1
metadata:
  name: pagerduty_key
  namespace: default
spec:
  id: secret/pagerduty#key
  provider: vault
EOF

Now you can securely pass your PagerDuty Integration Key in the handlers, and mutators by referring to the pagerduty_key secret. In the add a handler section, you’ll use your pagerduty_key secret in your handler definition.

Add a handler

Register the PagerDuty Handler dynamic runtime asset

To begin, register the Sensu PagerDuty Handler dynamic runtime asset with sensuctl asset add:

sensuctl asset add sensu/sensu-pagerduty-handler:1.2.0 -r pagerduty-handler

This example uses the -r (rename) flag to specify a shorter name for the dynamic runtime asset: pagerduty-handler.

NOTE: You can adjust the dynamic runtime asset definition according to your Sensu configuration if needed.

Run sensuctl asset list --format yaml to confirm that the dynamic runtime asset is ready to use.

With this handler, Sensu can trigger and resolve PagerDuty incidents. However, you still need to add your secret to the handler spec so that it requires your backend to request secrets from your secrets provider.

Add your secret to the handler spec

To create a handler definition that uses your pagerduty_key secret, run:

cat << EOF | sensuctl create
---
api_version: core/v2
type: Handler
metadata:
  namespace: default
  name: pagerduty
spec:
  type: pipe
  command: pagerduty-handler --token $PD_TOKEN
  secrets:
  - name: PD_TOKEN
    secret: pagerduty_key
  runtime_assets:
  - pagerduty-handler
  timeout: 10
  filters:
  - is_incident
EOF

Now that your handler is set up and Sensu can create incidents in PagerDuty, you can automate this workflow by adding your pagerduty handler to your Sensu service check definitions. See Monitor server resources to learn more.

Next steps

Read the secrets or secrets providers reference for in-depth secrets management documentation.