Learn Sensu Go

In this tutorial, you’ll download the Sensu sandbox and create a monitoring workflow with Sensu.


Set up the sandbox

1. Install Vagrant and VirtualBox

2. Download the sandbox

Download from GitHub or clone the repository:

git clone https://github.com/sensu/sandbox && cd sandbox/sensu-go

NOTE: If you’ve cloned the sandbox repository before, run cd sandbox/sensu-go and git pull https://github.com/sensu/sandbox instead.

3. Start Vagrant

ENABLE_SENSU_SANDBOX_PORT_FORWARDING=1 vagrant up

The Learn Sensu sandbox is a CentOS 7 virtual machine pre-installed with Sensu, InfluxDB, and Grafana. It’s intended for you to use as a learning tool — we do not recommend using it in a production installation. To install Sensu in production, use the installation guide instead.

The sandbox startup process takes about 5 minutes.

NOTE: The sandbox configures VirtualBox to forward TCP ports 3002 and 4002 from the sandbox virtual machine (VM) to the localhost to make it easier for you to interact with the sandbox dashboards. Dashboard links provided in this tutorial assume port forwarding from the VM to the host is active.

4. SSH into the sandbox

Thanks for waiting! To start shell into the sandbox:

vagrant ssh

You should be greeted with this prompt:

[sensu_go_sandbox]$
  • To exit out of the sandbox, press CTRL+D.
  • To erase and restart the sandbox, run vagrant destroy and then vagrant up.
  • To reset the sandbox’s Sensu configuration to the beginning of this tutorial, run vagrant provision.

NOTE: The sandbox pre-configures sensuctl with the Sensu Go admin user, so you won’t have to configure sensuctl each time you spin up the sandbox to try out a new feature. Before installing sensuctl outside of the sandbox, read the first time setup reference to learn how to configure sensuctl.


Lesson #1: Create a Sensu monitoring event

First, make sure everything is working correctly using the sensuctl command line tool. Use sensuctl to see that your Sensu backend instance has a single namespace, default, and two users: the default admin user and the user created for a Sensu agent to use.

sensuctl namespace list
  Name    
─────────
 default  
sensuctl user list
 Username       Groups       Enabled  
────────── ──────────────── ───────── 
admin      cluster-admins   true     
agent      system:agents    true    

Sensu keeps track of monitored components as entities. Start by using sensuctl to make sure Sensu hasn’t connected to any entities yet:

sensuctl entity list
 ID   Class   OS   Subscriptions   Last Seen  
──── ─────── ──── ─────────────── ─────────── 

Now you can start the Sensu agent to begin monitoring the sandbox:

sudo systemctl start sensu-agent

Use sensuctl to see that Sensu is now monitoring the sandbox entity:

sensuctl entity list
        ID          Class    OS          Subscriptions                  Last Seen            
────────────────── ─────── ─────── ───────────────────────── ─────────────────────────────── 
sensu-go-sandbox   agent   linux   entity:sensu-go-sandbox   2019-01-24 21:29:06 +0000 UTC  

Sensu agents send keepalive events to help you monitor agent status. Use sensuctl to see the keepalive events generated by the sandbox entity:

sensuctl event list
      Entity          Check                                       Output                                     Status   Silenced             Timestamp            
────────────────── ─────────── ──────────────────────────────────────────────────────────────────────────── ──────── ────────── ─────────────────────────────── 
sensu-go-sandbox   keepalive   Keepalive last sent from sensu-go-sandbox at 2019-01-24 21:29:06 +0000 UTC        0   false      2019-01-24 21:29:06 +0000 UTC 

The sensu-go-sandbox keepalive event has status 0, which means the agent is in an OK state and is able to communicate with the Sensu backend.

You can also see the event and the entity in the Sensu dashboard. Log in to the dashboard with the default admin credentials: username admin and password P@ssw0rd!.

Lesson #2: Pipe keepalive events into Slack

Now that you know the sandbox is working properly, let’s get to the fun stuff: creating a workflow. In this lesson, you’ll create a workflow that sends keepalive alerts to Slack.

NOTE: If you’d rather not create a Slack account, you can skip ahead to Lesson #3.

1. Get your Slack webhook URL

Create a Slack workspace (or use an existing workspace, if you’re already a Slack admin).

Then, visit YOUR-WORKSPACE-NAME.slack.com/services/new/incoming-webhook. Follow the steps to add the Incoming WebHooks integration and save your webhook. Your webhook channel and URL will be listed under Integration Settings — you’ll need both later in this lesson.

2. Register the Sensu Slack handler asset

Assets are shareable, reusable packages that make it easy to deploy Sensu plugins. In this lesson, we’ll use the Sensu Slack handler asset to power a slack handler.

Use sensuctl to register the Sensu Slack handler asset.

sensuctl asset create sensu-slack-handler --url "https://assets.bonsai.sensu.io/3149de09525d5e042a83edbb6eb46152b02b5a65/sensu-slack-handler_1.0.3_linux_amd64.tar.gz" --sha512 "68720865127fbc7c2fe16ca4d7bbf2a187a2df703f4b4acae1c93e8a66556e9079e1270521999b5871473e6c851f51b34097c54fdb8d18eedb7064df9019adc8"

You should see a confirmation message from sensuctl.

Created

The sensu-slack-handler asset is now ready to use with Sensu. Use sensuctl to see the complete asset definition.

sensuctl asset info sensu-slack-handler --format yaml

PRO TIP: You can use resource definitions to create and update resources (like assets) using sensuctl create --file filename.yaml. See the sensuctl docs for more information.

3. Create a Sensu Slack handler

Open the sensu-slack-handler.json handler definition provided with the sandbox in your preferred text editor. Edit the definition to include your Slack channel, webhook URL, and the sensu-slack-handler asset.

NOTE: If you aren’t sure how to open the handler and edit the definition, try these Vi/Vim gist instructions.

"env_vars": [
  "KEEPALIVE_SLACK_WEBHOOK=https://hooks.slack.com/services/AAA/BBB/CCC",
  "KEEPALIVE_SLACK_CHANNEL=#monitoring"
],
"runtime_assets": ["sensu-slack-handler"]

Now you can create a Slack handler named keepalive to process keepalive events.

sensuctl create --file sensu-slack-handler.json

Use sensuctl to see available event handlers — in this case, you’ll only see the keepalive handler you just created..

sensuctl handler list
  Name      Type   Timeout   Filters   Mutator                                                   Execute                                                                                                              Environment Variables                            Assets         
─────────── ────── ───────── ───────── ───────── ────────────────────────────────────────────────────────────────────────────────────────────────────────── ────────────────────────────────────────────────────────────────────────────────────────────────── ───────────────────── 
 keepalive   pipe         0                       RUN:  /usr/local/bin/sensu-slack-handler -c "${KEEPALIVE_SLACK_CHANNEL}" -w "${KEEPALIVE_SLACK_WEBHOOK}"   KEEPALIVE_SLACK_WEBHOOK=https://hooks.slack.com/services/AAA/BBB/CCC,KEEPALIVE_SLACK_CHANNEL=#monitoring   sensu-slack-handler  

Sensu monitoring events should begin arriving in your Slack channel, indicating that the sandbox entity is in an OK state.

4. Filter keepalive events

Now that you’re generating Slack alerts, let’s reduce the potential for alert fatigue by adding a filter that sends only warning, critical, and resolution alerts to Slack.

To accomplish this, you’ll interactively add the built-in is_incident filter to the keepalive handler, which will make sure you only receive alerts when the sandbox entity fails to send a keepalive event.

sensuctl handler update keepalive

The first prompt will be for environment variables. Just press return to continue. The second prompt is for the filters selection — enter is_incident to apply the is_incident filter.

? Filters: is_incident

For each of the mutator, timeout, type, runtime assets, and command prompts, just press return.

Use sensuctl to confirm that the keepalive handler now includes the is_incident filter:

sensuctl handler info keepalive
=== keepalive
Name:                  keepalive
Type:                  pipe
Timeout:               0
Filters:               is_incident
Mutator:               
Execute:               RUN:  sensu-slack-handler -c "${KEEPALIVE_SLACK_CHANNEL}" -w "${KEEPALIVE_SLACK_WEBHOOK}"
Environment Variables: KEEPALIVE_SLACK_WEBHOOK=https://hooks.slack.com/services/AAA/BBB/CCC, KEEPALIVE_SLACK_CHANNEL=#monitoring
Runtime Assets:        sensu-slack-handler

With the filter in place, you should no longer receive messages in your Slack channel every time the sandbox entity sends a keepalive event.

Let’s stop the agent and confirm that you receive the expected warning message.

sudo systemctl stop sensu-agent

After a couple minutes, you should see a warning message in your Slack channel informing you that the sandbox entity is no longer sending keepalive events.

Start the agent to resolve the warning.

sudo systemctl start sensu-agent

Lesson #3: Automate event production with the Sensu agent

So far, you’ve used the Sensu agent’s built-in keepalive feature, but in this lesson, you’ll create a check that automatically produces workload-related events. Instead of sending alerts to Slack, you’ll store event data with InfluxDB and visualize it with Grafana.

1. Make sure the Sensu agent is running

sudo systemctl restart sensu-agent

2. Install Nginx and the Sensu HTTP Plugin

You’ll use the Sensu HTTP Plugin to monitor an Nginx server running on the sandbox.

First, install the EPEL release package:

sudo yum install -y epel-release

Then, install and start Nginx:

sudo yum install -y nginx && sudo systemctl start nginx

Make sure it’s working:

curl -I http://localhost:80
HTTP/1.1 200 OK
...

Then install the Sensu HTTP Plugin:

sudo sensu-install -p sensu-plugins-http

You’ll use the metrics-curl.rb plugin. Test its output with:

/opt/sensu-plugins-ruby/embedded/bin/metrics-curl.rb -u "http://localhost"
...
sensu-go-sandbox.curl_timings.http_code 200 1535670975

3. Create an InfluxDB pipeline

Now, let’s create the InfluxDB pipeline to store these metrics and visualize them with Grafana. To create a pipeline to send metric events to InfluxDB, start by registering the Sensu InfluxDB handler asset.

sensuctl asset create sensu-influxdb-handler --url "https://assets.bonsai.sensu.io/b28f8719a48aa8ea80c603f97e402975a98cea47/sensu-influxdb-handler_3.1.2_linux_amd64.tar.gz" --sha512 "612c6ff9928841090c4d23bf20aaf7558e4eed8977a848cf9e2899bb13a13e7540bac2b63e324f39d9b1257bb479676bc155b24e21bf93c722b812b0f15cb3bd"

You should see a confirmation message from sensuctl.

Created

The sensu-influxdb-handler asset is now ready to use with Sensu. Use sensuctl to see the complete asset definition.

sensuctl asset info sensu-influxdb-handler --format yaml

Open the influx-handler.json handler definition provided with the sandbox, and edit the runtime_assets attribute to include the sensu-influxdb-handler asset.

"runtime_assets": ["sensu-influxdb-handler"]

Now you can use sensuctl to create the influx-db handler:

sensuctl create --file influx-handler.json

Use sensuctl to confirm that the handler was created successfully.

sensuctl handler list

The influx-db handler should be listed. If you completed lesson #2, you’ll also see the keepalive handler.

4. Create a check to monitor Nginx

The curl_timings-check.json file provided with the sandbox will create a service check that runs the metrics-curl.rb check plugin every 10 seconds on all entities with the entity:sensu-go-sandbox subscription and sends events to the InfluxDB pipeline. The metrics-curl.rb plugin is already included as the value of the command field in curl_timings-check.json &emdash; you just need to create the file:

sensuctl create --file curl_timings-check.json
sensuctl check list
     Name                                        Command                                     Interval   Cron   Timeout   TTL        Subscriptions        Handlers   Assets   Hooks   Publish?   Stdin?     Metric Format      Metric Handlers  
────────────── ──────────────────────────────────────────────────────────────────────────── ────────── ────── ───────── ───── ───────────────────────── ────────── ──────── ─────── ────────── ──────── ──────────────────── ───────────────── 
curl_timings   /opt/sensu-plugins-ruby/embedded/bin/metrics-curl.rb -u "http://localhost"         10                0     0   entity:sensu-go-sandbox                               true       false    graphite_plaintext   influx-db        

This check specifies a metrics handler and metric format. In Sensu Go, metrics are a core element of the data model: you can build pipelines to handle metrics separately from alerts. This allows you to customize your monitoring workflows to get better visibility and reduce alert fatigue.

After about 10 seconds, you can see the event produced by the entity:

sensuctl event info sensu-go-sandbox curl_timings --format json | jq .
...
  "history": [
    {
      "status": 0,
      "executed": 1556472457
    },
  ],
  "output": "sensu-go-sandbox.curl_timings.time_total 0.005 1556472657\n...",
  ...
  "output_metric_format": "graphite_plaintext",
  "output_metric_handlers": [
    "influx-db"
  ],
...

Because the check definition specified a metric format of graphite_plaintext, the Sensu agent will treat the output of the check command as Graphite-formatted metrics and translate them into a set of Sensu-formatted metrics (not shown in the output). These metrics are then sent to the InfluxDB handler, which reads Sensu-formatted metrics and converts them to a format InfluxDB accepts.

NOTE: Metric support isn’t limited to Graphite! The Sensu agent can extract metrics in multiple line protocol formats, including Nagios performance data.

5. See the HTTP response code events for Nginx in Grafana.

Log in to Grafana with username: admin and password: admin. You should see a graph of live HTTP response codes for Nginx.

Now, if you turn Nginx off, you should see the impact in Grafana:

sudo systemctl stop nginx

Start Nginx:

sudo systemctl start nginx

6. Automate disk usage monitoring for the sandbox

Now that you have an entity set up, you can add more checks. For example, let’s say you want to monitor disk usage on the sandbox.

First, install the plugin:

sudo sensu-install -p sensu-plugins-disk-checks

Test the plugin:

/opt/sensu-plugins-ruby/embedded/bin/metrics-disk-usage.rb
sensu-core-sandbox.disk_usage.root.used 2235 1534191189
sensu-core-sandbox.disk_usage.root.avail 39714 1534191189
...

Then create the check using sensuctl and the disk_usage-check.json file included with the sandbox, assigning it to the entity:sensu-go-sandbox subscription and the InfluxDB pipeline:

sensuctl create --file disk_usage-check.json

You don’t need to make any changes to disk_usage-check.json before running sensuctl create --file disk_usage-check.json.

You should see the check working in the dashboard entity view and via sensuctl:

sensuctl event list

Now, you should be able to see disk usage metrics for the sandbox in Grafana: reload your Grafana tab to show the Sensu Go Sandbox Combined.

You made it! You’re ready for the next level of Sensu-ing.

Before you move on, take a moment to remove the virtual machine and resources installed during this sandbox lesson. Press CTRL+D to exit the sandbox. Then run:

vagrant destroy

Now you can continue exploring Sensu with a clean slate. Here are some resources to help continue your journey: