- Hardware sizing
- Communications security
- Common Sensu architectures
This guide describes various deployment considerations and recommendations for a production-ready Sensu deployment, including details related to communication security and common deployment architectures.
etcd is a key-value store that is used by applications of varying complexity, from simple web apps to Kubernetes. The Sensu backend uses an embedded etcd instance for storing both configuration and event data, so you can get Sensu up and running without external dependencies.
By building atop etcd, Sensu’s backend inherits a number of characteristics to consider when you’re planning for a Sensu deployment.
Because etcd’s design prioritizes consistency across a cluster, the speed with which write operations can be completed is very important to the performance of a Sensu cluster. This means that you should provision Sensu backend infrastructure to provide sustained IO operations per second (IOPS) appropriate for the rate of monitoring events the system will be required to process.
Our hardware requirements documentation describes the minimum and recommended hardware specifications for running the Sensu backend.
Whether you’re using using a single Sensu backend or multiple Sensu backends in a cluster, communication with the backend’s various network ports (web UI, HTTP API, WebSocket API, etcd client and peer) occurs in cleartext by default. We recommend that you encrypt network communications via TLS, which requires planning and explicit configuration.
Planning TLS for etcd
The URLs for each member of an etcd cluster are persisted to the database after initialization. As a result, moving a cluster from cleartext to encrypted communications requires resetting the cluster, which destroys all configuration and event data in the database. Therefore, we recommend planning for encryption before initiating a clustered Sensu backend deployment.
WARNING: Reconfiguring a Sensu cluster for TLS post-deployment will require resetting all etcd cluster members, resulting in the loss of all data.
As described in Secure Sensu, the backend uses a shared certificate and key for web UI and agent communications. You can secure communications with etcd using the same certificate and key. The certificate’s common name or subject alternate names must include the network interfaces and DNS names that will point to those systems.
Common Sensu architectures
Depending on your infrastructure and the type of environments you’ll be monitoring, you may use one or a combination of these architectures to best fit your needs.
Single backend with embedded etcd
The single backend (standalone) with embedded etcd architecture requires minimal resources but provides no redundancy in the event of failure.
Sensu single backend (standalone) architecture with embedded etcd
You can reconfigure a single backend as a member of a cluster, but this operation is destructive: it requires destroying the existing database.
The simplicity of the single backend (standalone) architecture may make it a good fit for small- to medium-sized deployments (such as monitoring a remote office or datacenter), deploying alongside individual auto-scaling groups, or in various segments of a logical environment spanning multiple cloud providers.
For example, in environments with unreliable WAN connectivity, having agents connect to a local backend may be more reliable than having agents connect over WAN or VPN tunnel to a backend running in a central location.
NOTE: Multiple Sensu backends can relay their events to a central backend using the sensu-relay-handler.
Clustered backend with embedded etcd
You can join the embedded etcd databases of multiple Sensu backend instances together in a cluster to provide increased availability and replication of both configuration and data. Read Run a Sensu cluster for more information.
Sensu clustered architecture with embedded etcd
Clustering requires an odd number of backend instances. Although larger clusters provide better fault tolerance, write performance suffers because data must be replicated across more machines. The etcd maintainers recommend clusters of 3, 5 or 7 backends. See the etcd documentation for more information.
Scale cluster performance with PostgreSQL
To achieve the high rate of event processing that many enterprises require, Sensu offers support for PostgreSQL event storage as a commmercial feature. See the Datastore reference for details on configuring the Sensu backend to use PostgreSQL for event storage.
Sensu clustered architecture with embedded etcd and PostgreSQL event storage
In load testing, Sensu Go has proven capable of processing 36,000 events per second when using PostgreSQL as the event store. See the sensu-perf project repository for a detailed explanation of our testing methodology and results.
Create and maintain clusters
Sensu’s embedded etcd supports initial cluster creation via a static list of peer URLs. After you create a cluster, you can add and remove cluster members with etcdctl tooling. See Run a Sensu cluster and the etcd documentation for more information.
Clustered deployments benefit from a fast and reliable network. Ideally, they should be co-located in the same network segment with as little latency as possible between all the nodes. We do not recommend clustering backends across disparate subnets or WAN connections.
Although 1GbE is sufficient for common deployments, larger deployments will benefit from 10GbE, which allows a shorder mean time to recovery.
As the number of agents connected to a backend cluster grows, so will the amount of communication between members of the cluster required for data replication. With this in mind, clusters with a thousand or more agents should use a discrete network interface for peer communication.
Although you can configure each Sensu agent with the URLs for multiple backend instances, we recommend that you configure agents to connect to a load balancer. This approach gives operators more control over agent connection distribution and makes it possible to replace members of the backend cluster without updates to agent configuration.
Conversely, you cannot configure the sensuctl command line tool with multiple backend URLs. Under normal conditions, sensuctl communications and browser access to the web UI should be routed via a load balancer.