TrackIt Blogs

Sensu event handlers are for taking action on events (produced by check results), such as sending an email alert, or storing metrics in InfluxDB. There are several types of handlers: pipe, tcp, udp, transport, and set.

TCP and UDP handlers send the event data to a remote socket. It is advisable to use UDP handler when you have several dozen hosts to monitor. With the UDP protocol, the load is practically non-existent on the Sensu server. This is what we are going to use in this article.

So we are going to see how to send metrics from Sensu into InfluxDB. To do that the Sensu handler will send all the data collected by Sensu into the InfluxDB database.

InfluxDB Prerequisite

For those who haven’t heard about it yet, InfluxDB is a time series, metrics, and analytics database. Time series databases are designed to address the problem of storing data resulting from successive measurements made over a period of time. It’s designed to support horizontal as well as vertical scaling and, best of all, it’s not written in Java — it’s written in Go.

To manage InfluxDB, we have two choices :
– Web Admin Interface (port 8083)
– Command Line Interface [API] (port 8086)

Like any good DevOps, we are going to do the job via CLI :), but you could connect to the web UI to see how it looks: http://influxdb-host-name:8083

So to start CLI, just execute the influx command on the InfluxDB server. It should look like this :

$ influx
Connected to http://localhost:8086 version 0.9
InfluxDB shell 0.9
>

For interacting with InfluxDB, the query language used is InfluxQL. It’s a SQL-like query language, so for those who are familiar with SQL, it should be easy.

When InfluxDB is first set up, it does not contain any actual databases beside the internal one, so we need to create a database which we will ultimately use to store our metrics.

To do that use the CREATE DATABASE <db-name> InfluxQL statement, where <db-name> is the name of the database you wish to create.

> CREATE DATABASE sensu
>

Then, we need to enable and configure the UDP section in the InfluxDB’s configuration file. Also, in order to have better performance and to avoid dropping metrics, there is some lines to uncomment and this will allow InfluxDB to enable its buffer. See below the UDP section :

[[udp]]
  enabled = true
   bind-address = ":8090"
   database = "sensu"
  # retention-policy = ""

  # These next lines control how batching works. You should have this enabled
  # otherwise you could get dropped metrics or poor performance. Batching
  # will buffer points in memory if you have many coming in.

   batch-size = 1000 # will flush if this many points get buffered
   batch-pending = 5 # number of batches that may be pending in memory
   batch-timeout = "1s" # will flush at least this often even if we haven't hit buffer limit
   read-buffer = 0 # UDP Read buffer size, 0 means OS default. UDP listener will fail if set above OS max.

Finally, restart the InfluxDB service and we should be ready to receive metrics.
So now let’s configure the Sensu handler.

Sensu Configuration

Requirements

The influxdb rubygem needs to be installed in Sensu’s Ruby environment in order to handle writing the data. Note that Sensu vendors their own embedded ruby, so having it available in the system Ruby won’t work.
E.g. /opt/sensu/embedded/bin/gem install influxdb

There is only one new file to create in your Sensu configuration directory, and this is the handler script :

/etc/sensu/extensions/influxdb_line_protocol.rb

#!/usr/bin/env ruby

require "sensu/extension"

module Sensu
    module Extension
        class InfluxDBLineProtocol < Mutator
            def name
                "influxdb_line_protocol"
            end

            def description
                "returns check output formatted for InfluxDB's line protocol"
            end

            def run(event)
                host = event[:client][:name]
                ip = event[:client][:address]
                metric = event[:check][:name]
                output = event[:check][:output]
            
                data = []
                output.split("\n").each do |result|
                    m = result.split
                    next unless m.count == 3
                    key = m[0].split('.', 2)[1]
                    key.gsub!('.', '_')
                    value = m[1].to_f
                    time = m[2].ljust(19, '0')
                    data << "#{key},host=#{host},ip=#{ip},metric=#{metric} value=#{value} #{time}"
                end

                yield data.join("\n"), 0
            end
        end
    end
end

What this script will do is collect all the data from Sensu output checks, parse them, and inject them into the InfluxDB database.

Next we have to define the above handler by editing the following JSON file :

/etc/sensu/conf.d/handlers.json

{
    "handlers": {
        "default": {
            "type": "set",
            "handlers": ["logstash", "influxdb"]
        },
        "logstash": {
            "type": "pipe",
            "command": "/etc/sensu/handlers/notification/logstash.rb"
        },
        "influxdb": {
            "type": "udp",
            "mutator": "influxdb_line_protocol",
            "socket": {
                       "host": "dockercompose_influxdb_1",
                       "port": 8090
            }
        }
}

Note : To fill this file, you can refer to the Sensu documentation : https://sensuapp.org/docs/latest/handlers

As you can see, I have another handler already defined (logstash). By doing that, I can define a set of handlers, it’s a way to send the same event data to multiple backends.

And finally, configure checks with the InfluxDB handler. In my case all my checks are defined with the handler ‘default’, so that all my checks results are sent to Logstash AND InfluxDB.

{
  "checks": {
    "load_metrics": {
       "type": "metric",
       "command": "/opt/sensu/embedded/bin/ruby /etc/sensu/plugins/linux/load-metrics.rb --scheme `grep name /etc/sensu/conf.d/client.json | cut -d\\\" -f4`",
       "interval": 60,
       "subscribers": ["linux"],
       "handlers": ["default"]
    },
...

Now restart sensu-server and you should see several metrics in the sensu database. To see the data, run the following query on the sensu database via the web admin interface :

select * from /.*/ order by time desc limit 3

Normally, nothing should appears in the sensu log. But to track the event sent to InfluxDB, you can launch this command from the InfluxDB server :

nc -u -l 8090

Display Data on Grafana

Set InfluxDB as Datasource

In this step, we will add our InfluxDB database as a source in Grafana.

To add the data source, open the side menu by clicking the Grafana icon in the top header. In the side menu, click Data Sources. Click on the Add New link in the top header to bring up the data source definition screen.

Populate this screen using your settings and test the connection to your database.

Build a Dashboard

Finally, you are ready to build your own dashboard with beautiful graphs.

Conclusion

In this article, we have seen how to inject data via a Sensu handler into an InfluxDB database, collect by Sensu engine and display this data in pretty graphs in Grafana.

We have proved that the stack of these 3 systems — Sensu, InfluxDB and Grafana — work together correctly and are ready to be put into a production environment.