TrackIt – Cloud Consulting & S/W Development
  • Home
  • Solutions
    • Media & EntertainmentMedia & Entertainment
    • HPC – High Performance ComputingHigh Performance Computing
    • Software DevSoftware Development; Serverless, CI/CD, UI/UX, API Integration
    • Migrate to AWS
    • DevOpsDevOps, AWS Cloud Consulting
    • StorageData Storage
    • Financial Services
  • Services
    • AWS Services
      • AWS Well-Architected Framework
      • Focus on Video AI/ML
      • Chime SDK
      • DevOps Guru
      • Virtual Desktop/Studio in the Cloud
      • Amazon API Gateway
      • AWS Lambda
      • Amazon DynamoDB
      • Amazon EKS
      • Amazon OpenSearch
      • Amazon CloudFront
      • AWS Elemental MediaConvert
    • Hammerspace Integration
    • Focus on Penetration Testing
    • Autodesk Service Provider
  • Resources
    • Blogs
    • White Papers
    • Case Studies
    • Press Releases
    • Trainings and Workshops – Unlock TrackIt’s Power
  • Tools & Apps
    • TrackIt Cost Management App
    • Tagbot
    • AI Video Reviewer
    • Trackflix
  • Managed Services
  • About Us
header image

TrackIt Blogs

Read the latest cloud technology news and learn new tips and tricks to get the most out of your cloud.

  • Contact Us

Categories

Automation AWS data rooms Data Storage DevOps DevTools Monitoring Optimization Services TrackIt

Tags

ami API Aurora aws AWS EC2 AWS S3 AWS Studio in the Cloud chargify Ci Cd Pipeline Circleci cloud Cloud Computing Cognito consul cost optimization dedicated Demonstration DevOps docker Dynamodb ec2 ESXi github Hammerspace hashicorp high availability HP Anyware HPA Tech Retreat 2023 HPTeradici json Marketing Media Workflows nomad open source packer pfSense Postgres public cloud reduce cost Remote Serverless terraform tutorial Video Windows Containers

Dec. 02, 2015

—

How to send Sensu events to ELK platform

logo-elastic

When we talk about log management, we talk about dealing with a large amount of computer-generated files and how to collect them, how to centralize them, how to aggregate them and how to analyze them. Logs can be produced everywhere by different types of systems. Collecting them is easy, but processing this large amout of data in order to obtain reliable information is a real challenge. Nowadays, we have tools to help us and this article describes an approach from log collecting to log analyzing, using Sensu and ELK stack.

sensu_flat_logo

Sensu is a software for monitoring servers, applications or services. Basically, you have Sensu server with a web interface and some Sensu client on your servers where you run your services. The client performs defined checks, collects metrics and sends them to the  server. With the web interface, you can see in-real-time what is happening on your services, if something goes wrong or if everything runs smoothly.

ELK stack stands for ElasticSearch, Logstash and Kibana. It is a combination of 3 powerful tools used for log processing and analytics. Logstash is used to receive and process logs, ElasticSearch is a search engine and Kibana is a data analytic front-end. These are used together to analyze huge amounts of data and extract useful information about your system.

In this article, we will explain how to combine these software to obtain a complete log analytic solution using Sensu for gathering, Logstash for processing, ElasticSearch for indexing, and Kibana for analyzing. Usually, we use Filebeat, a software provided by Elastic, in order to gather log. Why are we using Sensu instead of Filebeat you would ask? In our case, we already monitor our infrastructure with Sensu. Then on our servers, we have a Sensu client running, checking the server status and return it to the Sensu server. So, we already have a service running on each server that send informations to a master server and Sensu, with its plugin system, allow us to customize these checks. Why bother installing another service that has exactly the same function, gather informations on clients and send it to a master server? Let’s create our own Sensu plugin that gathers the log on the clients and send it to the Sensu server that forward it to Logstash.

We will use 2 servers on Ubuntu 16.04 LTS to deploy our solution: one running Sensu Server and ELK stack, the second running Sensu client. As we will discuss, this is just an example and multiple designs are possible, especially if you want it clustered.


Setting up Sensu server

First, we need to install Sensu dependencies: RabbitMQ and Redis. RabbitMQ, a messaging broker, used by Sensu services to communicate between them.

$ wget -O- http://www.rabbitmq.com/rabbitmq-signing-key-public.asc | sudo apt-key add -
OK
$ echo "deb http://www.rabbitmq.com/debian/ testing main" | sudo tee /etc/apt/sources.list.d/rabbitmq.list
deb http://www.rabbitmq.com/debian/ testing main
$ sudo apt-get update && sudo apt-get -y install rabbitmq-server
[...]

The RabbitMQ broker is now installed and has already started.
Then we need to add Sensu vhost, user, and its permission in order to use RabbitMQ to transfer message between the server and client.

$ sudo rabbitmqctl add_vhost /sensu
Creating vhost "/sensu" ...
$ sudo rabbitmqctl add_user sensu password
Creating user "sensu" ...
$ sudo rabbitmqctl set_permissions -p /sensu sensu ".*" ".*" ".*"
Setting permissions for user "sensu" in vhost "/sensu" ...

We could use RabbitMQ in the current state but the communication are not encrypted, let’s create some SSL certificate (they will be valid for 5 years).

$ sudo apt-get install openssl
$ openssl version
OpenSSL 1.0.2g  1 Mar 2016
$ cd $(mktemp -d) && wget http://sensuapp.org/docs/0.25/files/sensu_ssl_tool.tar
[...]
$ tar -xvf sensu_ssl_tool.tar
sensu_ssl_tool/
sensu_ssl_tool/sensu_ca/
sensu_ssl_tool/sensu_ca/openssl.cnf
sensu_ssl_tool/ssl_certs.sh
$ cd sensu_ssl_tool && ./ssl_certs.sh generate
[...]
$ mkdir /etc/rabbitmq/ssl && cp sensu_ca/cacert.pem server/cert.pem server/key.pem /etc/rabbitmq/ssl

Finally, let’s update our RabbitMQ configuration in order to use these certificate in its communication with Sensu.
Open `/etc/rabbitmq/rabbitmq.config` and paste this content:

[
  {rabbit, [
     {ssl_listeners, [5671]},
     {ssl_options, [{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},
                    {certfile,"/etc/rabbitmq/ssl/cert.pem"},
                    {keyfile,"/etc/rabbitmq/ssl/key.pem"},
                    {versions, ['tlsv1.2']},
                    {ciphers,  [{rsa,aes_256_cbc,sha256}]},
                    {verify,verify_none},
                    {fail_if_no_peer_cert,false}]}
   ]}
].

Rabbitmq is now ready to be used by Sensu, let’s set up Redis, a key-value database used for storing data.

$ sudo apt-get -y install redis-server
[...] 
$ redis-cli ping 
PONG

The server is now ready to be installed.

$ wget -q https://sensu.global.ssl.fastly.net/apt/pubkey.gpg -O- | sudo apt-key add -
OK 
$ echo "deb https://sensu.global.ssl.fastly.net/apt sensu main" | sudo tee /etc/apt/sources.list.d/sensu.list
$ sudo apt-get update && sudo apt-get install sensu uchiwa
[...]

The /etc/sensu/config.json contains connections information about RabbitMQ and Redis.

{
  "rabbitmq": {
    "host": "127.0.0.1",
    "ssl": {
      "cert_chain_file": "/etc/sensu/ssl/cert.pem",
      "private_key_file": "/etc/sensu/ssl/key.pem"
    },
    "port": 5671,
    "user": "sensu",
    "password": "password",
    "vhost": "/sensu"
  },
  "redis": {
    "host": "127.0.0.1",
    "port": 6379,
    "reconnect_on_error": true,
    "auto_reconnect": true
  },
  "api": {
    "host": "127.0.0.1",
    "port": 4567
  }
}

Let’s copy our SSL certificate.

$ mkdir /etc/sensu/ssl && cp /etc/rabbitmq/ssl/cert.pem /etc/rabbitmq/ssl/key.pem /etc/sensu/ssl/

Then Sensu is ready, we can start it and make it persistent on reboot:

 
$ sudo update-rc.d sensu-server defaults
$ sudo update-rc.d sensu-api defaults
$ sudo update-rc.d uchiwa defaults
$ sudo /etc/init.d/sensu-server start 
* Starting sensu-server [ OK ] 
$ sudo /etc/init.d/sensu-api start 
* Starting sensu-api [ OK ]
$ sudo /etc/init.d/uchiwa start
uchiwa started.

The server is now up and running, let’s set up the client on the second server. We will come back later on the Sensu server in order to configure the check we want to perform.

Setting up Sensu client

On your second server, you just have to perform the same operation as the Sensu server, in fact Sensu client is a part of Sensu core but it is not activated by default.

$ wget -q https://sensu.global.ssl.fastly.net/apt/pubkey.gpg -O- | sudo apt-key add - 
OK
$ echo "deb https://sensu.global.ssl.fastly.net/apt sensu main" | sudo tee /etc/apt/sources.list.d/sensu.list
deb https://sensu.global.ssl.fastly.net/apt sensu main
$ sudo apt-get update && apt-get -y install sensu
[...]

Since we use Sensu client on a remote host, we need to configure the client to send information to the server to the RabbitMQ.
First, get the SSL certificate we created previously then copy it on the /etc/sensu/ssl folder.

 
$ mkdir /etc/sensu/ssl && cp cert.pem key.pem /etc/sensu/ssl

We need to create the /etc/sensu/conf.d/rabbitmq.json on the client.

{
  "rabbitmq": {    
    "host": "rabbitmq-server_ip",
    "port": 5671,
    "ssl": {
      "cert_chain_file": "/etc/sensu/ssl/cert.pem",
      "private_key_file": "/etc/sensu/ssl/key.pem"
    },
    "vhost": "/sensu",
    "user": "sensu",
    "password": "password"
  }
}

Just replace rabbitmq-server_ip by the RabbitMQ server IP or hostname.
Then, create the client definition file, this file is very important because it is where you will describe your client and enable some checks.

{
  "client": {
    "name": "client",
    "address": "127.0.0.1",
    "subscriptions": [ "port_open" ]
  }
}

Name and address is used to describe your client, it will be show on the Uchiwa interface and will be useful to recognize the client metrics in the Elastic Stack.
I will explain subscription later when we will configure Sensu to forward metrics. This is enough for now, start the client.

$ update-rc.d sensu-client defaults
$ /etc/init.d/sensu-client start 
* Starting sensu-client [ OK ]

The part on Sensu installation is finished, you now have a Sensu client and a Sensu server running on two different servers that communicate. If you go to your browser and open http://:3000, you should see the client on the web interface. Now, we will configure Sensu to collect metrics from client, and send them to the Elastic Stack. Then first, we need to install and configure Elastic Stack on the server.

Setting up Logstash

Before installing the Elastic Stack, we will need to install Oracle JDK 8. It is not compactible with Java 9 yet.

$ sudo apt-get install software-properties-common
[...]
$ sudo add-apt-repository ppa:webupd8team/java
[...]
$ sudo apt-get update
[...]
$ sudo apt-get install oracle-java8-installer
[...]
$ java -version
java version "1.8.0_101"
Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)

Let’s install Logstash.

$ wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
OK
$ echo "deb https://packages.elastic.co/logstash/2.4/debian stable main" | sudo tee -a /etc/apt/sources.list
deb https://packages.elastic.co/logstash/2.4/debian stable main
$ sudo apt-get update && sudo apt-get install logstash
[...]
$ sudo update-rc.d logstash defaults
$ sudo service logstash start

We will configure it later, now we will setup ElasticSearch.

Setting up ElasticSearch

Next we will install Elastic Search on the server, using APT repository. First, we need to install Java 1.8, then we download and install ElasticSearch 2.0 and set it up as a service.

$ curl -L -O https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/deb/elasticsearch/2.4.1/elasticsearch-2.4.1.deb
$ sudo dpkg -i elasticsearch-2.4.1.deb
$ sudo update-rc.d elasticsearch defaults
$ sudo /etc/init.d/elasticsearch start
[ ok ] Starting elasticsearch (via systemctl): elasticsearch.service.
$ curl 'localhost:9200' 
{
  "name" : "Angel Salvadore",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "TYVW3XWRTOywkyF6rAa-VQ",
  "version" : {
    "number" : "2.4.1",
    "build_hash" : "c67dc32e24162035d18d6fe1e952c4cbcbe79d16",
    "build_timestamp" : "2016-09-27T18:57:55Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.2"
  },
  "tagline" : "You Know, for Search"
}

ElasticSearch is now ready to index our logs, let’s set up Kibana.

Setting up Kibana

$ curl -L -O https://download.elastic.co/kibana/kibana/kibana-4.6.1-amd64.deb
[...]
$ sudo dpkg -i kibana-4.6.1-amd64.deb
[...]
$ sudo update-rc.d kibana defaults
$ sudo /etc/init.d/kibana start
kibana started

Then, go check http://localhost:5601 to see Kibana working.

Now, the ELK stash is ready to be used! We are all set with the installation, we will now work on the most interesting part of this tutorial, how to setup Sensu in order to forward metrics to Logstash.
We will go back to Sensu, we will need to configure some checks in order to get metrics that we will forward to Elastic Stack.

Configure Sensu to forward logs to Logstash

First, get the Sensu plugins which get the metrics you want. For this article, I will use the netstat plugins. You can find a lot of different check plugins in the Sensu Community Github page. Here, we will try to get some stats about our server using the check-ports.rb script. Go on your client and run these commands.

$ sudo apt-get install ruby-dev build-essential
[...]
$ sudo gem install sensu-plugins-disk-checks
[...]
$ metrics-sockstat.rb
network.sockets.total_used 121 1476689348
network.sockets.TCP.inuse 17 1476689348
network.sockets.TCP.orphan 0 1476689348
network.sockets.TCP.tw 43 1476689348
network.sockets.TCP.alloc 25 1476689348
network.sockets.TCP.mem 2 1476689348
network.sockets.UDP.inuse 2 1476689348
network.sockets.UDP.mem 1 1476689348
network.sockets.UDPLITE.inuse 0 1476689348
network.sockets.RAW.inuse 0 1476689348
network.sockets.FRAG.inuse 0 1476689348
network.sockets.FRAG.memory 0 1476689348

The next step is to register this check, create the file /etc/sensu/conf.d/check.json on Sensu client and write the description of your check.

{
    "checks": {
        "check_ssh": {
            "type": "metric",
            "command": "check-ports.rb -h 127.0.0.1 -p 22 -t 30"
            "interval": 60,
            "subscribers": ["port_open"],
            "handlers": ["logstash"]
        },
        "check_dns": {
            "type": "metric",
            "command": "check-ports.rb -h 127.0.0.1 -p 53 -t 30"
            "interval": 60,
            "subscribers": ["port_open"],
            "handlers": ["logstash"]
        },
        "check_http": {
            "type": "metric",
            "command": "check-ports.rb -h 127.0.0.1 -p 80 -t 30"
            "interval": 60,
            "subscribers": ["port_open"],
            "handlers": ["logstash"]
        },
        "check_https": {
            "type": "metric",
            "command": "check-ports.rb -h 127.0.0.1 -p 443 -t 30"
            "interval": 60,
            "subscribers": ["port_open"],
            "handlers": ["logstash"]
        }
    }
}

This file describes each check to the client. It is pretty easy to understand what each field means. Except maybe subscribers and handlers.
Subscribers field contains some tags given to a group of check. On the client, you describe the checks on the /etc/sensu/conf.d/check.json file, give them different tags, then put the same tag on /etc/sensu/conf.d/client.json file and the Sensu clients will perform these checks.
Handlers field is used to choose the handler script that will process the metrics. This is the mecanism we will use to tell Sensu to forward the log to Logstash.

How, your Uchiwa UI should look like this.

sensu

Create the file /etc/sensu/conf.d/handlers.json on Sensu server and add the content below.

{
  "handlers": {
    "logstash": {
      "type": "udp",
      "socket": {
        "host": "127.0.0.1",
        "port": 5514
      }
    }
  }
}

As describe in this configuration file, we will send each metric gathered by Sensu server to Logstash using udp connexion on port 5514.

The server configuration and it should be ready. Logstash needs to be configure in order to get the metrics and insert them in ElasticSearch.
Copy this configuration file in /etc/logstash/conf.d/00-sensu-metrics.conf. This file describes how Logstash should listen to in order to get logs.

input {
    udp {
        port => 5514
        codec => "json"
        type => "rsyslog"
    }
}

Restart Sensu client, Sensu server and Logstash, open Kibana on your Internet browser then you should see some metrics appear on Kibana.

You should have more metrics on Kibana after setting up Sensu

This is now the end of this article. We have seen how to install ELK and Sensu, how to get metrics using Sensu checks and how to forward it as log to ElasticSearch.
In the next article, based on what we did here, we will use this to get logs from a service and try to get informations from it. We will see how to create our own Sensu check in order to get metrics from Media & Entertainment solution Harmonic WFS, then configure Logstash to process it with Grok and finally, we will draw some charts with these data in Kibana. See you next time !

Previous Next

AWS Advanced Consulting Partner - Cloud Management, Consulting, and Software Development Solutions

Quick links

    • Home
    • Solutions
    • Services
    • White Papers
    • Blog
    • About Us

Tools & Apps

    • TrackIt AI Video Reviewer
    • Trackflix
    • TrackIt App
    • Tagbot

get in touch

    • Phone: +1 323-230-0042
    • Email: info@trackit.io

    • TrackIt 578 Washington Blvd #260 Marina Del Rey, CA 90292
  • @TrackIt 2022