TrackIt – Cloud Consulting & S/W Development
  • Home
  • Solutions
    • Media & EntertainmentMedia & Entertainment
    • HPC – High Performance ComputingHigh Performance Computing
    • Software DevSoftware Development; Serverless, CI/CD, UI/UX, API Integration
    • Migrate to AWS
    • DevOpsDevOps, AWS Cloud Consulting
    • StorageData Storage
    • Financial Services
  • Services
    • AWS Services
      • AWS Well-Architected Framework
      • Focus on Video AI/ML
      • Chime SDK
      • DevOps Guru
      • Virtual Desktop/Studio in the Cloud
      • Amazon API Gateway
      • AWS Lambda
      • Amazon DynamoDB
      • Amazon EKS
      • Amazon OpenSearch
      • Amazon CloudFront
      • AWS Elemental MediaConvert
    • Hammerspace Integration
    • Focus on Penetration Testing
    • Autodesk Service Provider
  • Resources
    • Blogs
    • White Papers
    • Case Studies
    • Press Releases
    • Trainings and Workshops – Unlock TrackIt’s Power
  • Tools & Apps
    • TrackIt Cost Management App
    • Tagbot
    • AI Video Reviewer
    • Trackflix
  • Managed Services
  • About Us
header image

TrackIt Blogs

Read the latest cloud technology news and learn new tips and tricks to get the most out of your cloud.

  • Contact Us

Categories

Automation AWS data rooms Data Storage DevOps DevTools Monitoring Optimization Services TrackIt

Tags

ami API Aurora aws AWS EC2 AWS S3 AWS Studio in the Cloud chargify Ci Cd Pipeline Circleci cloud Cloud Computing Cognito consul cost optimization dedicated Demonstration DevOps docker Dynamodb ec2 ESXi github Hammerspace hashicorp high availability HP Anyware HPA Tech Retreat 2023 HPTeradici json Marketing Media Workflows nomad open source packer pfSense Postgres public cloud reduce cost Remote Serverless terraform tutorial Video Windows Containers

Jun. 03, 2020

—

Monitor AWS Infrastructure Health with Elasticsearch & Kibana 2020

Elasticsearch is an open-source, distributed search and analytics engine that is commonly used for log analytics, full-text search, and operational intelligence. Kibana is a free open-source data visualization tool that provides a tight integration with Elasticsearch and is its default choice for visualization.

Read more: Elastisearch & Kibana

 

Monitor the health of your application infrastructure with Elasticsearch & Kibana

 kibana et elasticsearch - logo kibana

Elasticsearch is an open-source, distributed search and analytics engine that is commonly used for log analytics, full-text search, and operational intelligence. Kibana is a free open-source data visualization tool that provides a tight integration with Elasticsearch and is the default choice for visualizing data stored in the latter. 

How They Work Together  

Data is sent to Elasticsearch in the form of JSON files via the Elasticsearch API or other ingestion tools such as Logstash or Amazon Kinesis Firehose. Elasticsearch then proceeds to store the document and adds a searchable reference to the document in the cluster’s index which can be retrieved using the Elasticsearch API. This data stored in Elasticsearch can be used to easily set up dashboards and reports with Kibana to gain access to analytics and additional operational insights. 

“The ability to make sense out of data is no longer simply a competitive advantage for enterprises, it has become an absolute necessity for any company in an increasingly complex and statistics-driven world. The visualizations provided by Kibana on Elasticsearch data can quickly provide deep business insight.” 

— Brad Winett, TrackIt President 

Helping ElephantDrive Take Advantage of Kibana Dashboards to Better Monitor their APIs 

ElephantDrive is a leading service supplier that provides individuals and businesses simple but powerful tools for protecting and accessing their data. With ElephantDrive, ordinary people enjoy the peace of mind that comes from the type of enterprise-class backup, storage, and data management that has historically only been available to big corporations. 

ElephantDrive wanted to improve its ability to store, analyze, and visualize log information, so they set up a basic ELK (Elasticsearch, Logstash, Kibana) stack. The initial Kibana implementation was in place but without any of the dashboards that make it such a valuable tool, so ElephantDrive approached the TrackIt team and asked us to analyze ElephantDrives’s current Elasticsearch logs to recommend dashboards that could be set up to allow for better log monitoring. Two were created for this specific purpose: 

  1. A ‘data.operation’ dashboard that displays the distribution of requests by operation in a pie chart 
  1. A ‘data.apiKey’ dashboard that displays the average response time per API key 

“We were able to get the basic stack up quickly, but wanted to turn the data into actionable information — the Track It team not only helped us leverage the power of Kibana’s visualizations, but also provided the education, documentation, and tools for us to take the next steps on our own” 

— Michael Fisher, ElephantDrive CEO and Co-Founder 

The following is a thorough tutorial that will first walk the reader through the general process of setting up dashboards using Elasticsearch and Kibana before illustrating the steps we took to set up these two dashboards for ElephantDrive. 

Accessing Elasticsearch & Kibana 

Communication with Elasticsearch is done via HTTP requests. We have used Postman in this example, which provides us with a more graphical interface to make requests. To access Elasticsearch, you can make requests in the following way using a curl in a shell script: 

curl -v “http://ec2-XXX-XX-X-XX.compute-1.amazonaws.com:9200/_cat/indices?v” 

To access Kibana, load this URL in your browser : 

http://ec2-XXX-XX-X-XX.compute-1.amazonaws.com:5601 

Logstash Ingestion Issue & How To Fix It 

ElephantDrive had an issue with their Logstash. Under some rare circumstances, the Logstash ingestion was failing and the following error message was received: 

[2020–03–04T22:34:52,349][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>[“index”, {:_id=>nil, :_index=>”logstash-2020.03.04", :_type=>”doc”, :_routing=>nil}, #<LogStash::Event:0x16a5ee83>], :response=>{“index”=>{“_index”=>”logstash-2020.03.04", “_type”=>”doc”, “_id”=>”AXCnr_f9Ski653_WeeEo”, “status”=>400, “error”=>{“type”=>”mapper_parsing_exception”, “reason”=>”failed to parse [data]”, “caused_by”=>{“type”=>”illegal_state_exception”, “reason”=>”Can’t get text on a START_OBJECT at 1:171"}}}}}

This error was thought to be coming from a malformed log entry arriving at the exact moment a new Elasticsearch index is created. This would happen if the malformed log entry is the first one sent to Logstash on a new day since Logstash creates a new index each day. 

Since the Elasticsearch mapping is dynamically created from the message parsed by Logstash, a malformed message will put a wrong mapping in the index, which will, in turn, stop the correct message from being ingested. 

Fixing the Logstash Ingestion Issue 

If you are facing a similar issue, the first step to take is to shut down Logstash. Once Logstash is shut down, you need to delete the incriminated index. The index name can be found in the Logstash log (and is typically “logstash-YYYY.MM.DD”). 

Open Kibana, and go to “Dev Tools”: 

 Open Kibana to start  

Delete the index with the following command (using the name of the incriminated index) : 

 Capture d’écran 2020 06 16 à 09.15.35

You can then restart Logstash and let it ingest new log entries. 

Once Logstash has recreated the index, you will also need to refresh the field list to get the correct mapping (under Management > Index Pattern > logstash-*) : 

Capture d’écran 2020 06 16 à 09.16.16 

 

Getting Started With Kibana 

How To Create A Dashboard with Kibana — Pie Chart Example 

In this initial example, we will walk you through the process of creating a pie chart dashboard that shows the most performed queries. 

  1. Go to the “Visualize” tab and click the “Create new visualization” button 

 Capture d’écran 2020 06 16 à 09.20.39 

  1. Select “Compare parts of a whole” (Pie chart)

 Capture d’écran 2020 06 16 à 09.21.06

  1. The data required to create this dashboard is located in the “logstash” database. 

Capture d’écran 2020 06 16 à 09.22.48  

  1. Select “Split Slices”

  Capture d’écran 2020 06 16 à 09.26.28

  1. Under “Aggregation”, choose “Terms”

 Capture d’écran 2020 06 16 à 09.26.56

  1. Select “data.operation.keyword”

 Capture d’écran 2020 06 16 à 09.27.19 

  1. Choose how you would like to order the dataand alsothe number of slices with “Size” 

 Capture d’écran 2020 06 16 à 09.27.40

  1. Click on “Apply changes”

 Capture d’écran 2020 06 16 à 09.28.16 

This is what the result looks like: 

 Results after completion of steps 1-8  

Creating the ‘Average response time per API Key’ Dashboard 

 

The first dashboard we created for ElephantDrive is a data table that displays the average response time per API key. The steps to implement this dashboard are as follows: 

  1. Go to the “Visualize” tab and click the “Create new visualization” button 

 Capture d’écran 2020 06 16 à 09.29.27

  1. For this type of data, choosing “Data Table” is a relevant choice.

 Capture d’écran 2020 06 16 à 09.29.52 

  1. The data required to create this dashboardis located inthe “logstash” database. 

 Capture d’écran 2020 06 16 à 09.30.17 

  1. Now we need to add a row for the API keys, so under “select buckets type”, choose “Split Rows”

 Capture d’écran 2020 06 16 à 09.30.40 

  1. To find API keys, choose Terms under “Aggregation” and choose data.apiKey.keyword under “Field”

 Capture d’écran 2020 06 16 à 09.30.57 

  1. To add the average response time per API key to the data table, click on “Add metrics” under “metrics”

 Capture d’écran 2020 06 16 à 09.31.18 

  1. Under “Aggregation” choose Average, and under “Field” choose data.response.totalTime

 Capture d’écran 2020 06 16 à 09.32.29 

  1. Click on “Apply changes”. We now have a dashboard that shows us the average response time per API key:

 Capture d’écran 2020 06 16 à 09.33.07 

Creating the ‘Average time per operation’ Visualization 

 

The second dashboard we created for ElephantDrive displays the average time elapsed (in ms) by operation type. The steps to implement this dashboard are as follows: 

  1. Go to the “Visualize” tab and click the “Create new visualization” button 

 Capture d’écran 2020 06 16 à 11.10.19

  1. For this type of data, choosing “Vertical Bar” is a relevant choice.

 Capture d’écran 2020 06 16 à 11.19.42 

  1. The data required to create this dashboardis located inthe “logstash” database. 

 Capture d’écran 2020 06 16 à 11.20.10 

  1. We first want to see the average time, so in the “metrics” section choose “Average” in Aggregation and “data.response.totalTime” in Field.

 Capture d’écran 2020 06 16 à 11.20.33

  1. Then we want to add a metric

 Capture d’écran 2020 06 16 à 11.22.06

  1. Select “Dote Size”

 Capture d’écran 2020 06 16 à 11.24.05

  1. And select “Count” as Aggregation, to be able to see how many times the query has been used.

 Capture d’écran 2020 06 16 à 11.24.32

  1. Now, to see which method is used (GET, POST, PUT, etc..) wehave togo in the Bucket section, and choose “Split Series” 

 Capture d’écran 2020 06 16 à 11.24.53

  1. And then “Terms” as Aggregation, “data.request.method.keyword” as field, and Order By “Average time” (created step 4)

 Capture d’écran 2020 06 16 à 11.25.27

  1. Finally, to see where the operation has been made, we will add a sub-bucket

 Capture d’écran 2020 06 16 à 11.27.05

  1. And create a “Terms” Sub Aggregation, with “data.operation.keyword” as Field

 Capture d’écran 2020 06 16 à 11.33.38

There you have it! 

 Capture d’écran 2020 06 16 à 11.34.20

And you can see more details by hovering over a section with your mouse 

 Capture d’écran 2020 06 16 à 11.34.41

Better Visibility & Enhanced Productivity In Log Monitoring 

Cloud based infrastructure can sometimes feel like a black box with only limited visibility into its efficiency. Utilizing Kibana and Elasticsearch provides ElephantDrive with informative dashboards that provide insight into their compute environment, enhancing the efficacy of their log monitoring efforts. 

Download PDF

About TrackIt

TrackIt, an Amazon Web Services Advanced Consulting Partner based in Marina del Rey, CA, offers a range of cloud management, consulting, and software development solutions. Their expertise includes Modern Software Development, DevOps, Infrastructure-As-Code, Serverless, CI/CD, and Containerization, with a focus on Media & Entertainment workflows, High-Performance Computing environments, and data storage.

TrackIt excels in cutting-edge software design, particularly in the areas of containerization, serverless architectures, and pipeline development. The company’s team of experts can help you design and deploy a custom solution tailored to your specific needs.

In addition to cloud management and modern software development services, TrackIt also provides an open-source AWS cost management tool to help users optimize their costs and resources on the platform. With its innovative approach and expertise, TrackIt is the ideal partner for organizations seeking to maximize the potential of their cloud infrastructure.

Previous Next

AWS Advanced Consulting Partner - Cloud Management, Consulting, and Software Development Solutions

Quick links

    • Home
    • Solutions
    • Services
    • White Papers
    • Blog
    • About Us

Tools & Apps

    • TrackIt AI Video Reviewer
    • Trackflix
    • TrackIt App
    • Tagbot

get in touch

    • Phone: +1 323-230-0042
    • Email: info@trackit.io

    • TrackIt 578 Washington Blvd #260 Marina Del Rey, CA 90292
  • @TrackIt 2022