ELK STACK - Elasticsearch, Logstash and Kibana Integration

  • 2 minute(s) read
Prev Next

Introduction

Elasticsearch is a distributed, free, and open search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured. TheELK Stackis a collection of three open-source products - Elasticsearch, Logstash, and Kibana.

Elasticsearch allows you to store, search, and analyze huge volumes of data quickly and in near real-time and give back answers in milliseconds. Logstash, one of the core products of the Elastic Stack, is used to aggregate and process data and send it to Elasticsearch. Kibana is a data visualization and management tool for Elasticsearch that provides real-time histograms, line graphs, pie charts, and maps.

We integrate with ELK stack using Test Data Webhook API to send data directly to elastic search using a public facing endpoint accessible over http or https. This data then can be visualized using Kibana.

Prerequisites

  • Elasticsearch
  • Kibana
  • Public facing endpoint to accept data from Catchpoint.
  • Catchpoint test enabled with Test Data Webhook.

Installation and Configuration

Setup In Catchpoint portal:

  1. In the Catchpoint Portal, go to the API
  2. Under Test data Webhook, Add URL link.
    1. Append this path to the public endpoint pointing to Elasticsearch /<index_name>/_doc. This sends the data as document to a particular index.
    2. If there are authentication headers for the endpoint, add them as request headers. Expand Request link to add if required.
  3. Enter a Name.
  4. Enter endpoint URL.
  5. Under Format choose Template.
  6. Click Add New.
  7. Provide a template Name.
  8. Select Format as JSON.
  9. Paste the below JSON template.
    {"@timestamp": "${timestamp(YYYY-MM-DDTHH:MI:SS.MSCZ)}", "Catchpoint": ${JsonPayload}}
  10. Click Save.
  11. Select the newly created template.
  12. Add email address to notify in case it fails to send data over API.
  13. Click Save button on the top of the page.

Note: Index name is used to reference your data in Kibana.

Setup in Kibana:
Create an Index pattern with time stamp in Kabana. This can be done once the data from Catchpoint is being pushed into Elasticsearch. With the first data push, index will be created will use the same to create an index pattern.

  1. Open Kibana.
  2. Expand Menu and select Stack Management.
  3. Click on Index Patterns.
  4. Click Create index pattern.
  5. Type in the index name you provided in the endpoint.
  6. Click Next Step.
  7. Under Time field, select @timestamp.
  8. Click Create index pattern.

Result

To view the data, go to menu and select Discover and from the drop down select the index pattern you just created.Elasticsearch.JPG

Based on your requirements build dashboards to consume the data.
ELK.JPG