Exporting Data From ELK for purposes of Visualization, Modelling & Forecasting

Knowledge Base & Community Wiki

Exporting Data From ELK for purposes of Visualization, Modelling & Forecasting

in

What Is ELK – Elasticsearch, Logstash, Kibana or ELK in short is a log Aggregation and Analytics solution based on a completely Open Source stack. ELK consists of consists of three different Open Source components –

  • Elasticsearch
  • Logstash
  • Kibana.

The three components that make up the ELK stack offer the following functionality –

  • Elasticsearch: A powerful open source search and analytics engine that makes data easy to explore. It is a search server based on Apache Lucene.
  • Logstash: A log management tool used for centralised logging, log enrichment and parsing.
  • Kibana: A browser-based HTML5 dashboard used to visualize Elasticsearch data.

The open-source ELK stack provides the ability to perform operational and data analytics including comprehensive text based search functionality on almost any type of structured or unstructured data source.

To learn more about ELK, how to install ELK and where to download ELK from please visit – http://community.visualize-it.co/knowledgebase/installation-configuration-of-elk-for-data-collection-aggregation/

Pre-requisites for exporting data from ELK – A pre-requisite for extracting data from ELK is as follows –

  • Setup ELK for your environment. This involves creating an account online and configuring it for use with your systems to collect relevant application, infrastructure metrics.
  • Configure ELK (logstash) collectors to collect data from your applications, infrastructure, network devices, etc. The installation of collectors is really straightforward and completely automated. The collector upgrade process was quite impressive too.
  • ELK setup will require – Kibana being used for the dashboard, Logstash being used to parse data, Logstash collectors being used to pipe data through from remote machine and Elasticsearch being used to index and store data.
  • Collect data for key Application, Infrastructure workload drivers i.e. CPU Utilization, Mem Utilization, Orders Placed Per Hour, Messages Transmitted Per Hour, etc. for an extended period of time.
  • Confirm that you are able to see the data from your various key applications, infrastructure being collected, indexed and stored by ELK.

To learn more about configuring and setting up ELK to collect data for key workload drivers please visit – http://community.visualize-it.co/knowledgebase/installation-configuration-of-elk-for-data-collection-aggregation/

 Exporting data from ELK – This section assumes that you have addressed all the pre-requisites as listed in the previous section. If required please go back and read through the pre-requisites section again. Exporting data from ELK is relatively easy. To get started let’s login to the Kibana portal.

Once you have logged into your ELK installation which by now holds a treasure trove of data you would need to execute a search using the Kibana Search option. Before we execute the Search query in Kibana to pick the relevant data it’s good practice to visualize the data for the given workload dimension. In this case the data we are dealing with is from the authentication logs and the source of the data being collected by ELK is System Auth Logs for one of our servers. Make sure you have relevant data for the given workload driver within ELK over the period of time you would like to perform modelling.

If ELK isn’t receiving the data or if you are missing chunks of data for given time periods you need to go back and debug the issues.

 ELK_1Kibana Dashboard with Search Results Displayed

 

Let’s use the search option presented by Sumologic to execute a custom query. The query we execute in this case is – ” host:”Frodo” and “iptables denied””.

This query requests Kibana to search through the datastore within Elasticsearch and provide us a view of data obtained from the source – “/var/logs/auth.log” for the given application server. The data in the source file “/var/logs/auth.log” has been logged by the Ubuntu Operating system with all the connections that were dropped. These are IPTables (Linux Firewall) entries on the system. To see how to setup basic system monitors that could feed data into ELK please look at the articles at the Statistical Modelling section of the wiki – http://community.visualize-it.co/knowledgebase/collecting-performance-data-from-unix-systems/

 

ELK_2Kibana Visualization for the Iptables Denied Search Results

 

There are a lot of other Kibana Seach tutorials out there. Feel free to dive into the tutorials and learn more about the power of ELK and Kibana – https://www.google.com.au/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=kibana%20search%20tutorial

To be able to export data for a given query you would need to follow the steps outlined below –

  • Identify the query you want to run
  • Save the query using the Kibana interface
  • Create a Visualization using the Kibana interface
  • Add the Visualization to a Dashboard
  • Open the relevant Dashboard within Kibana with the Visualization in it

 

ELK_3Kibana Visualization with “Raw Data” Export Option Displayed

 

To learn more about configuring and setting up ELK to collect data for key workload drivers please visit – http://community.visualize-it.co/knowledgebase/installation-configuration-of-elk-for-data-collection-aggregation/

Once you’ve opened up the dashboard with the Visualization in it you will see a small arrow at the bottom of the visualization. Clicking on this arrow reveals an option called “Raw Data”. Click on the “Raw Data” option to export data for the given Visualization on Kibana.

What does the exported data look like – ELK’s CSV exports will provide data in the following format i.e.

“@timestamp per 3 hours”,Count

1456686000000,114

1456696800000,184

1456707600000,176

1456718400000,208

1456729200000,210

1456740000000,192

1456750800000,172

1456761600000,290

1456772400000,220

1456783200000,226

 

The ELK CSV format is supported by VisualizeIT and you are now ready to import this data directly into VisualizeIT using the Data Management capability within the Statistical Modelling section.

Conclusion – In this article we’ve looked at a brief introduction to ELK, the pre-requisites for exporting data from ELK, the approach to query and export data using ELK and final the supported CSV formats from a ELK standpoint. Importing data from ELK is pretty straight forward, it’s the configuration of ELK to collect data from your various workload drivers that can tend to be an arduous task. Happy hacking!!!

Modelling Solution: VisualizeIT offers access to a bunch of Analytical Models, Statistical Models and Simulation Mcropped-visualize_it_logo__transparent_090415.pngodels for purposes of Visualization, Modelling & Forecasting. Access to all the Analytical (Mathematical) models is free. We recommend you try out the Analytical models at VisualizeIT which are free to use and drop us a note with your suggestions, input and comments. You can access the VisualizeIT website here and the VisualizeIT modelling solution here –VisualizeIT.

This entry was posted in   .
Bookmark the   permalink.

Admin has written 0 articles

VisualizeIT Administrator & Community Moderator