Installation & Configuration of ELK For Data Collection & Aggregation

Knowledge Base & Community Wiki

Installation & Configuration of ELK For Data Collection & Aggregation

in

What Is ELK –  Elasticsearch, Logstash, Kibana or ELK in short is a log Aggregation and Analytics solution based on a completely Open Source stack. ELK consists of consists of three different Open Source components –

  • Elasticsearch
  • Logstash
  • Kibana.

The three components that make up the ELK stack offer the following functionality –

  • Elasticsearch: A powerful open source search and analytics engine that makes data easy to explore. It is a search server based on Apache Lucene.
  • Logstash: A log management tool used for centralised logging, log enrichment and parsing.
  • Kibana: A browser-based HTML5 dashboard used to visualize Elasticsearch data.

The open-source ELK stack provides the ability to perform operational and data analytics including comprehensive text based search functionality on almost any type of structured or unstructured data source.

elk-flow

Although they’ve all been built to work exceptionally well together, each one is a separate project that is driven by the open-source vendor Elastic—which itself began as an enterprise search platform vendor. Elasticsearch has now become a full-service analytics software company, mainly because of the success of the ELK stack and its acceptance globally as an Open Source analytics and visualization solution. Wide adoption of Elasticsearch for analytics has been the main driver of its popularity. ELK is very similar to Splunk or Sumologic in terms of basic functionality but is run as an Open Source platform.

To learn more about ELK (Elasticsearch, Logstash, Kibana) please visit – Elastic.

Installation, Configuration and Setup of Elasticsearch, Logstash and Kibana – There are quite a few tutorials available out there that dive into the technicality associated with installation and configuration of ELK. Like most Open Source products, there are a million ways of installing and configuration ELK. We recommend the following set of articles at Digital Ocean which will help you setup, configure and install Elasticsearch, Logstash and Kibana. We can also personally vouch for the fact that the setup process documented in the Digital Ocean ELK tutorial works since we’ve tried the entire setup process ourselves on a Linux Ubuntu machine.

  1. How To Install Elasticsearch, Logstash, and Kibana (ELK Stack) on Ubuntu 14.04
  2. How To Gather Infrastructure Metrics with Topbeat and ELK on Ubuntu 14.04
  3. Adding Logstash Filters To Improve Centralized Logging
  4. How To Use Kibana Dashboards and Visualizations
  5. How To Map User Location with GeoIP and ELK (Elasticsearch, Logstash, and Kibana)

Installation and configuration of ELK is a slightly complicated process and we wouldn’t recommend you proceed with it unless you have fairly strong Linux/Unix systems administration skills.

ELK as a service – If installation, configuration and maintenance of ELK is not your cup of tea then why not look at SaaS based solution providers. Here are some of the leading SaaS based providers for ELK SaaS based solutions –

Conclusion – We hope this article has helped you decide on which approach to install, configure and deploy ELK works for you. As we have mentioned earlier there are a million ways of doing anything in the Open Source world and the same applies to ELK (Elasticsearch, Logstash, Kibana) as well. A point also to be noted is that while Open Source is highly configurable, scalable and reliable, it does come with a lot complexity attached and here is where the SaaS based solution offerings above might help. Au Revior!!!

Modelling Solution: VisualizeIT offers access to a bunch of Analytical Models, Statistical Models and Simulation Mcropped-visualize_it_logo__transparent_090415.pngodels. Access to all the Analytical (Mathematical) models is free. We recommend you try out the Analytical models at VisualizeIT which are free to use and drop us a note with your suggestions, input and comments. You can access the VisualizeIT website here and the VisualizeIT modelling solution here –VisualizeIT.

This entry was posted in   .
Bookmark the   permalink.

Admin has written 0 articles

VisualizeIT Administrator & Community Moderator