Knowledge Base & Community Wiki
Collecting Performance Data From Unix Systems Using TopBeat and ELK
What is ELK (Elasticsearch – Logstash – Kibana) – Elasticsearch, Logstash, Kibana or ELK in short is a log Aggregation and Analytics solution based on a completely Open Source stack. ELK consists of consists of three different Open Source components –
The three components that make up the ELK stack offer the following functionality –
- Elasticsearch: A powerful open source search and analytics engine that makes data easy to explore. It is a search server based on Apache Lucene.
- Logstash: A log management tool used for centralised logging, log enrichment and parsing.
- Kibana: A browser-based HTML5 dashboard used to visualize Elasticsearch data.
The open-source ELK stack provides the ability to perform operational and data analytics including comprehensive text based search functionality on almost any type of structured or unstructured data source.
Although they’ve all been built to work exceptionally well together, each one is a separate project that is driven by the open-source vendor Elastic—which itself began as an enterprise search platform vendor. Elasticsearch has now become a full-service analytics software company, mainly because of the success of the ELK stack and its acceptance globally as an Open Source analytics and visualization solution. Wide adoption of Elasticsearch for analytics has been the main driver of its popularity. ELK is very similar to Splunk or Sumologic in terms of basic functionality but is run as an Open Source platform.
To learn more about ELK (Elasticsearch, Logstash, Kibana) please visit – Elastic.
Installation, Configuration and Setup of Elasticsearch, Logstash and Kibana – There are quite a few tutorials available out there that dive into the technicality associated with installation and configuration of ELK. Like most Open Source products, there are a million ways of installing and configuration ELK. We recommend the following set of articles at Digital Ocean which will help you setup, configure and install Elasticsearch, Logstash and Kibana. We can also personally vouch for the fact that the setup process documented in the Digital Ocean ELK tutorial works since we’ve tried the entire setup process ourselves on a Linux Ubuntu machine.
- How To Install Elasticsearch, Logstash, and Kibana (ELK Stack) on Ubuntu 14.04
- How To Gather Infrastructure Metrics with Topbeat and ELK on Ubuntu 14.04
- Adding Logstash Filters To Improve Centralized Logging
- How To Use Kibana Dashboards and Visualizations
- How To Map User Location with GeoIP and ELK (Elasticsearch, Logstash, and Kibana)
Installation and configuration of ELK is a slightly complicated process and we wouldn’t recommend you proceed with it unless you have fairly strong Linux/Unix systems administration skills.
ELK as a service – If installation, configuration and maintenance of ELK is not your cup of tea then why not look at SaaS based solution providers. Here are some of the leading SaaS based providers for ELK SaaS based solutions –
Using Topbeat and ELK to capture System Performance Metrics – Topbeat is a lightweight shipper that you can install on your servers to periodically read system-wide and per-process CPU and memory statistics and then index the statistics in Elasticsearch. Topbeat helps you monitor your servers by collecting metrics like:
- System load: in the last minute, in the last 5 minutes, and in the last 15 minutes
- System wide CPU usage: user (and percentage), system, idle, IOWait, and so on at both per CPU and overall level
- System wide memory usage: total, used (and percentage), free, and so on
- System wide swap usage: total, used (and percentage), free, and so on
- Process name
- Process parent pid
- Process state
- Process pid
- Process CPU usage: user (and percentage), system, total, and start time
- Process Memory usage: virtual memory, resident memory (and percentage), and shared memory
File system statistics
- List of available disks
- For each disk, the name, type, and where it is mounted
- For each disk, the total, used (and percentage), free, and available space
Topbeat can insert the collected metrics directly into Elasticsearch or use Logstash. Here are a few sample Kibana Dashboards that provide a view of what Topbeat and ELK can provide in terms of view of System Performance Metrics.
To configure Topbeat to collect Performance Metrics for your Ubuntu box please see the link below –
To download and install Topbeat, use the commands that work with your system (deb for Debian/Ubuntu, rpm for Redhat/Centos/Fedora, mac for OS X). If you use Apt or Yum, you can install Topbeat from the Topbeat repositories to update to the newest version more easily.
See the Topbeat download page for other installation options, such as 32-bit images.
curl -L -O https://download.elastic.co/beats/topbeat/topbeat_1.1.2_amd64.deb sudo dpkg -i topbeat_1.1.2_amd64.deb
curl -L -O https://download.elastic.co/beats/topbeat/topbeat-1.1.2-x86_64.rpm sudo rpm -vi topbeat-1.1.2-x86_64.rpm
curl -L -O https://download.elastic.co/beats/topbeat/topbeat-1.1.2-darwin.tgz tar xzvf topbeat-1.1.2-darwin.tgz
Please look at the configuration files and the various configuration options which you will need to setup so that Topbeat can send the System Performance Metrics it collects to your ELK installation. Details on how to go about configuring Topbeat can be found here Topbeat Configuration.
Run Topbeat by issuing the following command:
sudo /etc/init.d/topbeat start
sudo /etc/init.d/topbeat start
sudo ./topbeat -e -c topbeat.yml -d "publish"
To learn more about the Topbeat, how to get started with Topbeat and installation of Top beat please see the link – Elastic – Topbeat Reference Documentation
Conclusion – We hope this article has helped you work out how to install, configure and obtain performance metrics for your system using ELK and Topbeat. Deciding on an approach to collect system performance metrics can be difficult especially with Open Source given the thousands of different ways one can get the job done. We hope we have set you off on your quest to install/configure/provision Topbeat and ELK (Elasticsearch, Kibana, Logstash) for your environment. A point also to be noted is that while Open Source is highly configurable, scalable and reliable, it does come with a lot complexity attached and here is where the SaaS based solution offerings above might help however you will still need to install the Topbeat agents on your machines by yourself. Au Revior!!!
Modelling Solution: VisualizeIT offers access to a bunch of Analytical Models, Statistical Models and Simulation Models. Access to all the Analytical (Mathematical) models is free. We recommend you try out the Analytical models at VisualizeIT which are free to use and drop us a note with your suggestions, input and comments. You can access the VisualizeIT website here and the VisualizeIT modelling solution here –VisualizeIT.