Knowledge Base & Community Wiki
Exporting Data From Sumologic for purposes of Visualization, Modelling & Forecasting
What Is Sumologic – Sumo Logic is a cloud-based log management and analytics service that leverages machine-generated big data to deliver real-time IT insights. Sumo Logic’s architecture features an elastic petabyte scale platform that collects, manages, and analyses enterprise log data, reducing millions of log lines into operational and security insights in real time. Their cloud-based approach overcomes the inherent problems of premises-based solutions, including limits on scalability, inefficient or haphazard analysis, and uncontrolled costs. Sumo Logic is built around a globally distributed data retention architecture that keeps all log data available for instant analysis, eliminating the need for an enterprise to manage the cost and complexity of data archiving, backups and restoration.
To learn more about Sumologic please visit – http://community.visualize-it.co/knowledgebase/installing-and-configuring-sumologic-for-data-collection-aggregation/
Pre-requisites for exporting data from Sumologic – A pre-requisite for extracting data from Sumologic is as follows –
- Setup Sumologic for your environment. This involves creating an account online and configuring it for use with your systems to collect relevant application, infrastructure metrics.
- Configure Sumologic collectors to collect data from your applications, infrastructure, network devices, etc. The installation of collectors is really straightforward and completely automated. The collector upgrade process was quite impressive too.
- Collect data for key Application, Infrastructure workload drivers i.e. CPU Utilization, Mem Utilization, Orders Placed Per Hour, Messages Transmitted Per Hour, etc. for an extended period of time.
- Confirm that you are able to see the data from your various key applications, infrastructure being collected, indexed and stored by Sumologic
To learn more about configuring and setting up Sumologic to collect data for key workload drivers please visit – http://community.visualize-it.co/knowledgebase/installing-and-configuring-sumologic-for-data-collection-aggregation/
Exporting data from Sumologic – This section assumes that you have addressed all the pre-requisites as listed in the previous section. If required please go back and read through the pre-requisites section again.
Exporting data from Sumologic is relatively easy. To get started let’s login to the Sumologic portal online.
Once you have logged into your Sumologic installation which by now holds a treasure trove of data you would need to execute a search using the Sumologic Search option. Before we execute the Search query in Sumologic to pick the relevant data it’s good practice to visualize the data for the given workload dimension. In this case the data we are dealing with is from varnish and the source of the data being collected by Sumologic is Varnish Logs for one of our servers. Make sure you have relevant data for the given workload driver within Sumologic over the period of time you would like to perform modelling.
If Sumologic isn’t receiving the data or if you are missing chunks of data for given time periods you need to go back and debug the issues.
Let’s use the search option presented by Sumologic to execute a custom query. The query we execute in this case is – _”sourceCategory=Varnish_Logs and _sourceHost=Gandalf | timeslice by 5m | count by _timeslice”.
This query requests Sumologic to provide us a view of data obtained from the source – “/var/logs/varnish/varnishncsa.log” for the given application server. We are requesting the data such that the data is rolled up into 5 minute segments. The data in the source file “/var/logs/varnish/varnishncsa.log” has been logged by the reverse application proxy Varnish which is in the open NCSA format. To see how to setup basic system monitors that could feed data into Splunk please look at the articles at the Statistical Modelling section of the wiki – http://community.visualize-it.co/knowledgebase/collecting-performance-data-from-unix-systems/
The default resolution for this data was initially set at “15 mins”. If you look at the right hand side top (Adjacent to the search button) We changed this to the period for which we wanted to extract data from Sumologic i.e. “Last 7 days” which in our case was the last 7 days for which we were collecting data. Sumologic will limit the display of data on the screen due to various reasons but do not fret, the data will be available when you export it to your machine.
Once you’ve run the query and changed the time resolution, you should give Sumologic time to complete running the query. The Sumologic interface if very intuitive and you will see a progress bar that provides a view of when the query has completed execution. Once the execution is complete you can hit the download button (left most button in the middle of the screen) to export the data from Sumologic. In the above screenshot this button has been highlighted to make it easier for you to spot.
Please keep in mind that this button is only visible once the query has completed execution. Also, avoid closing the Sumologic window while the query is executing or the file is still downloading, else you’ll end up with missing data in your data exports. Sumologic will provide the results of the query in a CSV (Comma Separate Values) format which is a format supported by VisualizeIT.
What does the exported data look like – Sumologic’s CSV exports will provide data in the following format i.e.
“02/01/2016 12:15:00.000 +1000″,”311″,”295”
“02/01/2016 12:20:00.000 +1000″,”448″,”418”
“02/01/2016 12:25:00.000 +1000″,”506″,”525”
“02/01/2016 12:30:00.000 +1000″,”442″,”409”
“02/01/2016 12:35:00.000 +1000″,”596″,”505”
“02/01/2016 12:40:00.000 +1000″,”567″,”600”
“02/01/2016 12:45:00.000 +1000″,”521″,”555”
“02/01/2016 12:50:00.000 +1000″,”532″,”552”
“02/01/2016 12:55:00.000 +1000″,”475″,”509”
“02/01/2016 13:00:00.000 +1000″,”472″,”429”
“02/01/2016 13:05:00.000 +1000″,”559″,”551”
The Sumologic CSV format is supported by VisualizeIT and you are now ready to import this data directly into VisualizeIT using the Data Management capability within the Statistical Modelling section.
Conclusion – In this article we’ve looked at a brief introduction to Sumologic, the pre-requisites for exporting data from Sumologic, the approach to query and export data using Sumologic and final the supported CSV formats from a Sumologic standpoint. Importing data from Sumologic is pretty straight forward, it’s the configuration of Sumologic to collect data from your various workload drivers that can tend to be an arduous task. Happy hacking!!!
Modelling Solution: VisualizeIT offers access to a bunch of Analytical Models, Statistical Models and Simulation Models for purposes of Visualization, Modelling & Forecasting. Access to all the Analytical (Mathematical) models is free. We recommend you try out the Analytical models at VisualizeIT which are free to use and drop us a note with your suggestions, input and comments. You can access the VisualizeIT website here and the VisualizeIT modelling solution here –VisualizeIT.