Knowledge Base & Community Wiki
Installing Splunk Forwarder On Ubuntu For Data Collection & Aggregation
What Is Splunk (http://www.splunk.com) – Splunk is the heavyweight commercial software which enables you to index, visualise and explore virtually any machine generated data. Splunk is often used to consume Apache, Varnish and Nginx web server logs as well as website clicks and any other data which maintains a constant format. Installing Splunk on any Debian based Linux distribution, such as Ubuntu, couldn’t be easier with the .deb package that available for download.
Visit the Splunk download page to download the Splunk .deb package: Download Splunk
What Is The Splunk Forwarder – The Splunk Universal Forwarder is a small, light weight daemon which forwards data to your main Splunk server from a variety of sources. This guide assumes that you have already installed the Splunk server to receive the data.
You can download the Splunk Universal Forwarder .deb file from the Splunk website – Splunk Forwarder Download
Installing The Splunk Forwarder – Upload the file to your Ubuntu server and place it a temporary directory. Run the dpkg command to install the Splunk server. The file name of the .deb file may change as new versions are made available so make sure that you have downloaded.
bash# dpkg -i splunkforwarder-6.0.3-204106-linux-2.6-amd64.deb
The output will look like the below. Once you see complete, the Splunk Forwarder installation will be complete.
Selecting previously unselected package splunkforwarder.
(Reading database … 28352 files and directories currently installed.)
Unpacking splunkforwarder (from splunkforwarder-6.0.3-204106-linux-2.6-amd64.deb) …
Setting up splunkforwarder (6.0.3-204106) …
Next we need to create the init.d script so that we can easily start and stop Splunk. Change the the Splunk directory and run the splunk executable with the below arguments.
bash# cd /opt/splunkforwarder/bin/
bash# ./splunk enable boot-start
Press SPACE to view all of the license agreement and then Y to accept it.
You can now start the forwarder daemon using the init.d script.
bash# service splunk start
See reading log files with the Splunk Forwarder to read your first log file and send the data to the Splunk server.
Configuration Of The Splunk Forwarder – Here are some additional instructions for configuration of the Splunk Forwarder.
Splunk Command Line Reference:
http://docs.splunk.com/Documentation/Splunk/latest/Admin/AccessandusetheCLIonaremoteserver Note: the CLI may ask you to authenticate – it’s asking for the LOCAL credentials, so if you haven’t changed the admin password on the forwarder, you should use admin/changeme. Listed below is the summary of steps for Installing/Configuring Splunk Forwarder on Linux.
- Step 1: Download Splunk Universal Forwarder: http://www.splunk.com/download/universalforwarder (64bit package if applicable!)
- Step 2: Install Forwarder
- Step 3: Enable boot-start/init script: /opt/splunkforwarder/bin/splunk enable boot-start (start splunk: /opt/splunkforwarder/splunk start)
- Step 4: Enable Receiving input on the Index Server Configure the Splunk Index Server to receive data, either in the manager: Manager -> sending and receiving -> configure receiving -> new or via the CLI: /opt/splunk/bin/splunk enable listen 9997 Where 9997 (default) is the receiving port for Splunk Forwarder connections
- Step 5: Configure Forwarder connection to Index Server: /opt/splunkforwarder/bin/splunk add forward-server hostname.domain:9997 (where hostname.domain is the fully qualified address or IP of the index server (like indexer.splunk.com), and 9997 is the receiving port you create on the Indexer: Manager -> sending and receiving -> configure receiving -> new)
- Step 6: Test Forwarder connection: /opt/splunkforwarder/bin/splunk list forward-server
- Step 7: Add Data: /opt/splunkforwarder/bin/splunk add monitor /path/to/app/logs/ -index main -sourcetype %app% Where /path/to/app/logs/ is the path to application logs on the host that you want to bring into Splunk, and %app% is the name you want to associate with that type of data. This will create a file: inputs.conf in /opt/splunkforwarder/etc/apps/search/local/ — here is some documentation on inputs.conf: http://docs.splunk.com/Documentation/Splunk/latest/admin/Inputsconf Note: System logs in /var/log/ are covered in the configuration part of Step 7. If you have application logs in /var/log/*/
- Step 8 (Optional): Install and Configure UNIX app on Indexer and *nix forwarders: On the Splunk Server, go to Apps -> Manage Apps -> Find more Apps Online -> Search for ‘Splunk App for Unix and Linux’ -> Install the “Splunk App for Unix and Linux’ Restart Splunk if prompted, Open UNIX app -> Configure Once you’ve configured the UNIX app on the server, you’ll want to install the related Add-on: “Splunk Add-on for Unix and Linux” on the Universal Forwarder. Go to http://apps.splunk.com/ and find the “Splunk Add-on for Unix and Linux” (Note you want the ADD-ON, not the App – there is a difference!). Copy the contents of the Add-On zip file to the Universal Forwarder, in: /opt/splunkforwarder/etc/apps/. If done correctly, you will have the directory “/opt/splunkforwarder/etc/apps/Splunk_TA_nix” and inside it will be a few directories along with a README & license files. Restart the Splunk forwarder (/opt/splunkforwarder/bin/splunk restart) Note: The data collected by the unix app is by default placed into a separate index called ‘os’ so it will not be searchable within splunk unless you either go through the UNIX app, or include the following in your search query: “index=os” or “index=os OR index=main” (don’t paste doublequotes)
- Step 9 (Optional): Customize UNIX app configuration on forwarders: Look at inputs.conf in /opt/splunkforwarder/etc/apps/unix/local/ and /opt/splunkforwarder/etc/apps/unix/default/ The ~default/inputs. path shows what the app can do, but everything is disabled. The ~local/inputs.conf shows what has been enabled – if you want to change polling intervals or disable certain scripts, make the changes in ~local/inputs.conf.
- Step 10 (Optional): Configure File System Change Monitoring (for configuration files):http://docs.splunk.com/Documentation/Splunk/4.3.2/Data/Monitorchangestoyourfilesystem
Note that Splunk also has a centralized configuration management server called Deployment Server. This can be used to define server classes and push out specific apps and configurations to those classes. So you may want to have your production servers class have the unix app configured to execute those scripts listed in ~local/inputs at the default values, but maybe your QA servers only need a few of the full stack, and at longer polling intervals. Using Deployment Server, you can configure these classes, configure the app once centrally, and push the appropriate app/configuration to the right systems.
You can also refer to the above configuration at Splunk Answers.
Extras : Splunk Installation & Configuration Videos – In addition to the installation guide provided above we thought it would be useful if we also included relevant tutorials from the vendor themselves. These videos have been created by Splunk and suggest an alternate approach to installation/configuration of the Splunk Forwarder service. These videos are available at their Youtube channel.
- Installation & Configuration of Splunk Forwarder for Linux
- Installation & Configuration of Splunk Forwarder for Windows
Important Links – Here are some important links which you might find useful –
Modelling Solution: VisualizeIT offers access to a bunch of Analytical Models, Statistical Models and Simulation Models. Access to all the Analytical (Mathematical) models is free. We recommend you try out the Analytical models at VisualizeIT which are free to use and drop us a note with your suggestions, input and comments. You can access the VisualizeIT website here and the VisualizeIT modelling solution here –VisualizeIT.