Setup Elasticsearch, Logstash, Kibana (ELK Stack) on Ubuntu 16.04 LTS
This post is mainly about my experience on setuping Elastic Stack on Ubuntu 16.04. The ELK (Elasticsearch, Logstash, Kibana) version I am using is 5.1.1. After Version 5.0, all products from Elasticsearch have been sharing the same versioning.
Briefly, ELK includes:
- Elasticsearch: a highly scalable full-text search and analytics engine.
- Logstash: a data collecting, parsing, filtering, pre-processing, and mapping engine with real-time pipelining capabilities.
- Kibana: a analytics and visualization web interface designed to work with Elasticsearch for searching and viewing logs from Elasticsearch.
- X-Pack: an Elastic Stack extention that bundles security, alerting, monitoring, reportnig, and graph capabilities into one easy-to-install package.
All three of ELK are open source. For X-Pack, you could have a 30-day trial. Alternatively, you could also use Nginx for proxy.
The data which are used for demonstrating the ELK Stack is from Twitter. Beside of ELK Stack, I will also install X-Pack since most of the companies need ELK but do not want their data/pivacy to be exposed outside.
In this setup, I will not install the software using sudo apt-get install on my Ubuntu. Instead, I will just download the binary files and run it in the command line. I would like to have the full control of it and also for the purpose that, if you are going to setup ELK Stack on a Acceptance/Production server, which requires high level of security, stability and reliability, you probably would like to do it in my way. At least you know exactly what you have done in case you screw it up, and you are able to scroll it back.
Before all, you have to check the Java version on your machine:
It is required and recommended by Elastic here that:
“Elasticsearch is built using Java, and requires at least Java 8 in order to run. Only Oracle’s Java and the OpenJDK are supported. The same JVM version should be used on all Elasticsearch nodes and clients. We recommend installing Java version 1.8.0_73 or later. Elasticsearch will refuse to start if a known-bad version of Java is used.”
First of all, I made a directory ELK for all the binaries
THen, I had everything downloaded and untared.
After untar every package, and you will have the followings:
I also have the X-Pack package ready but I will set it up when the full ELK Stack is ready.
Now, if you go to the Elasticsearch directory, you will find:
It is very easy to fire up Elasticsearch:
NOW, we have a working Elasticsearch:
Giving ?pretty is used for getting a more strutured-printed reponse.
Instead of running in front, you could also run Elasticsearch as a daemon in the background like this:
By specifying -d , you put the program as daemon; by giving -p pid , the process ID is recorded in a file, so that you can do kill ‘cat pid’ in order to kill it.
It is also easy to start Kibana:
You can access Kibana in a browser by typing: http://localhost:5601.
Till now, we have not had any data yet. We could put some simple test data as following:
Since there is no date mapping, the index test-* cannot be time-based:
We can create a non-time-based index:
There are the fields:
Now, let’s get Logstash into play and get some real data:
Here’s how we can start Logstash:
THe parameter –config.reload.automatic is for automatically reloading conf file.
The configuration file first.conf is showing like:
The input plugin twitter specifies the data source you are using, with several parameters to setup related to your Twitter application info.
The filter plugins grok and geoip , repectively, are used to parse the twitter message and map the client ip address to a Geo location.
The output plugin elasticsearch gives Logstash the destination of the processed, filtered and mapped data, which, here, is to Elasticsearch.
Now, we have a logstash-* as time-based:
And we are getting data from twitter:
Here is one twitter data sample:
It can also be shown in JSON format:
Perfection is finally attained not when there is no longer more to add, but when there is no longer anything to take away.