• Home
  • Archive
  • Tools
  • Contact Us

The Customize Windows

Technology Journal

  • Cloud Computing
  • Computer
  • Digital Photography
  • Windows 7
  • Archive
  • Cloud Computing
  • Virtualization
  • Computer and Internet
  • Digital Photography
  • Android
  • Sysadmin
  • Electronics
  • Big Data
  • Virtualization
  • Downloads
  • Web Development
  • Apple
  • Android
Advertisement
You are here: Home » Install Elastic Stack on Ubuntu 16.04, CentOS 7 Single Cloud Server

By Abhishek Ghosh January 30, 2017 9:13 am Updated on January 30, 2017

Install Elastic Stack on Ubuntu 16.04, CentOS 7 Single Cloud Server

Advertisement

Three softwares – Elasticsearch, Logstash, and Kibana better known before as ELK Stack. Here is How to Install Elastic Stack on Ubuntu 16.04, CentOS 7 on Single Cloud Server Instance For Server Log Analysis, Big Data Processing. In this tutorial, we will show how to installa of the mentioned softwares on Ubuntu 16.04 and CentOS 7. Additionally we will show you to configure, visualize the syslogs. The server will run Nginx web server.

 

Why Install Elastic Stack Instead of Apache Hadoop or Spark With Elastic Search?

 

Previously we discussed how to install Apache Hadoop, how to install Apache Spark and how to install fluentd on single cloud server instance. This guide to install Elastic Stack is not on Apache Hadoop as not all users are used to handle real Big Data software. In separate guide we will show how to integrate Elasticsearch with Apache Hadoop. This kind of setups and examples are practical usage of the software collections.

We are showing how to install Elastic Stack on Ubuntu and CentOS in this easy manner so that any level go users can install and use for basic log analysis purpose without going in to the fuss of Big Data softwares.

Advertisement

---

 

Why Install Elastic Stack on Ubuntu 16.04, CentOS 7 Single Cloud Server?

 

You can install Elastic Stack on same dedicated server where your web software or database software is/are running. But Cloud Server can be cheap like VPSDime 6GB OpenVZ instance costs only $7/month. Such resources are great for the small to medium business owners and webmasters. ES crash on OpenVZ not uncommon but a workaround, low load average does make it working. We are installing on a single cloud server instance for cost saving and isolation purpose. We suggest to use a server with 4GB of RAM for running Elastic Stack. 1GB RAM instance can run it somehow, 1GB RAM VMWare Cloud Server from Aruba Cloud will cost just 1 Euro per month.

The individual components does these works :

  1. Logstash collects, enrich and send data it to Elasticsearch
  2. ElasticSearch stores incoming logs from Logstash and provides the ability to search in real time.
  3. Kibana provides the way of visualization of logs. There are other softwares as alternative but Kibana is commonly used.
  4. Beats/Filebeat/fluentd when installed on client, sends logs to Logstash.
  5. Nginx works as web server, reverse proxy.
Install Elastic Stack on Ubuntu 16-04 CentOS 7 Single Cloud Server

 

Steps to Install Elastic Stack on Ubuntu 16.04, CentOS 7 Single Cloud Server

 

Step 1 : SELinux and Install Nginx

First we need to make sure that SELINUX is disabled. Open :

Vim
1
nano /etc/sysconfig/selinux

Change SELINUX value :

/etc/sysconfig/selinux
Vim
1
SELINUX=disabled

then reboot the server :

Vim
1
reboot

SSH to the server again and run this command to check the SELinux state :

Vim
1
getenforce

Install Nginx :

Vim
1
2
3
4
5
# ubuntu
apt install nginx
# centos
yum -y install epel-release
yum install nginx

Make sure to allow traffic through TCP port 9200, port 5044, port 80, port 443 in your firewall.

Step 2 : Install Oracle Java

For Ubuntu 16.04 :

Vim
1
2
3
sudo add-apt-repository -y ppa:webupd8team/java
sudo apt update
sudo apt install oracle-java8-installer

For CentOS 7 :

Vim
1
2
3
cd ~
wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http:%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u77-b02/jdk-8u77-linux-x64.rpm"
rpm -ivh jdk-8u77-linux-x64.rpm

For both of them :

Check the version of Java :

Vim
1
java -version

Example output :

Vim
1
2
3
4
5
java -version
 
java version "1.8.0_11"
Java(TM) SE Runtime Environment (build 1.8.0_11-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.11-b03, mixed mode)

Step 3 : Install ElasticSearch

For Ubuntu 16.04 :

Vim
1
2
3
4
5
cd ~
wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb http://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list
sudo apt update
sudo apt install elasticsearch

If you need to restrict access to Elasticsearch instance at port 9200, do this:

Vim
1
nano /etc/elasticsearch/elasticsearch.yml

Edit :

/etc/elasticsearch/elasticsearch.yml
Vim
1
network.host: localhost

Save, exit, start Elasticsearch, configure to start Elasticsearch on boot up::

Vim
1
2
3
4
5
6
sudo systemctl restart elasticsearch
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch
# foolproof
sudo systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service

Test elasticsearch :

Vim
1
curl localhost:9200

For CentOS 7 :

Vim
1
2
3
4
5
6
cd ~
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.1.1.rpm
rpm -ivh elasticsearch-5.1.1.rpm
cd /etc/elasticsearch/
nano elasticsearch.yml

If you need to restrict access to Elasticsearch instance at port 9200, do this:

Vim
1
nano /etc/elasticsearch/elasticsearch.yml

Edit :

/etc/elasticsearch/elasticsearch.yml
Vim
1
2
3
bootstrap.memory_lock: true
network.host: localhost
http.port: 9200

Save, exit. This disables memory swapping for Elasticsearch :

Vim
1
nano /usr/lib/systemd/system/elasticsearch.service

Uncomment :

Vim
1
LimitMEMLOCK=infinity

Save and exit. Edit :

Vim
1
nano /etc/sysconfig/elasticsearch

Vim
1
MAX_LOCKED_MEMORY=unlimited

Save and exit. To start Elasticsearch, configure to start Elasticsearch on boot up::

Vim
1
2
3
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch

Test :

Vim
1
curl -XGET 'localhost:9200/?pretty'

Step 4 : Install Kibana

For Ubuntu 16.04 :

Vim
1
2
wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb http://packages.elastic.co/kibana/4.5/debian stable main" | sudo tee -a /etc/apt/sources.list

Install the Kibana using the following command.

Vim
1
2
sudo apt update
sudo apt install kibana

Start and enable kibana on system startup :

Vim
1
2
sudo systemctl start kibana
sudo systemctl enable kibana

Access the Kibana using the following URL :

Vim
1
http://your-ip-address:5601/

For CentOS 7 :

Vim
1
2
wget https://artifacts.elastic.co/downloads/kibana/kibana-5.1.1-x86_64.rpm
rpm -ivh kibana-5.1.1-x86_64.rpm

Now edit the Kibana configuration file :

Vim
1
nano /etc/kibana/kibana.yml

Uncomment the configuration lines for server.port, server.host and elasticsearch.url.

Vim
1
2
3
server.port: 5601
server.host: "localhost"
elasticsearch.url: "http://localhost:9200"

Save and exit. Add Kibana to run at boot and start it :

Vim
1
2
sudo systemctl enable kibana
sudo systemctl start kibana

Kibana will run on port 5601 as node application.

For both of them :

Now configure with Nginx :

Vim
1
nano /etc/nginx/sites-available/default

This is typical example of nginx config:

nginx config
Vim
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
server {
    listen 80;
 
    server_name example.com;
 
    # auth_basic "Restricted Access";
    # auth_basic_user_file /etc/nginx/htpasswd.users;
 
    location / {
        proxy_pass http://localhost:5601;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;        
    }
}

Save and exit. Test restart Nginx :

Vim
1
2
nginx -t
sudo systemctl restart nginx

Step 5 : Install Logtash

For Ubuntu 16.04 :

Vim
1
2
3
echo "deb https://packages.elastic.co/logstash/2.3/debian stable main" | sudo tee -a /etc/apt/sources.list
sudo apt-get update
sudo apt-get install logstash

For CentOS 7 :

Open :

Vim
1
nano /etc/yum.repos.d/logstash.repo

add :

Vim
1
2
3
4
5
6
[logstash]
name=Logstash
baseurl=http://packages.elasticsearch.org/logstash/2.2/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1

Install the Logstash package:

Vim
1
yum install logstash

For Both of them :

We are showing for subdomain/domain installation, which is easy :

Vim
1
nano /etc/hosts

Add entry with real values :

Vim
1
10.11.12.13 server.your.local

Then run after changing “server.your.local” :

Vim
1
2
cd /etc/ssl/
openssl req -x509 -nodes -newkey rsa:2048 -days 365 -keyout logstash-forwarder.key -out logstash-forwarder.crt -subj /CN=server.your.local

If you want to use IP address then :

Vim
1
nano /etc/ssl/openssl.cnf

Edit to your IP :

Vim
1
subjectAltName = IP:10.11.12.13

Vim
1
2
cd /etc/ssl/
openssl req -x509 -days 365 -batch -nodes -newkey rsa:2048 -keyout logstash-forwarder.key -out logstash-forwarder.crt

Logstash configuration found at /etc/logstash/conf.d/. logstash configuration file consists of 3 parts – input, filter, and output. It is better to use a single file for these three.

Vim
1
nano /etc/logstash/conf.d/logstash.conf

Keep in mind to change /etc/ssl/logstash-forwarder.crt :

Vim
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
input {
beats {
   port => 5044
   ssl => true
   ssl_certificate => "/etc/ssl/logstash-forwarder.crt"
   ssl_key => "/etc/ssl/logstash-forwarder.key"
   congestion_threshold => "40"
  }
}
filter {
if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGLINE}" }
    }
 
    date {
match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
  }
 
}
output {
elasticsearch {
  hosts => localhost
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
       }
stdout {
    codec => rubydebug
       }
}

Save and exit. Now run :

Vim
1
2
3
sudo systemctl daemon-reload
sudo systemctl start logstash
sudo systemctl enable logstash

You can run cat :

Vim
1
sudo cat /var/log/logstash/logstash.log

Step 6 : Install Filebeat

There are four beats clients available. Packetbeat – Analyzes network packet data, Filebeat gives insight into log data, Topbeat gives insights from infrastructure data, Metricbeat delivers metrics to Elasticsearch.

For Ubuntu 16.04 :

Vim
1
2
3
echo "deb https://packages.elastic.co/beats/apt stable main" |  sudo tee -a /etc/apt/sources.list.d/beats.list
sudo apt-get update
sudo apt-get install filebeat

For CentOS 7 :

Vim
1
2
3
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.1.1-x86_64.rpm
rpm -ivh filebeat-5.1.1-x86_64.rpm

For both :

Filebeat uses SSL certificate for validating logstash server identity, so we need to copy the logstash-forwarder.crt from the logstash server to the client :

Vim
1
scp -pr root@server.your.local:/etc/ssl/logstash-forwarder.crt /etc/ssl

Now :

Vim
1
2
cd /etc/filebeat/
nano filebeat.yml

Add two files ‘/var/log/secure’ for ssh activity and ‘/var/log/messages’ for the server log :

Vim
1
2
3
  paths:
    - /var/log/secure
    - /var/log/messages

Also :

Vim
1
2
input_type: log
document_type: syslog

Under output uncomment line with logstash, edit the IP address of ELK server and port where Logstash is listening hosts, make sure the path to the certificate points to the actual file. Thereafter restart :

Vim
1
systemctl restart filebeat

Now go to the IP address where we can get Kibana. Installing Elastic Stack is difficult work.

Fully optional testing :

For TESTING purpose, you can download sample Kibana dashboards and Beats index patterns :

Vim
1
2
3
4
curl -L -O https://download.elastic.co/beats/dashboards/beats-dashboards-1.1.0.zip
unzip beats-dashboards-1.1.0.zip
cd beats-dashboards-1.1.0
./load.sh

Download the filebeat index template

Vim
1
curl -O https://gist.githubusercontent.com/thisismitch/3429023e8438cc25b86c/ raw/d8c479e2a1adcea8b1fe86570e42abab0f10f364/filebeat-index-template.json

Run the following CURL command :

Vim
1
curl -XPUT 'http://localhost:9200/_template/filebeat?pretty' -d@filebeat-index-template.json

Tagged With centos install elstic fluentd , elastic stack  6 7 0 インストール ubuntu , elasticstack , install elastic stack ubuntu , ubuntu server image with elastic stack

This Article Has Been Shared 150 Times!

Facebook Twitter Pinterest

Abhishek Ghosh

About Abhishek Ghosh

Abhishek Ghosh is a Businessman, Surgeon, Author and Blogger. You can keep touch with him on Twitter - @AbhishekCTRL.

Here’s what we’ve got for you which might like :

Articles Related to Install Elastic Stack on Ubuntu 16.04, CentOS 7 Single Cloud Server

  • OpenShift PaaS Marketplace is a Matured Step by Red Hat

    OpenShift PaaS marketplace is a matured step by Red Hat in the PaaS world. Users have seen so matured marketplace only from IaaS providers.

  • Docker on Ubuntu on Cloud Server to Manage Containers

    Docker can be installed on Ubuntu Cloud Server to manage containers. Docker is a free software that automates the deployment of applications.

  • How will Be the Future in Software Defined, Cloud and IoT?

    How will Be the Future in Software Defined, Cloud and IoT? How we will access to the new Internet? What About Technology and medicine?

  • Install Rackspace Monitoring Agent (Free) on Cloud Server

    Here is the details on How To Install Rackspace Monitoring Agent on Cloud Server and Set Monitoring Client on Dashboard to Get Nice Graphs.

  • Run Facebook Quiz Application on OpenShift PaaS

    Instantly build and deploy an awesome Facebook Quiz Application For Free of Cost With OpenShift Free PaaS and Their Prototype Application.

Additionally, performing a search on this website can help you. Also, we have YouTube Videos.

Take The Conversation Further ...

We'd love to know your thoughts on this article.
Meet the Author over on Twitter to join the conversation right now!

If you want to Advertise on our Article or want a Sponsored Article, you are invited to Contact us.

Contact Us

Subscribe To Our Free Newsletter

Get new posts by email:

Please Confirm the Subscription When Approval Email Will Arrive in Your Email Inbox as Second Step.

Search this website…

 

Popular Articles

Our Homepage is best place to find popular articles!

Here Are Some Good to Read Articles :

  • Cloud Computing Service Models
  • What is Cloud Computing?
  • Cloud Computing and Social Networks in Mobile Space
  • ARM Processor Architecture
  • What Camera Mode to Choose
  • Indispensable MySQL queries for custom fields in WordPress
  • Windows 7 Speech Recognition Scripting Related Tutorials

Social Networks

  • Pinterest (24.3K Followers)
  • Twitter (5.8k Followers)
  • Facebook (5.7k Followers)
  • LinkedIn (3.7k Followers)
  • YouTube (1.3k Followers)
  • GitHub (Repository)
  • GitHub (Gists)
Looking to publish sponsored article on our website?

Contact us

Recent Posts

  • Is it Good to Run Apache Web server and MySQL Database on Separate Cloud Servers? March 27, 2023
  • Advantages of Cloud Server Over Dedicated Server for Hosting WordPress March 26, 2023
  • Get Audiophile-Grade Music on Your Smartphone March 25, 2023
  • Simple Windows Security and Privacy Checklist for 2023 March 24, 2023
  • 7 Best Artificial Intelligence (AI) Software March 24, 2023

About This Article

Cite this article as: Abhishek Ghosh, "Install Elastic Stack on Ubuntu 16.04, CentOS 7 Single Cloud Server," in The Customize Windows, January 30, 2017, March 28, 2023, https://thecustomizewindows.com/2017/01/install-elastic-stack-on-ubuntu-16-04-centos-7-single-cloud-server/.

Source:The Customize Windows, JiMA.in

PC users can consult Corrine Chorney for Security.

Want to know more about us? Read Notability and Mentions & Our Setup.

Copyright © 2023 - The Customize Windows | dESIGNed by The Customize Windows

Copyright  · Privacy Policy  · Advertising Policy  · Terms of Service  · Refund Policy

We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept”, you consent to the use of ALL the cookies.
Do not sell my personal information.
Cookie SettingsAccept
Manage consent

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.
CookieDurationDescription
cookielawinfo-checkbox-analytics11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional11 monthsThe cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
viewed_cookie_policy11 monthsThe cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.
Functional
Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.
Performance
Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.
Analytics
Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.
Advertisement
Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.
Others
Other uncategorized cookies are those that are being analyzed and have not been classified into a category as yet.
SAVE & ACCEPT