There are many ways of server log data collection and analysis. Many of them we have discussed and provided tutorials like Installing fluentd Agent, Installing Elastic Stack, Using IBM Demo Cloud To Process Server Log in Hadoop, Pig. Most are related with Big Data Analysis. Apache Flume Can Be Used For Collecting, Aggregating, Moving Log Data For Analytic Applications. Here is How To Install Apache Flume on Ubuntu 16.04 Server. Here is official website of Apache Flume :
|
1 |
http://flume.apache.org/ |
We are not going to detailed discussion around how to setup real time Apache HTTP web server log analysis system :
|
1 |
https://blog.cloudera.com/blog/2015/02/how-to-do-real-time-log-analytics-with-apache-kafka-cloudera-search-and-hue/ |
How To Install Apache Flume on Ubuntu 16.04 Server
You need Apache Hadoop to be pre-installed. In our other guides we have written how to install and configure Apache Hadoop on single cloud server.
---
We will get latest version of Apache Flume here :
|
1 |
http://archive.apache.org/dist/flume/ |
I am providing commands for overview of total project, you need not to run commands for Hadoop, JDK if already installed :
|
1 2 3 |
apt update apt upgrade apt install openjdk-7-jdk |
Now we need to create Flume’s directory and change the ownership and permissions of the directory :
|
1 2 3 |
mkdir -p /usr/local/flume sudo chown -R hduser /usr/local/flume sudo chmod -R 755 /usr/local/flume |
If you have downloaded Apache Flume tar.gz file is in /home/hduser/downloads directory, then :
|
1 2 3 |
cd /home/hduser/downloads tar -xzvf apache-flume-1.8.0-bin.tar.gz mv apache-flume-1.8.0-bin/* /usr/local/flume |
Now, edit $HOME/.bashrc file :
|
1 2 3 4 5 6 |
nano $HOME/.bashrc ## Add the below lines export FLUME_HOME=/usr/local/flume export FLUME_CONF_DIR=$FLUME_HOME/conf export FLUME_CLASSPATH=$FLUME_CONF_DIR export PATH=$PATH:$FLUME_HOME/bin |
Reload .bashrc :
|
1 |
source $HOME/.bashrc |
Now, navigate to /usr/local/flume/conf :
|
1 2 3 4 |
cd $FLUME_HOME/conf ls -al cp flume-env.sh.template flume-env.sh nano flume-env.sh |
Add java path to flume-env.sh :
|
1 |
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64 |
Change the path according to your version. Now copy flume-conf.properties.template to flume-conf.properties :
|
1 2 3 |
cp flume-conf.properties.template flume-conf.properties # cd to /usr/local/flume/bin cd $FLUME_HOME/bin |
We can verify Installation with this command :
|
1 |
./flume-ng version |
Of course it is easy just because it is part of bigger plan.
Tagged With https://thecustomizewindows com/2018/04/install-apache-flume-ubuntu-16-04-server/ , how to install apa he flume on ubuntu 18 04 , how to upgrade apache flume , how to install flume and variables in ubuntu , configure apache flume for apache webserver , boundfi9 , apache flume installation on windows , apache flume in ubuntu 18 04 , apache flume command line install , ubuntu 16 04 install flume