There are many ways of server log data collection and analysis. Many of them we have discussed and provided tutorials like Installing fluentd Agent, Installing Elastic Stack, Using IBM Demo Cloud To Process Server Log in Hadoop, Pig. Most are related with Big Data Analysis. Apache Flume Can Be Used For Collecting, Aggregating, Moving Log Data For Analytic Applications. Here is How To Install Apache Flume on Ubuntu 16.04 Server. Here is official website of Apache Flume :
|
1 |
http://flume.apache.org/ |
We are not going to detailed discussion around how to setup real time Apache HTTP web server log analysis system :
|
1 |
https://blog.cloudera.com/blog/2015/02/how-to-do-real-time-log-analytics-with-apache-kafka-cloudera-search-and-hue/ |
---
How To Install Apache Flume on Ubuntu 16.04 Server
You need Apache Hadoop to be pre-installed. In our other guides we have written how to install and configure Apache Hadoop on single cloud server.
We will get latest version of Apache Flume here :
|
1 |
http://archive.apache.org/dist/flume/ |
I am providing commands for overview of total project, you need not to run commands for Hadoop, JDK if already installed :
|
1 2 3 |
apt update apt upgrade apt install openjdk-7-jdk |
Now we need to create Flume’s directory and change the ownership and permissions of the directory :
|
1 2 3 |
mkdir -p /usr/local/flume sudo chown -R hduser /usr/local/flume sudo chmod -R 755 /usr/local/flume |
If you have downloaded Apache Flume tar.gz file is in /home/hduser/downloads directory, then :
|
1 2 3 |
cd /home/hduser/downloads tar -xzvf apache-flume-1.8.0-bin.tar.gz mv apache-flume-1.8.0-bin/* /usr/local/flume |
Now, edit $HOME/.bashrc file :
|
1 2 3 4 5 6 |
nano $HOME/.bashrc ## Add the below lines export FLUME_HOME=/usr/local/flume export FLUME_CONF_DIR=$FLUME_HOME/conf export FLUME_CLASSPATH=$FLUME_CONF_DIR export PATH=$PATH:$FLUME_HOME/bin |
Reload .bashrc :
|
1 |
source $HOME/.bashrc |
Now, navigate to /usr/local/flume/conf :
|
1 2 3 4 |
cd $FLUME_HOME/conf ls -al cp flume-env.sh.template flume-env.sh nano flume-env.sh |
Add java path to flume-env.sh :
|
1 |
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64 |
Change the path according to your version. Now copy flume-conf.properties.template to flume-conf.properties :
|
1 2 3 |
cp flume-conf.properties.template flume-conf.properties # cd to /usr/local/flume/bin cd $FLUME_HOME/bin |
We can verify Installation with this command :
|
1 |
./flume-ng version |
Of course it is easy just because it is part of bigger plan.
About Abhishek Ghosh
Here’s what we’ve got for you which might like :
Additionally, performing a search on this website can help you. Also, we have YouTube Videos.
Take The Conversation Further ...
We'd love to know your thoughts on this article.
Meet the Author over on Twitter to join the conversation right now!
If you want to Advertise on our Article or want a Sponsored Article, you are invited to Contact us.
Contact Us