• Home
  • Archive
  • Tools
  • Contact Us

The Customize Windows

Technology Journal

  • Cloud Computing
  • Computer
  • Digital Photography
  • Windows 7
  • Archive
  • Cloud Computing
  • Virtualization
  • Computer and Internet
  • Digital Photography
  • Android
  • Sysadmin
  • Electronics
  • Big Data
  • Virtualization
  • Downloads
  • Web Development
  • Apple
  • Android
Advertisement
You are here: Home » Install fluentd Agent : Log Data Collection For Hadoop

By Abhishek Ghosh January 23, 2017 8:57 am Updated on January 23, 2017

Install fluentd Agent : Log Data Collection For Hadoop

Advertisement

Our previous two guides were on how to install Apache Hadoop and how to install Apache Spark. Next step technically is to get data from source like server log files. Here is How To Install fluentd Agent on Ubuntu 16.04 As Intermediate Step of Log Data Collection For Hadoop. We Can Receive Data on Laptop. This step is on your servers to monitor, not on other servers.

 

Install fluentd Agent : Preface For Log Data Collection For Hadoop

 

Here is official website of fluentd :

Vim
1
http://www.fluentd.org

It is a beautiful software written in Ruby. Fluentd is a Big Data tool and can work with unstructured data in real time. Exactly like an another tool Kafka, it analyzes the event logs, application logs, and clickstreams. It can simply output the collected data over HTTPS with settings to secure the transport. It is better to get used with the usage of such tools before jumping to analyse data with Apache Hadoop, Spark or plain Elastic Search. You must not use the public IP of the web server associated with domain under question for higher security except for debugging, initial setup etc.

Advertisement

---

Install fluentd Agent - Log Data Collection For Hadoop

The above illustration shows our plan of tutorials. In future we will stream this data to :

  1. Local computer like MacBook Pro
  2. Server running Big Data Specific tool like Apache Hadoop, Spark alone or in combination.
  3. Server running primarily basic analysis tools like ElasticSearch rather ELK Stack.
  4. Server running Big Data Specific tool like Apache Hadoop, Spark alone or in combination and ElasticSearch and Kibana.

 

This way of learning will help you to never fumble with – “will I use ElasticSearch or Apache Hadoop?”. Elastic analytics stack is gaining popularity for various reasons. ElasticSearch uses JSON based query language which is much easier to master than Hadoop’s MapReduce. Also the Developers are more comfortable maintaining ElasticSearch instance over Hadoop. Of course, this architecture would have data loss issue on the ElasticSearch side. You need Treasure Data in such case in front of ElasticSearch to “buffer” it. We are simply using fluentd to maintain backward and forward compatibility without disturbing the main server too much. Where there’s a shell, there is a way. The way is for the hackers too.

 

Steps To Install fluentd Agent on Ubuntu 16.04

 

It is recommended to set up ntpd on the main server under question of logging to prevent invalid timestamps in logs. Increase the maximum number of file descriptors, run :

Vim
1
2
3
4
su - root
ulimit -n
ulimit -Hn
ulimit -Sn

Expect value like 65535 against ulimit -n. If you get value like 1024, open :

Vim
1
nano /proc/sys/fs/file-max

add/modify :

/proc/sys/fs/file-max
Vim
1
sysctl -w fs.file-max=100000

Exit session after saving and reboot. open :

Vim
1
nano /etc/sysctl.conf

add/modify :

/etc/sysctl.conf
Vim
1
fs.file-max = 100000

Exit session after saving and reboot. open :

Vim
1
nano /etc/security/limits.conf

add/modify :

/etc/security/limits.conf
Vim
1
2
root soft nofile 65536
root hard nofile 65536

Exit session after saving and reboot. open :

Vim
1
nano /etc/sysctl.conf

add/modify :

/etc/sysctl.conf
Vim
1
2
3
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 10240    65535

Exit session after saving and reboot. For Ubuntu 16.04, run this :

Vim
1
2
3
4
5
6
7
8
9
10
cd ~
sudo -k
curl https://packages.treasuredata.com/GPG-KEY-td-agent | apt-key add -
echo "deb http://packages.treasuredata.com/2/ubuntu/xenial/ xenial contrib" > /etc/apt/sources.list.d/treasure-data.list
apt-get update
apt-get install -y td-agent
/opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-elasticsearch
/opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-record-modifier
/opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-secure-forward
usermod -G adm td-agent

You can control the software with :

Vim
1
2
3
4
/etc/init.d/td-agent start
/etc/init.d/td-agent stop
/etc/init.d/td-agent restart
/etc/init.d/td-agent status

Carefully run this command after changing IP, port :

Vim
1
nc -vzw1 log-ip  24224

log-ip is your log server ip, 24224 is your port.

open :

Vim
1
nano /etc/td-agent/td-agent.conf

add/modify :

/etc/td-agent/td-agent.conf
Vim
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
###############################################
##
## Forwarder
## Section 1 :: Sending Logs
###############################################
## Secure forward logs to fluentd server
<match **>
  type forward
  send_timeout 10s
  recover_wait 5s
  heartbeat_interval 1s
  phi_threshold 16
  hard_timeout 60s
  <server>
    host log-ip
    port 24224
    weight 20
  </server>
</match>
## Section 2 :: Reading Logs
###############################################
## tail nginx logs
## add tag nginx.access
#
<source>
  type tail
  format nginx
  path /var/log/nginx/access.log
  tag nginx.access
  # Select a file to store offset position
  pos_file /tmp/nginx-access-td-agent
</source>
################################################
## tail nginx logs
## add tag nginx.error
<source>
  type tail
  format /^(?<time>[^ ]+ [^ ]+) \[(?<log_level>.*)\] (?<pid>\d*).(?<tid>[^:]*): (?<message>.*)$/
  path /var/log/nginx/error.log
  tag nginx.error
  # Select a file to store offset position
  pos_file /tmp/nginx-error-td-agent
  time_format %Y/%m/%d %H:%M:%S
</source>
################################################
## tail syslog
## add tag syslog
<source>
  type tail
  format syslog
  path /var/log/syslog
  tag syslog
  pos_file /tmp/syslog-td-agent.tmp
<source>

Run :

Vim
1
2
/etc/init.d/td-agent restart
/etc/init.d/td-agent status

You can run cURL to POST :

Vim
1
curl -X POST -d 'json={"json":"message"}' http://localhost:24224/debug.test

There are more documentation on port part :

Vim
1
http://docs.fluentd.org/articles/out_secure_forward

Here is plugin management :

Vim
1
http://docs.fluentd.org/articles/plugin-management

fluentd has one of the worst documentation on this earth. I ran the command many steps before :

Vim
1
/opt/td-agent/embedded/bin/fluent-gem install fluent-plugin-secure-forward

else you would get error. They actually have lot of stuffs :

Vim
1
https://github.com/fluent

including web UI.

Tagged With Fluentd , fluentd installation step by step to read logs in ubuntu , fluentd register on hadoop , fluentd server log agent , from windows td-agent to fluentd , how to setup fluentd as a service on windows , paperuri:(da9ad54f0ba8fad911ea136b4ff7537a) , td-agent conf window server logs , td-agent nginx_access

This Article Has Been Shared 736 Times!

Facebook Twitter Pinterest

Abhishek Ghosh

About Abhishek Ghosh

Abhishek Ghosh is a Businessman, Surgeon, Author and Blogger. You can keep touch with him on Twitter - @AbhishekCTRL.

Here’s what we’ve got for you which might like :

Articles Related to Install fluentd Agent : Log Data Collection For Hadoop

  • Cloud Computing Business Model

    Cloud Computing Business Model, often we hear about this phrase or its shorter phrase Cloud Computing Biz Model. What is exactly this Business Model is?

  • Cloud Sound and Cloud Music Technologies

    Cloud Sound and Cloud Music Technologies are one of the hottest technologies on discussion right now. In this article we have focused on the basic offerings of Cloud Sound and Cloud Music Technologies.

  • Service Level Agreement and Cloud Computing Services

    Service Level Agreement is an important part of Cloud Computing Services like SaaS, PaaS and IaaS. As a customer of any Cloud Computing Services, you must understand this Service Level Agreement.

  • Big Data : Companies are Increasingly Demanding

    Big Data will reach revenues of close to $34 billion of IT spending in 2013, this what Gartner is predicting. Analysis if of global market interested in cloud.

  • WordPress Lost Post Recovery Options on Cloud

    WordPress Lost Post Recovery Options Are Not Less in Number on Cloud, Even Without Backup Failure. If Your FTP Server is Running, Data Can Be Recovered.

Additionally, performing a search on this website can help you. Also, we have YouTube Videos.

Take The Conversation Further ...

We'd love to know your thoughts on this article.
Meet the Author over on Twitter to join the conversation right now!

If you want to Advertise on our Article or want a Sponsored Article, you are invited to Contact us.

Contact Us

Subscribe To Our Free Newsletter

Get new posts by email:

Please Confirm the Subscription When Approval Email Will Arrive in Your Email Inbox as Second Step.

Search this website…

 

Popular Articles

Our Homepage is best place to find popular articles!

Here Are Some Good to Read Articles :

  • Cloud Computing Service Models
  • What is Cloud Computing?
  • Cloud Computing and Social Networks in Mobile Space
  • ARM Processor Architecture
  • What Camera Mode to Choose
  • Indispensable MySQL queries for custom fields in WordPress
  • Windows 7 Speech Recognition Scripting Related Tutorials

Social Networks

  • Pinterest (24.3K Followers)
  • Twitter (5.8k Followers)
  • Facebook (5.7k Followers)
  • LinkedIn (3.7k Followers)
  • YouTube (1.3k Followers)
  • GitHub (Repository)
  • GitHub (Gists)
Looking to publish sponsored article on our website?

Contact us

Recent Posts

  • Cyberpunk Aesthetics: What’s in it Special January 27, 2023
  • How to Do Electrical Layout Plan for Adding Smart Switches January 26, 2023
  • What is a Data Mesh? January 25, 2023
  • What is Vehicular Ad-Hoc Network? January 24, 2023
  • Difference Between Panel Light, COB Light, Track Light January 21, 2023

About This Article

Cite this article as: Abhishek Ghosh, "Install fluentd Agent : Log Data Collection For Hadoop," in The Customize Windows, January 23, 2017, January 30, 2023, https://thecustomizewindows.com/2017/01/install-fluentd-agent-log-data-collection-for-hadoop/.

Source:The Customize Windows, JiMA.in

PC users can consult Corrine Chorney for Security.

Want to know more about us? Read Notability and Mentions & Our Setup.

Copyright © 2023 - The Customize Windows | dESIGNed by The Customize Windows

Copyright  · Privacy Policy  · Advertising Policy  · Terms of Service  · Refund Policy

We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept”, you consent to the use of ALL the cookies.
Do not sell my personal information.
Cookie SettingsAccept
Manage consent

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to function properly. These cookies ensure basic functionalities and security features of the website, anonymously.
CookieDurationDescription
cookielawinfo-checkbox-analytics11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional11 monthsThe cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance11 monthsThis cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
viewed_cookie_policy11 monthsThe cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.
Functional
Functional cookies help to perform certain functionalities like sharing the content of the website on social media platforms, collect feedbacks, and other third-party features.
Performance
Performance cookies are used to understand and analyze the key performance indexes of the website which helps in delivering a better user experience for the visitors.
Analytics
Analytical cookies are used to understand how visitors interact with the website. These cookies help provide information on metrics the number of visitors, bounce rate, traffic source, etc.
Advertisement
Advertisement cookies are used to provide visitors with relevant ads and marketing campaigns. These cookies track visitors across websites and collect information to provide customized ads.
Others
Other uncategorized cookies are those that are being analyzed and have not been classified into a category as yet.
SAVE & ACCEPT