How to Install, Configure Elasticsearch with Apache Hadoop

There is reason why we compared Elasticsearch with Apache Hadoop. Here is How To How to Install, Configure Elasticsearch with Apache Hadoop, Flume, Kibana. Also We Provided Links to Official Configuration. Before running the commands, we will suggest to read the text under the next sub header.

 

README To Install, Configure Elasticsearch with Apache Hadoop

 

Previously we have published few important guides. Even you are used for few months with Apache Hadoop, it will not harm to read the above links. They have more textual information more than mere commands to install.

1. Installing Apache Hadoop
2. Installing Apache Spark
3. Installing Fluentd
4. Installing Elastic Stack / ELK Stack

 

How to Install, Configure Elasticsearch with Apache Hadoop

 

Minimum needs are HDP 2.0 or HDP Sandbox for HDP 2.0 on CentOS. In other words, follow previous guide to install Hadoop which is on numbered one on previous list of guides. Now we need to work with Java and PATH:

Next we need to install Flume :

Next, we need to install Elasticsearch, which is on numbered four on previous list of guides. We need to open and modify the file elasticsearch.yml :

These are things you need to modify :

cluster.name: “logsearch”
node.name: “node1”
node.master: true
node.data: true
index.number_of_shards: 5
index.number_of_replicas : 1
path.data: /data1,/data2,/data3,/data4
discovery.zen.minimum_master_nodes: 1
discovery.zen.ping.timeout: 3s
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: [“host1”, “host2:port”]

Log location is at /var/log/elasticsearch. You can CD, open and adjust later. We can control Elasticsearch with usual commands :

Next we need to install Kibana, which is on numbered four on previous list of guides. Update the logstash index pattern to Flume supported index pattern. Under app/dashboards/logstash.json the entries [logstash-]YYYY.MM.DD need to be separated by dash – [logstash-]YYYY-MM-DD. Now, we need to work with Flume :

We can update Flume configuration to use a local file and index into Elasticsearch in logstash format. But in real cases, Flume Log4j Appender, Syslog TCP Source, Flume Client SDK, Spool Directory Source are used. The below is for Flume configuration to use a local file. You must understand what you are doing. You are only testing.

agent.sources = tail
agent.channels = memoryChannel
agent.channels.memoryChannel.type = memory
agent.sources.tail.channels = memoryChannel
agent.sources.tail.type = exec
agent.sources.tail.command = tail -F /tmp/es_log.log
agent.sources.tail.interceptors=i1 i2 i3
agent.sources.tail.interceptors.i1.type=regex_extractor
agent.sources.tail.interceptors.i1.regex = (\w.*):(\w.*):(\w.*)\s
agent.sources.tail.interceptors.i1.serializers = s1 s2 s3
agent.sources.tail.interceptors.i1.serializers.s1.name = source
agent.sources.tail.interceptors.i1.serializers.s2.name = type
agent.sources.tail.interceptors.i1.serializers.s3.name = src_path
agent.sources.tail.interceptors.i2.type=org.apache.flume.interceptor.TimestampInterceptor$Builder
agent.sources.tail.interceptors.i3.type=org.apache.flume.interceptor.HostInterceptor$Builder
agent.sources.tail.interceptors.i3.hostHeader = host
agent.sinks = elasticsearch
agent.sinks.elasticsearch.channel = memoryChannel
agent.sinks.elasticsearch.type=org.apache.flume.sink.elasticsearch.ElasticSearchSink
agent.sinks.elasticsearch.batchSize=100
agent.sinks.elasticsearch.hostNames = your.IP.here:9300
agent.sinks.elasticsearch.indexName = logstash
agent.sinks.elasticsearch.clusterName = logsearch
agent.sinks.elasticsearch.serializer = org.apache.flume.sink.elasticsearch.ElasticSearchLogStashEventSerializer

Now, create :

open :

populate with this kind of things :

Restart Flume :

Frankly, this is a typical setup. You need to read :

for case specific setup and usage.