Install Apache Kafka on Ubuntu 16.04 : Single Cloud Server

Apache Kafka is a stream processing platform which aims to provide a low-latency platform for handling real-time data feeds. Its storage is a massively scalable pub/sub message queue architected as a distributed transaction log making it valuable to process streaming data. Kafka can connect to external systems for data import/export. Apache Kafka is a part of Big Data analysis too. Here are the steps on how to install Apache Kafka on Ubuntu 16.04 running one single cloud server instance. We have a list of tutorials on Big Data cloud tools. But Kafka is preferred among many of the tools for various reasons. The way of installation of Apache Kafka is more closer to installation of other Apache Big Data tools.

 

Install Apache Kafka on Ubuntu 16.04

 

Apache Kafka needs a Java runtime environment and a user with sudo privilege. In the example, we will use the user name kafka. The command adduser will automatically create the user, initial group, and home directory. Run these commands one by one :

Now we will set the user for sudo permission to allow to be in suoders list :

Type exit to exit the session and run command SSH with that username :

After login, type :

Now you’ll become root user. Now, you may be in need in future to close the way for kafka to SSH. Open this file :

You’ll add entry of kafka to SSH if needed by having syntax like this :

save the file. Run these commands :

During initial installation do not do the changes in /etc/ssh/sshd_config. You’ll install the software by SSH to the system as any permitted user and run su kafka to become kafka.

Next we will install Java. There are two ways, one is using oracle-java8 :

Another is the default Java Run Time provided by apt :

Follow the way you want. Verify that JDK 8 is installed properly

Kafka cluster depends on ZooKeeper to perform operations such as detecting failed nodes. Kafka brokers need it to form a cluster, and the configuration is stored in ZK nodes. Newer versions of Kafka have decoupled the clients to consumers and producers. We still need ZooKeeper to run Kafka brokers. Once we install Kafka, we can use the ZooKeeper available with Kafka. But we will use ZooKeeper package that is available in Ubuntu’s repository.

ZooKeeper will be started as a daemon automatically. By default, it will listen on port 2181 :

Now go to :

We will install the current stable version from :

Version 0.10.2.1 is latest at the time of publication. We are showing example with one mirror, browse here on browser:

and click 0.10.2.1 link, these are the files :

You know that version 0.10.2.1 is latest. So here /0.10.2.1/kafka-0.10.2.1-src.tgz will be your file.

Open this file and look for delete.topic.enable :

Make it like this :

Save the file. Start Kafka by running kafka-server-start.sh script :

Now, we can check listening ports of ZooKeeper and Kafka :

Expected output :

To stop, you’ll need to run script kafka-server-stop.sh in the same way like we started above. Of course you can create topic :

 

Install Apache Kafka on Ubuntu 16.04 : Configure With Spark

 

I am showing additional steps :

Add spark configuration to profile :

Source it :

Test configuration :

Rest you should read from :