How to install Apache Kafka Distributed Streaming Platform on Ubuntu

Apache Kafka is a distributed streaming platform developed by the Apache Software Foundation and written in Java and Scala. Apache Kafka was originally developed by LinkedIn and was opened in 2011.

Apache Kafka is used to create a real-time streaming data pipeline that reliably collects data between system and applications. Provides unified, high-throughput and low-latency data processing in real time.

In this tutorial, we'll show you how to install and configure Apache Kafka step-by-step on Ubuntu 18.04. This guide will cover the installation and configuration of Apache Kafka and Apache Zookeeper.

Prerequisites

• Ubuntu 18.04
• Root privileges

What we will do?

1. Install Java OpenJDK 8
2. Install Apache Zookeeper
3. Download and configure Apache Kafka
4. Configure Apache Kafka and Zookeeper as a service
5. analysis

Step 1: Install Java OpenJDK 8

Apache Kafka was written in Java and Scala, so you need to install java on the server.

Before installing any package, update the repository and update all packages.

sudo apt update
sudo apt upgrade

Now install Java OpenJDK 8 from the Ubuntu repository using the apt command below.

sudo apt installs openjdk-8-jdk -y

At the end of the installation, check the installed java version.

java -version

Now you will see Java OpenJDK 8 installed on Ubuntu 18.04.

Step 2: Install Apache Zookeeper

Apache Kafka uses zookeeper for the election controller, cluster membership and argument configuration. Zookeeper is a distributed configuration and synchronization service.

In this step, we will install Zookeeper from the Ubuntu repository.

Run the apt command below.

sudo apt install zookeeperd -y

Wait until the installation is complete.

Step 3: download and configure Apache Kafka

In this step we will install Apache Kafka using the binary files that can be downloaded from the Kafka website. We will install and configure apache Kafka and run as a non-root user.

Add a new user called & # 39; kafka & # 39 ;.

useradd -d / opt / kafka -s / bin / bash kafka
passwd kafka

Now go to the & # 39; / opt & # 39; directory and download the Apache Kafka binaries using wget.

cd / opt
wget http://www-eu.apache.org/dist/kafka/2.0.0/kafka_2.11-2.0.0.tgz

Now create a new kafka directory.

mkdir -p / opt / kafka

Extract the kafka _ * file. Tar.gz in the & # 39; kafka & # 39; directory and change the owner of the directory in the user and in the & # 39; kafka & # 39; group.

tar -xf kafka_2.11-2.0.0.tgz -C / opt / kafka –strip-components = 1
sudo chown -R kafka: kafka / opt / kafka

Now log in to the "kafka" user and change the server.properties configuration.

su – kafka
vim config / server.properties

Paste the following configuration to the end of the line.

delete.topic.enable = true

Save and exit.

The Apache Kafka configuration has been completed.

Step 4: Configure Apache Kafka and Zookeeper as services

In this step, we will configure Apache Kafka as a service and configure the customs service configuration for the zoo keeper.

Go to the & # 39; / lib / systemd / system & # 39; directory and create a new service file & # 39; zookeeper.service & # 39 ;.

cd / lib / systemd / system /
vim zookeeper.service

Paste the configuration below.

[Unit]



Requires = network.target remote-fs.target
After = network.target remote-fs.target

[Service]
Type = simple
User = kafka
ExecStart = / opt / kafka / bin / zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties
ExecStop = / opt / Kafka / bin / zookeeper-server-stop.sh
Restart = on-abnormal

[Install]
WantedBy = multi-user.target

Save and exit.

Now create the Apache Kafka service file & # 39; kafka.service & # 39 ;.

vim kafka.service

Paste the configuration below.

[Unit]



Requires = zookeeper.service
Next = zookeeper.service

[Service]
Type = simple
User = kafka
ExecStart = / bin / sh -c & # 39; /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties'
ExecStop = / opt / Kafka / bin / kafka-server-stop.sh
Restart = on-abnormal

[Install]
WantedBy = multi-user.target

Save and exit.

Reload the system manager configuration.

demon systemctl-reload

Now it starts the Apache Zookeeper and Apache Kafka services.

systemctl start zookeeper
systemctl enable zookeeper

systemctl starts kafka
systemctl enables kafka

The apache zookeeper and Kafka are active and functioning.

Zookeeper that runs under the door & # 39; 2181 & # 39; and Kafka on port '9092', check it using the netstat command below.

netstat -plntu

Step 5: Apache Kafka test

Log in to the user & # 39; kafka & # 39; and go to the & # 39; bin / & # 39; directory.

su – kafka
Trash can /

Now create a new topic called & # 39; HakaseTesting & # 39; using the executable file & # 39; kafka-topics.sh & # 39 ;.

./kafka-topics.sh –create –zookeeper localhost: 2181
–replication-factor 1 –partitions 1
–topic HakaseTesting

Run "kafka-console-producer.sh" with the "HakaseTesting" topic.

./kafka-console-producer.sh –broker-list localhost: 9092
–topic HakaseTesting

Now open a new terminal and log in to the server, then log in to the user & # 39; kafka & # 39 ;.

Run "kafka-console-consumer.sh" for the "HakaseTesting" topic.

./kafka-console-consumer.sh –bootstrap-server localhost: 9092
–topic HakaseTesting – from the beginning

And when you type any input from the shell & # 39; kafka-console-producer.sh & # 39 ;, you'll get the same result on the shell & # 39; kafka-console-consumer.sh & # 39 ;.

The installation and configuration of Apache Kafka on Ubuntu 18.04 have been completed successfully.

Reference