How to install kafka - Standalone Installation

How to install kafka - Standalone Installation

As part of the this tutorial, we will be setting up Apache Kafka in a single node. Apache Kafka is an open-source message broker project developed by the Apache Software Foundation written in Scala. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.

Download Kafka

Download Latest Kafka From the official Apache Kafka Website

[root@monitor tmp]# wget http://apachemirror.wuchna.com/kafka/2.4.0/kafka_2.11-2.4.0.tgz
--2020-02-28 14:38:08--  http://apachemirror.wuchna.com/kafka/2.4.0/kafka_2.11-2.4.0.tgz
Resolving apachemirror.wuchna.com (apachemirror.wuchna.com)... 159.65.154.237
Connecting to apachemirror.wuchna.com (apachemirror.wuchna.com)|159.65.154.237|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 70057083 (67M) [application/x-gzip]
Saving to: ‘kafka_2.11-2.4.0.tgz’

100%[==========================================================================================>] 70,057,083  19.6MB/s   in 4.5s

2020-02-28 14:38:12 (15.0 MB/s) - ‘kafka_2.11-2.4.0.tgz’ saved [70057083/70057083]

[root@monitor tmp]#

Install Java

[root@monitor tmp]# yum install java-1.8.0-openjdk
[root@monitor tmp]# yum install java-1.8.0-openjdk-devel

Untar the tarball

[root@monitor tmp]# tar xzvf kafka_2.11-2.4.0.tgz
kafka_2.11-2.4.0/
kafka_2.11-2.4.0/LICENSE
kafka_2.11-2.4.0/NOTICE
kafka_2.11-2.4.0/bin/
kafka_2.11-2.4.0/bin/kafka-delete-records.sh
kafka_2.11-2.4.0/bin/trogdor.sh
kafka_2.11-2.4.0/bin/kafka-preferred-replica-election.sh
kafka_2.11-2.4.0/bin/connect-mirror-maker.sh
kafka_2.11-2.4.0/bin/kafka-console-consumer.sh

Start Zookeeper

Kafka relies on Zookeeper to for configuration information.

[root@monitor tmp]# bin/zookeeper-server-start.sh config/zookeeper.properties

Start the Kafka Servics

[root@monitor tmp]# kafka_2.11-2.4.0/bin/kafka-server-start.sh kafka_2.11-2.4.0/config/server.properties
[2020-02-28 14:41:17,097] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2020-02-28 14:41:17,386] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler)
[2020-02-28 14:41:17,386] INFO starting (kafka.server.KafkaServer)
[2020-02-28 14:41:17,387] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2020-02-28 14:41:17,431] INFO [ZooKeeperClient Kafka server] Initializing a new session to localhost:2181. (kafka.zookeeper.ZooKeeperClient)

 Create Topic

Kafka partitions incoming messages for a topic, and assigns those partitions to the available Kafka brokers. The number of partitions is configurable and can be set per-topic and per-broker. The following command will create a topic test with one partition

[root@monitor kafka_2.11-2.4.0]# bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Created topic test.
[root@monitor kafka_2.11-2.4.0]#

 List Topics

To make sure wheather the topic is created successfully, we can list out the topics hosted by the broker with the following command.

[root@monitor kafka_2.11-2.4.0]# bin/kafka-topics.sh --list --zookeeper localhost:2181
test
[root@monitor kafka_2.11-2.4.0]#

Check more Kafka Articles:

KafkaManager Error: Yikes! Ask timed out
Kafka Manager OutOfMemoryError : java heap space?
Updated: How to Configure kafka Monitoring with JMX Exporter and Prometheus.

kafka connection problem

How to install kafka - Standalone Installation

How to install  Kafka Manager for Managing kafka cluster

How to Install  Confluent  Kafka on Centos & Ubuntu

How to Manually delete Apache Kafka topics

Start Producer

Producers publish messages to Kafka topics and are able to choose which topic, and which partition within the topic, to publish the message to. we will start a producer no

[root@monitor kafka_2.11-2.4.0]# bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
>test
>this is manas
>who are you
>how are you
>^CYou have new mail in /var/spool/mail/root
[root@monitor kafka_2.11-2.4.0]#

Start Consumer

Consumers subscribe to these topics and consume the messages. Lets consume the messages with the following command

[root@monitor kafka_2.11-2.4.0]# bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
test
this is manas
who are you
how are you
^CProcessed a total of 4 messages
You have new mail in /var/spool/mail/root
[root@monitor kafka_2.11-2.4.0]#