Kafka Queue system with example

Today, I would like to write something about Kafka Queue system. I would like cover following topics.

1. Brief Introduction of Kafka Queue System.
2. Setting up Kafka and configuring with maven projects.
3. Examples for Kafka Producer and Consumer
4. Gotchas which I faced in configuration.

Introduction:

In today’s world of data, It is necessary to implement the queue system. We can start using simple data structure of queue in our implementation but as our data in queue increases, we need to pull that out as separate service and rely on services such Amazon SQS or Kafka Service. I choose to go with Kafka Service since there is lots of control over configuration and system.

There are some other implementations of queues such as MQTT which are worth doing research, which I haven’t done yet.

Today’s blog will cover the first few basics of Kafka Queue system.

Few terminologies to understand:

Topic: It is the name of the queue.

Partitions: Kafka cluster maintains different logs of each topic. It maintains

Producers: Producer will add the message on the topic of configured partitions.

Consumer: Consumer will read messages from subscribed topics.

Setup & Configuration:

• Download the latest version of Kafka.
• Please follow below commands to untar your downloaded file.

tar -xzvf kafka_2.11-0.10.1.0.tgz cd kafka_2.11-0.10.1.0

• Kafka requires zookeeper server, If you don’t have one you can start using below command

bin/zookeeper-server-start.sh config/zookeeper.properties

• Next step is to start Kafka service. In this example, I working on standalone Kafka server.

bin/kafka-server-start.sh config/server.properties

• Next steps are to create the topic, add the message to the topic(produce) and read the message from the queue(consume)
• Create a topic as test

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test

Producer and Consumer Examples

• I wrote simple Producer and Consumer in Java. Please take a look Git project at Kafka101
• Please run, ProducerDemo to push messages to topic test queue.
• Then following, Please run ConsumerDemo to read messages.

Gotchas?

• Implementing Kafka Consumer, there was the problem with Library kafka_clients latest version 0.10.x.x, KafkaConsumer.poll the method was not implemented which will return always null;
• Please make sure you use 0.9.x.x version of the library or else check with latest version implementation.

Thank you for following steps, I hope this blog and Kafka101 will provide the basic understanding of Kafka queues and its implementation.

Please let me know your comments below.

Originally published at www.sreedharbukya.com on December 11, 2016.