Kafka Notes

Kafka Architecture


Kafka uses ZooKeeper to manage the cluster. ZooKeeper is used to coordinate the brokers/cluster topology. ZooKeeper is a consistent file system for configuration information. ZooKeeper gets used for leadership election for Broker Topic Partition Leaders.

Reference: https://dzone.com/articles/kafka-cluster-1

Apache Kafka Topics

Producer Properties

This is for information only. We can set all this through java program.

Consumer Properties

This is for information only. We can set all this through java program.

Consumer Groups:

  1. Consumers can join a group by using the same group.id.
  2. The maximum parallelism of a group is that the number of consumers in the group ← no of partitions.
  3. Kafka assigns the partitions of a topic to the consumer in a group so that each partition is consumed by exactly one consumer in the group.
  4. Kafka guarantees that a message is only ever read by a single consumer in the group.
  5. Consumers can see the message in the order they were stored in the log.

Reference: https://www.tutorialspoint.com/apache_kafka/apache_kafka_consumer_group_example.htm

Author: Martin Kleppmann

Kafka Tools

Kafka Monitoring Tools: https://docs.lenses.io/1.0/lenses/monitoring/index.html

Quick tool for developers: https://www.conduktor.io/download

Kafka Installation

Start Zookeeper
E:\software\kafka_2.12-2.3.0\bin\windows>zookeeper-server-start.bat ../../config/zookeeper.properties

Start Kafka Server
E:\software\kafka_2.12-2.3.0\bin\windows>kafka-server-start.bat ../../config/server.properties

Start Conduktor to monitor and check
Download and double click jar

Setting up multi broker

Setting up multi cluster

Kafka Interview Questions:

Batch ETL: https://www.infoq.com/articles/batch-etl-streams-kafka/

Apache Kafka vs. Integration Middleware (MQ, ETL, ESB)