Kafka Notes

Kafka Architecture


Reference:http://cloudurable.com/blog/kafka-architecture/index.html

Kafka uses ZooKeeper to manage the cluster. ZooKeeper is used to coordinate the brokers/cluster topology. ZooKeeper is a consistent file system for configuration information. ZooKeeper gets used for leadership election for Broker Topic Partition Leaders.


Reference: https://dzone.com/articles/kafka-cluster-1

Apache Kafka Topics

Producer Properties


This is for information only. We can set all this through java program.

Consumer Properties


This is for information only. We can set all this through java program.

Consumer Groups:

  1. Consumers can join a group by using the same group.id.
  2. The maximum parallelism of a group is that the number of consumers in the group ← no of partitions.
  3. Kafka assigns the partitions of a topic to the consumer in a group so that each partition is consumed by exactly one consumer in the group.
  4. Kafka guarantees that a message is only ever read by a single consumer in the group.
  5. Consumers can see the message in the order they were stored in the log.

Reference: https://www.tutorialspoint.com/apache_kafka/apache_kafka_consumer_group_example.htm


Author: Martin Kleppmann

—-
Kafka Tools

Kafka Monitoring Tools: https://docs.lenses.io/1.0/lenses/monitoring/index.html

Quick tool for developers: https://www.conduktor.io/download

Kafka Installation

Start Zookeeper
E:\software\kafka_2.12-2.3.0\bin\windows>zookeeper-server-start.bat ../../config/zookeeper.properties

Start Kafka Server
E:\software\kafka_2.12-2.3.0\bin\windows>kafka-server-start.bat ../../config/server.properties

Start Conduktor to monitor and check
Download and double click jar

Setting up multi broker
https://kafka.apache.org/quickstart

Setting up multi cluster
https://dzone.com/articles/how-to-setup-kafka-cluster

Kafka Interview Questions:
https://www.edureka.co/blog/interview-questions/top-apache-kafka-interview-questions-for-beginners/

Batch ETL: https://www.infoq.com/articles/batch-etl-streams-kafka/

Apache Kafka vs. Integration Middleware (MQ, ETL, ESB)
https://dzone.com/articles/apache-kafka-vs-integration-middleware-mq-etl-esb

AWS Cloud KAFKA
https://blog.mitocgroup.com/my-architecture-apache-kafka-by-confluent-on-aws-cloud-ee5303c49f2f

https://docs.confluent.io/current/platform.html

#kafka