Apache Kafka Replica, Partition, Leader and ISR

Let's first start the zookeeper.
open a new command window, and run the following command.

zookeeper-server-start %KAFKA_HOME%\config\zookeeper.properties

zookeeper by default, starts at port 2181.
start the kafka server.
open a new command window, and run the following command.

kafka-server-start %KAFKA_HOME%\config\server.properties

Create new server properties in kafka config folder
broker.id=1
listeners=PLAINTEXT://:9093
log.dirs=/tmp/kafka-logs1
Now we have two brokers available. Broker-0, broker-1

Topic Replication

In Kafka, each broker contains the data. But, if the broker or the machine fails? The data could be loss. Precautionary, Apache Kafka enables a feature of replication to secure data loss even when a broker fails down. To do so, a replication factor is created for the topics contained in any particular broker. A replication factor is the number of copies of data over multiple brokers. The replication factor value should be greater than 1 always (between 2 or 3). This helps to store a replica of the data in another broker from where the user can access it. For example, suppose we have a cluster containing three brokers say Broker 1, Broker 2. A topic, namely Topic-X is split into 4 Partitions with a replication factor of 2.


It is obvious to have confusion when both the actual data and its replicas are present. The cluster may get confuse that which broker should serve the client request. To remove such confusion, the following task is done by Kafka: It chooses one of the broker's partition as a leader, and the rest of them becomes its followers.


Partition 0 has replica of 1 and 0. Leader 1- that means primary copy is with broker 1 and second copy(follower) with broker-0. Partition 1 has replica of 0 and 1. Leader 0- that means primary copy is with broker 0 and second copy(follower) with broker-1. The followers(brokers) will be allowed to synchronize the data. But, in the presence of a leader, none of the followers is allowed to serve the client's request. These replicas are known as ISR(in-sync-replica). So, Apache Kafka offers multiple ISR(in-sync-replica) for the data. Therefore, only the leader is allowed to serve the client request. The leader handles all the read and writes operations of data for the partitions. The leader and its followers are determined by the zookeeper. If the broker holding the leader for the partition fails to serve the data due to any failure, one of its respective ISR replicas will takeover the leadership. Afterward, if the previous leader returns back, it tries to acquire its leadership again.
try to stop broker 0 and see what will happen using desribe command.