Let us consider every topic in Kafka is a table. The table has partitions similarly the topic also has partitions. Every consumer is assigned one or more topic partitions exclusively.
What is a rebalance in Kafka?
The rebalance in Kafka is the re-assignment of partitions among consumers. In a consumer group, the consumer is assigned a partition. When the rebalance happens the group co-ordinator revokes the partitions from the consumer and assigns new partitions.
When a rebalance happens?
- When a consumer JOINS the group
- When a consumer is DEAD
- When a consumer is considered DEAD by the group coordinator. This may happen after a crash or when the consumer is busy with a long-running processing, which means that no heartbeats has been sent in the meanwhile by the consumer to the group coordinator within the configured session interval
- When new partitions are added
- Topic is created or deleted