Apache Kafka is ruling in the world of Big Data. It is just not a messaging queue but a full-fledged event streaming platform. We have looked through the basic idea of Kafka and what makes it faster than any other messaging queue. You can read about it from my previous blog. Also, we looked through Partitions, Replicas, and ISR. We are now ready for our next learning- Rebalancing. Often heard, but never bothered to look at what is going on under the hood. Let’s find out!

Rebalancing means to balance out. Working on a real-time cluster comes with a bunch of problems. Nothing different with Apache Kafka. There will be scenarios when your application shuts down, or one of the nodes is unreachable, consumers are being added or removed from the consumer group, etc. The aim is to rebalance the uneven load in the cluster.

#apache kafka #big data and fast data #scala #tech blogs #consumer rebalancing