July, 2020 ( Kafka Consumer Group Monitoring )

Consumer Group Monitoring is very Important because it provides stats about consumer applications and how far aka lag the application is from actual stream of data.

First: Yahoo Kafka Manager aka CMAK

There is a minor problem with CMAK, It will only show you the latest stats, you can never compare historical results. If you only want to know the latest stats about the consumers then this is one of the best tools for Apache Kafka Consumer Group Monitoring.

**Second: **Custom Monitoring Script ( Python 3 Based )

With this tool my architecture for monitoring becomes very consistent, Python Script for Metric Extraction + Splunk for Indexing & Dashboard.

We need a python package called kafka-python because when we a create consumer using the above library it allows consumer to fetch its stats as well but Same things is not supported in Confluent-Kafka-python Package for now.

#metrics #python #kafka #kafka-consumer-group #monitoring