This is a step-by-step article that guides you from scratch. I assume you only have access to a vanilla Kubernetes cluster. In this article, I’m using k8s external metrics to autoscales my kafka consumers. There are other articles online that shows you how to use custom metrics instead. I chose to use external metrics since it’s more applicable to real production environment where you might want to scale based on other metrics that are available on Prometheus. For a great tutorial for custom metrics you can go to this medium article.

Outline

There are a few steps we need to do to use HPA with custom and/or external metrics. You can skip these steps if you happen to already have kafka or prometheus running.

1. Deploy Kafka
2. Deploy Prometheus
3. Expose Prometheus and Grafana services
4. Deploy a Kafka consumer application
5. Deploy Prometheus Adapter
6. Deploy HPA

Deploy Kafka

We will use Helm for most of our installation process out of convenience. Run this command on your terminal to install Kafka.

$ helm install \
    --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false \

kp prometheus-community/kube-prometheus-stack

#autoscaling #prometheus #kubernetes #hpa #kafka