Apache Kafka is one of the most popular event-based distributed streaming platforms. LinkedIn first developed it, and technology leaders such as Uber, Netflix, Slack, Coursera, Spotify, and others currently use it.
Though very powerful, Kafka is equally complex and requires a highly available robust platform to run on. Most of the time, engineers struggle to feed and water the Kafka servers, and standing one up and maintaining it is not a piece of cake.
With microservices in vogue and most companies adopting distributed computing, standing up Kafka as the core messaging backbone has its advantages. Kubernetes is a popular choice to run container-based microservices, and using Kafka as an eventing platform is another.
If you are running your microservices in Kubernetes, it will make sense to run your Kafka cluster within Kubernetes to take advantage of its in-built resilience and high-availability. The Kubernetes pods can also easily interact with the Kafka pods within the cluster using the inbuilt Kubernetes service discovery.
Let’s take a peek at how to build a distributed Kafka cluster on Kubernetes through a hands-on exercise. We will use Helm charts and stateful sets. Kubernetes will dynamically request persistent volumes from the cloud provider and use them to persist data.
Prerequisites
You will need a running Kubernetes cluster for this exercise. I have used Google Kubernetes Engine 1.16.13-gke.1.
Install Helm
Helm is the package manager for Kubernetes. It is one of the most popular tools allowing people to version and share their manifests. Kafka also contains production-tested Helm charts that will help you install a production-ready cluster. You can also customise the Helm installation according to your requirements.
Helm v3 does not require Tiller and therefore is a simple binary download and path setup.
wget https://get.helm.sh/helm-v3.3.0-linux-amd64.tar.gz
tar -zxvf helm-v3.3.0-linux-amd64.tar.gz
sudo cp -a linux-amd64/helm /usr/local/bin/helm
chmod +x /usr/local/bin/helm
#kubernetes #programming #devops #containers #kafka