Over the course of the first two parts of this blog series, we setup a single-node Kafka cluster on Kubernetes, secured it using TLS encryption and accessed the broker using both internal and external clients. Let’s keep iterating! In this post, we will continue the Kafka on Kubernetes journey with Strimzi and cover:

• How to apply different authentication types: TLS and SASL SCRAM-SHA-512
• Use Strimzi Entity operator to manage Kafka users and topics
• How to configure Kafka CLI and Go client applications to securely connect to the Kafka cluster

The code is available on GitHub - https://github.com/abhirockzz/kafka-kubernetes-strimzi/

What Do I Need to Go Through This Tutorial?

kubectl - https://kubernetes.io/docs/tasks/tools/install-kubectl/

I will be using Azure Kubernetes Service (AKS)to demonstrate the concepts, but by and large it is independent of the Kubernetes provider (e.g. feel free to use a local setup such as minikube). If you want to use AKS, all you need is a Microsoft Azure account which you can get for FREE if you don’t have one already.

I will not be repeating some of the common sections (such as Installation/Setup (Helm, Strimzi, Azure Kubernetes Service), Strimzi overview) in this or subsequent part of this series and would request you to refer to part one.

Create a Kafka Cluster With TLS Authentication

To enforce 2-way mutual TLS auth, all we need to do is tweak the Strimzi Kafka resource. I am highlighting the key part below. The other parts remain the same (here is the manifest from part 2) i.e. single node Kafka and Zookeeper, ephemeral storage along with TLS encryption

Java

1       external:
2
        type: loadbalancer
3
        tls: true
4
        authentication:
5
          type: tls

All we did is all the tls authentication type as a property of the external listener. In addition to this, we also include the entityOperator configuration as such:

Java

1
  entityOperator:
2
    userOperator: {}
3
    topicOperator: {}

This activates the Strimzi Entity Operator which in turn comprises of the Topic Operator and User Operator. Just as the Kafka CRD allows you to control Kafka clusters on Kubernetes, a Topic Operator allows you to manage topics in a Kafka cluster through a custom resource called KafkaTopic i.e. you can create, delete and update topics in your Kafka cluster.

The goal of the User Operator is to make Kafka user management easier with help of a KafkaUser CRD. All you do is create instances of KafkaUser CRDs and Strimzi takes care of the Kafka specific user management parts

Read more about Entity Operator here ; background-color: transparent;">https://strimzi.io/docs/operators/master/using.html#assembly-kafka-entity-operator-deployment-configuration-kafka

We will dive into the practical bit of these two operators in upcoming sections.

To create the Kafka cluster:

Java

kubectl apply -f https://raw.githubusercontent.com/abhirockzz/kafka-kubernetes-strimzi/master/part-3/kafka-tls-auth.yaml

#tutorial #big data #docker #kubernetes #kafka #cncf