In this ultimate guide I will give you a simple step-by-step tutorial on installing Kafka Docker on Kubernetes. This post includes a complete video walk-through.
There has been a lot of interest lately about deploying Kafka to a Kubernetes cluster. If you are wanting to take the deep dive yourself then you found the right article. Now that we have Kafka Docker, deploying a Kafka cluster to Kubernetes is a snap.
Kafka relies on ZooKeeper to keep track of its configuration including what topics are available.
Before we deploy Kafka we need to deploy ZooKeeper.
Create a file called zookeeper.yml and add these contents:
--- kind: Deployment apiVersion: extensions/v1beta1 metadata: name: zookeeper-deployment-1 spec: template: metadata: labels: app: zookeeper-1 spec: containers: - name: zoo1 image: digitalwonderland/zookeeper ports: - containerPort: 2181 env: - name: ZOOKEEPER_ID value: "1" - name: ZOOKEEPER_SERVER_1 value: zoo1 --- apiVersion: v1 kind: Service metadata: name: zoo1 labels: app: zookeeper-1 spec: ports: - name: client port: 2181 protocol: TCP - name: follower port: 2888 protocol: TCP - name: leader port: 3888 protocol: TCP selector: app: zookeeper-1
This creates a Kubernetes Deployment that will schedule zookeeper pods and a Kubernetes Service to route traffic to the pods. The service has a short name of zoo1 which we will use later when we deploy the Kafka Brokers.
Create the resource:
$ kubectl create -f zookeeper.yml
Now lets start deploying Kafka.
The first thing we need to do is deploy a Kubernetes Service that will manage our Kafka Broker deployments.
Create a new file called kafka-service.yml and add the following contents:
--- apiVersion: v1 kind: Service metadata: name: kafka-service labels: name: kafka spec: ports: - port: 9092 name: kafka-port protocol: TCP selector: app: kafka id: "0" type: LoadBalancer
You might notice that we have set the type to LoadBalancer
. If your Kubernetes Cluster is deployed to bare-metal don't freak out. There is a new Kubernetes add-on called MetalLB that allows this. Checkout my article Kubernetes metallb bare metal loadbalancer for instructions on how to enable it. It will make your life much easier.
Create the service.
$ kubectl create -f kafka-service
Now we need to get the external IP for the service, because we will need it in order to spin up a Kafka Broker in the next section.
$ kubectl describe svc kafka-service Name: kafka-service Namespace: default Labels: name=kafka Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"name":"kafka"},"name":"kafka-service","namespace":"default"},"spec":{"ports... Selector: app=kafka,id=0 Type: LoadBalancer IP: 10.105.148.62 LoadBalancer Ingress: 192.168.1.240 Port: kafka-port 9092/TCP TargetPort: 9092/TCP NodePort: kafka-port 30718/TCP Endpoints: 10.44.0.4:9092 Session Affinity: None External Traffic Policy: Cluster Events: <none>
In the example above, I would note that the LoadBalancer Ingress is set to 192.168.1.240. Now we can start our Kafka Broker.
We have the Kubernetes Service deployed but all it does is load balance our Kafka pods which are not deployed yet.
Follow these steps to deploy them.
Create a new file called kafka-broker.yml and add the following contents:
--- kind: Deployment apiVersion: extensions/v1beta1 metadata: name: kafka-broker0 spec: template: metadata: labels: app: kafka id: "0" spec: containers: - name: kafka image: wurstmeister/kafka ports: - containerPort: 9092 env: - name: KAFKA_ADVERTISED_PORT value: "30718" - name: KAFKA_ADVERTISED_HOST_NAME value: 192.168.1.240 - name: KAFKA_ZOOKEEPER_CONNECT value: zoo1:2181 - name: KAFKA_BROKER_ID value: "0" - name: KAFKA_CREATE_TOPICS value: admintome-test:1:1
Notice that the KAFKA_ADVERTISED_HOST_NAME is set to the IP address we noted earlier. Also note we tell the Kafka Broker to automatically create a topic admintome-test with 1 partition and 1 replica. You can create multiple topics using the same vernacular and separating them by commas (i.e. - topic1:1:1,topic2:1:1
).
Save the file and create the resource.
$ create -f kafka-broker.yml
You can validate that everything is running.
$ kubectl get pod kafka-broker0
To scale your Kafka Brokers, create another file but give it a different name (i.e. kafka-broker1) and update the ID to match.
Lets test our Kafka Deployment
We are going to test our Kafka deployment by using an application called KafkaCat.
To install:
$ apt-get install kafkacat
After the application is installed we will run it in consumer mode (which is the default).
kafkacat -b 192.168.1.240:9092 -t admintome-test
This should not show anything yet because we haven't sent anything to our topic yet...
To send stuff we can copy any text file into our current directory and send it to our Kafka Topic. In another window, run the following command.
$ cat README | kafkacat -b 192.168.1.240 -t admintome-test
You should see the output in the first window which has KafkaCat still running in consumer mode.
Congratulations! You have successfully deployed a Kafka Cluster to Kubernetes.
Thanks for reading. If you liked this post, share it with all of your programming buddies!
Further reading
☞ The Complete Node.js Developer Course (3rd Edition)
☞ Angular & NodeJS - The MEAN Stack Guide
☞ NodeJS - The Complete Guide (incl. MVC, REST APIs, GraphQL)
☞ MongoDB - The Complete Developer’s Guide
☞ The Complete Developers Guide to MongoDB
☞ Creating RESTful APIs with NodeJS and MongoDB Tutorial
☞ MEAN Stack Tutorial MongoDB, ExpressJS, AngularJS and NodeJS
☞ How To Build a Node.js Application with Docker
☞ Authenticate a Node ES6 API with JSON Web Tokens
☞ Creating a RESTful Web API with Node.js and Express.js from scratch
Originally published on https://www.admintome.com
#big-data #docker #web-development