Alex Tyler

Alex Tyler

1562830029

How to Installing Kafka Docker on Kubernetes

In this ultimate guide I will give you a simple step-by-step tutorial on installing Kafka Docker on Kubernetes. This post includes a complete video walk-through.

There has been a lot of interest lately about deploying Kafka to a Kubernetes cluster. If you are wanting to take the deep dive yourself then you found the right article. Now that we have Kafka Docker, deploying a Kafka cluster to Kubernetes is a snap.

Deploy ZooKeeper to Kubernetes

Kafka relies on ZooKeeper to keep track of its configuration including what topics are available.

Before we deploy Kafka we need to deploy ZooKeeper.

Create a file called zookeeper.yml and add these contents:

---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: zookeeper-deployment-1
spec:
  template:
    metadata:
      labels:
        app: zookeeper-1
    spec:
      containers:
      - name: zoo1
        image: digitalwonderland/zookeeper
        ports:
        - containerPort: 2181
        env:
        - name: ZOOKEEPER_ID
          value: "1"
        - name: ZOOKEEPER_SERVER_1
          value: zoo1
---
apiVersion: v1
kind: Service
metadata:
  name: zoo1
  labels:
    app: zookeeper-1
spec:
  ports:
  - name: client
    port: 2181
    protocol: TCP
  - name: follower
    port: 2888
    protocol: TCP
  - name: leader
    port: 3888
    protocol: TCP
  selector:
    app: zookeeper-1

This creates a Kubernetes Deployment that will schedule zookeeper pods and a Kubernetes Service to route traffic to the pods. The service has a short name of zoo1 which we will use later when we deploy the Kafka Brokers.

Create the resource:

$ kubectl create -f zookeeper.yml

Now lets start deploying Kafka.

Deploying a Kafka Docker Service

The first thing we need to do is deploy a Kubernetes Service that will manage our Kafka Broker deployments.

Create a new file called kafka-service.yml and add the following contents:

---
apiVersion: v1
kind: Service
metadata:
  name: kafka-service
  labels:
    name: kafka
spec:
  ports:
  - port: 9092
    name: kafka-port
    protocol: TCP
  selector:
    app: kafka
    id: "0"
  type: LoadBalancer

You might notice that we have set the type to LoadBalancer. If your Kubernetes Cluster is deployed to bare-metal don't freak out. There is a new Kubernetes add-on called MetalLB that allows this. Checkout my article Kubernetes metallb bare metal loadbalancer for instructions on how to enable it. It will make your life much easier.

Create the service.

$ kubectl create -f kafka-service

Now we need to get the external IP for the service, because we will need it in order to spin up a Kafka Broker in the next section.

$ kubectl describe svc kafka-service
Name: kafka-service
Namespace: default
Labels: name=kafka
Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"name":"kafka"},"name":"kafka-service","namespace":"default"},"spec":{"ports...
Selector: app=kafka,id=0
Type: LoadBalancer
IP: 10.105.148.62
LoadBalancer Ingress: 192.168.1.240
Port: kafka-port 9092/TCP
TargetPort: 9092/TCP
NodePort: kafka-port 30718/TCP
Endpoints: 10.44.0.4:9092
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>

In the example above, I would note that the LoadBalancer Ingress is set to 192.168.1.240. Now we can start our Kafka Broker.

Deploying the Kafka Broker to Kubernetes

We have the Kubernetes Service deployed but all it does is load balance our Kafka pods which are not deployed yet.

Follow these steps to deploy them.

Create a new file called kafka-broker.yml and add the following contents:

---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: kafka-broker0
spec:
  template:
    metadata:
      labels:
        app: kafka
        id: "0"
    spec:
      containers:
      - name: kafka
        image: wurstmeister/kafka
        ports:
        - containerPort: 9092
        env:
        - name: KAFKA_ADVERTISED_PORT
          value: "30718"
        - name: KAFKA_ADVERTISED_HOST_NAME
          value: 192.168.1.240
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: zoo1:2181
        - name: KAFKA_BROKER_ID
          value: "0"
        - name: KAFKA_CREATE_TOPICS
          value: admintome-test:1:1

Notice that the KAFKA_ADVERTISED_HOST_NAME is set to the IP address we noted earlier. Also note we tell the Kafka Broker to automatically create a topic admintome-test with 1 partition and 1 replica. You can create multiple topics using the same vernacular and separating them by commas (i.e. - topic1:1:1,topic2:1:1).

Save the file and create the resource.

$ create -f kafka-broker.yml

You can validate that everything is running.

$ kubectl get pod kafka-broker0

To scale your Kafka Brokers, create another file but give it a different name (i.e. kafka-broker1) and update the ID to match.

Lets test our Kafka Deployment

Testing With KafkaCat

We are going to test our Kafka deployment by using an application called KafkaCat.

To install:

$ apt-get install kafkacat

After the application is installed we will run it in consumer mode (which is the default).

kafkacat -b 192.168.1.240:9092 -t admintome-test

This should not show anything yet because we haven't sent anything to our topic yet...

To send stuff we can copy any text file into our current directory and send it to our Kafka Topic. In another window, run the following command.

$ cat README | kafkacat -b 192.168.1.240 -t admintome-test

You should see the output in the first window which has KafkaCat still running in consumer mode.

Congratulations! You have successfully deployed a Kafka Cluster to Kubernetes.

Thanks for reading. If you liked this post, share it with all of your programming buddies!

Further reading

☞ The Complete Node.js Developer Course (3rd Edition)

☞ Angular & NodeJS - The MEAN Stack Guide

☞ NodeJS - The Complete Guide (incl. MVC, REST APIs, GraphQL)

☞ MongoDB - The Complete Developer’s Guide

☞ The Complete Developers Guide to MongoDB

☞ Creating RESTful APIs with NodeJS and MongoDB Tutorial

☞ MEAN Stack Tutorial MongoDB, ExpressJS, AngularJS and NodeJS

☞ How To Build a Node.js Application with Docker

☞ Authenticate a Node ES6 API with JSON Web Tokens

☞ Creating a RESTful Web API with Node.js and Express.js from scratch


Originally published on https://www.admintome.com

#big-data #docker #web-development

What is GEEK

Buddha Community

How to Installing Kafka Docker on Kubernetes
Christa  Stehr

Christa Stehr

1602964260

50+ Useful Kubernetes Tools for 2020 - Part 2

Introduction

Last year, we provided a list of Kubernetes tools that proved so popular we have decided to curate another list of some useful additions for working with the platform—among which are many tools that we personally use here at Caylent. Check out the original tools list here in case you missed it.

According to a recent survey done by Stackrox, the dominance Kubernetes enjoys in the market continues to be reinforced, with 86% of respondents using it for container orchestration.

(State of Kubernetes and Container Security, 2020)

And as you can see below, more and more companies are jumping into containerization for their apps. If you’re among them, here are some tools to aid you going forward as Kubernetes continues its rapid growth.

(State of Kubernetes and Container Security, 2020)

#blog #tools #amazon elastic kubernetes service #application security #aws kms #botkube #caylent #cli #container monitoring #container orchestration tools #container security #containers #continuous delivery #continuous deployment #continuous integration #contour #developers #development #developments #draft #eksctl #firewall #gcp #github #harbor #helm #helm charts #helm-2to3 #helm-aws-secret-plugin #helm-docs #helm-operator-get-started #helm-secrets #iam #json #k-rail #k3s #k3sup #k8s #keel.sh #keycloak #kiali #kiam #klum #knative #krew #ksniff #kube #kube-prod-runtime #kube-ps1 #kube-scan #kube-state-metrics #kube2iam #kubeapps #kubebuilder #kubeconfig #kubectl #kubectl-aws-secrets #kubefwd #kubernetes #kubernetes command line tool #kubernetes configuration #kubernetes deployment #kubernetes in development #kubernetes in production #kubernetes ingress #kubernetes interfaces #kubernetes monitoring #kubernetes networking #kubernetes observability #kubernetes plugins #kubernetes secrets #kubernetes security #kubernetes security best practices #kubernetes security vendors #kubernetes service discovery #kubernetic #kubesec #kubeterminal #kubeval #kudo #kuma #microsoft azure key vault #mozilla sops #octant #octarine #open source #palo alto kubernetes security #permission-manager #pgp #rafay #rakess #rancher #rook #secrets operations #serverless function #service mesh #shell-operator #snyk #snyk container #sonobuoy #strongdm #tcpdump #tenkai #testing #tigera #tilt #vert.x #wireshark #yaml

Maud  Rosenbaum

Maud Rosenbaum

1601051854

Kubernetes in the Cloud: Strategies for Effective Multi Cloud Implementations

Kubernetes is a highly popular container orchestration platform. Multi cloud is a strategy that leverages cloud resources from multiple vendors. Multi cloud strategies have become popular because they help prevent vendor lock-in and enable you to leverage a wide variety of cloud resources. However, multi cloud ecosystems are notoriously difficult to configure and maintain.

This article explains how you can leverage Kubernetes to reduce multi cloud complexities and improve stability, scalability, and velocity.

Kubernetes: Your Multi Cloud Strategy

Maintaining standardized application deployments becomes more challenging as your number of applications and the technologies they are based on increase. As environments, operating systems, and dependencies differ, management and operations require more effort and extensive documentation.

In the past, teams tried to get around these difficulties by creating isolated projects in the data center. Each project, including its configurations and requirements were managed independently. This required accurately predicting performance and the number of users before deployment and taking down applications to update operating systems or applications. There were many chances for error.

Kubernetes can provide an alternative to the old method, enabling teams to deploy applications independent of the environment in containers. This eliminates the need to create resource partitions and enables teams to operate infrastructure as a unified whole.

In particular, Kubernetes makes it easier to deploy a multi cloud strategy since it enables you to abstract away service differences. With Kubernetes deployments you can work from a consistent platform and optimize services and applications according to your business needs.

The Compelling Attributes of Multi Cloud Kubernetes

Multi cloud Kubernetes can provide multiple benefits beyond a single cloud deployment. Below are some of the most notable advantages.

Stability

In addition to the built-in scalability, fault tolerance, and auto-healing features of Kubernetes, multi cloud deployments can provide service redundancy. For example, you can mirror applications or split microservices across vendors. This reduces the risk of a vendor-related outage and enables you to create failovers.

#kubernetes #multicloud-strategy #kubernetes-cluster #kubernetes-top-story #kubernetes-cluster-install #kubernetes-explained #kubernetes-infrastructure #cloud

Iliana  Welch

Iliana Welch

1595249460

Docker Explained: Docker Architecture | Docker Registries

Following the second video about Docker basics, in this video, I explain Docker architecture and explain the different building blocks of the docker engine; docker client, API, Docker Daemon. I also explain what a docker registry is and I finish the video with a demo explaining and illustrating how to use Docker hub

In this video lesson you will learn:

  • What is Docker Host
  • What is Docker Engine
  • Learn about Docker Architecture
  • Learn about Docker client and Docker Daemon
  • Docker Hub and Registries
  • Simple demo to understand using images from registries

#docker #docker hub #docker host #docker engine #docker architecture #api

Kafka Manger installation using Docker Compose | Kafka | Docker | Kafka Manager

We will install Kafka Manager using docker compose. In this video we will learn to install three components using docker compose
Kafka
Zookeeper
Kafka Manager

Post Link : http://selftuts.com/install-kafka-and-kafka-manger-using-docker-compose/

#kafka #kafka manger #docker compose

Kubernetes Vs. Docker: Primary Differences You Should Know

Kubernetes vs Docker is an essential topic of debate among professionals. Both of them are related to containerization, and both of them have their sets of features. So, the community is divided into two sections, which can lead to confusion.

That’s why you should read this article as we’ve discussed all the significant differences between these two solutions. Let’s get started.

#kubernetes vs docker #kubernetes #docker