Kubernetes (K8s) is one of the most famous open-source projects and it is being continuously adapted. Kafka is an open-source stream-processing software platform that is used by a lot of companies. For example, LinkedIn customizes Apache Kafka for 7 trillion messages per day

So, what is Kafka-connect now? Kafka Connect is an open-source component of Kafka, a framework for connecting Kafka with external systems such as databases, key-value stores, search indexes, and file systems. It makes it simple to quickly define connectors that move large data sets into and out of Kafka.

As Kafka-connect is a component of Kafka, the setup will need Kafka broker(s) and Zookeepers (at least, until the Zookeeper dependency is removed). Our setup would look like something below:

We will start by setting up a Kafka cluster first. In this blog, we are going to use Confluent’s open-source Helm chart to do that. I am one of the contributors to these Helm charts (bragging alert ;).

For getting started, make sure a Kubernetes cluster is running (e.g. GKE by Google, EKS by AWS, AKS by Azure, Minikube, etc.) and the following tools are installed in your local system:

  1. helm (version used for this blog: v2.16.1)
  2. Kubectl (version used for this blog: v1.15.3)
  3. docker (version used for this blog: 19.03.1)
  4. MySql (Version used for this blog: 5.7)

Let’s start by cloning the repository and updating the dependencies.

#helm #kafka #mysql #kubernetes

How to Deploy Kafka-connect on Kubernetes using Helm
38.65 GEEK