Exploring Kafka listeners

Exploring Kafka listeners

Some time ago I started to explore the world of Kafka in my new project. While my wife still thinks I joined a literature club, I realize this distributed baby is no joke.

Let’s see how we can pimp our Kafka cluster to help us address some of the day-to-day problems we face when introducing our corporate (or startup) Apps ecosystems to the world of distributed messaging madness!

Motivation…

Some time ago I started to explore the world of Kafka in my new project. While my wife still thinks I joined a literature club, I realize this distributed baby is no joke.

I would like to share some experience, tips, and tricks of a DevSecOps guy with a genuine interest in exploring what is trendy and cool nowadays.

To set a context, I will be talking about running Kafka in AKS (Azure Kubernetes services). In our project, we did not go for a Kafka operator and we decided to use (and customize) the confluent helm charts. As far as I know, those are not officially supported, so one might need to tweak and customise them here and there.

We also decided not to use any available service mesh (such as linkerd or istio) for our setup. The reason was mainly a fear of the significantly increased complexity of managing a service mesh on top of Kafka. For our case it looked a bit like overkill — only time will tell if we were right.

Few examples (and fuel for future posts):

  • Utilization of multiple Kafka listeners (discussed in this article)
  • DNS based service discovery setup (for brokers and zookeeper)
  • Integration with configuration and secret management system (Azure AppConfig and Keyvaults)
  • Implementing security in transfer — TLS interfaces
  • Introduction of more zookeepers to increase resiliency

kafka azure-kubernetes-service devsecops architecture security

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

50+ Useful Kubernetes Tools for 2020 - Part 2

Our original Kubernetes tool list was so popular that we've curated another great list of tools to help you improve your functionality with the platform.

Private Azure Kubernetes Service Clusters with Azure Private Links?

.. and How to securely access private AKS clusters over a bastion? Terraform code included!

Manage Azure Event Hubs with Azure Service Operator on Kubernetes

Azure Service Operator is an open source project to help you provision and manage Azure services using Kubernetes. Developers can use it to provision Azure services from any environment, be it Azure, any other cloud provider or on-premises — Kubernetes is the only common denominator!

Securing Azure Functions using an Azure Virtual Network

In this post, an Azure Function is deployed in a Azure Virtual Network and the access to the Azure Function is restricted so that it cannot be reach from the Internet. Only Applications deployed in…

Getting Started With Azure Event Grid Viewer

In the article, we will go to the next step to create a subscription and use webhook event handlers to view those logs in our Azure web application.