Introduction

An event-driven architecture utilizes events to trigger and communicate between microservices. An event is a change in the service’s state, such as an item being added to the shopping cart. When an event occurs, the service produces an event notification which is a packet of information about the event.

The architecture consists of an event producer, an event router, and an event consumer. The producer sends events to the router, and the consumer receives the events from the router. Depending on the capability, the router can push the events to the consumer or send the events to the consumer on request (poll). The producer and the consumer services are decoupled, which allows them to scale, deploy, and update independently.

Apache Kafka is one of the most popular open-source event streaming platforms. It is horizontally scalable, distributed, and fault-tolerant by design. Kafka’s programming model is based on the publish-subscribe pattern. With Kafka, publishers send messages to topics, which are named logical channels. A subscriber to a topic receives all the messages published to the topic. In an event-driven architecture, Kafka is used as an event router, and the microservices publish and subscribe to the events.

In this article, we will learn how to prepare the local environment for development and publish messages to Kafka. My subsequent articles will focus on building the components of an end-to-end application that will help you build event-driven microservices.

Source Code

The complete source code of the application and other artifacts is available in my GitHub repository.

Kafka Components

I will briefly discuss the components of Kafka that are relevant to us for using Kafka as a message broker. Apart from the publish-subscribe model, Kafka also supports a Streams API that is useful for transforming data from one topic to another, and a Connect API that helps you implement connectors that pull data from external systems into Kafka or push data from Kafka to external systems. These APIs are outside the scope of this article. To understand Kafka’s architecture in detail, please read the Introduction to Kafka article on the Confluent docs website.

We understand that Kafka acts as a middleman that enables exchanging information from producers to consumers. Kafka can be set up across multiple servers, which are called Kafka brokers. With multiple brokers, you get the benefit of data replication, fault tolerance, and high availability of your Kafka cluster.

Following is a high-level system design of a Kafka cluster:

The metadata of Kafka cluster processes is stored in an independent system called Apache Zookeeper. Zookeeper helps Kafka perform several critical functions, such as electing a leader in case of node failure. It also maintains the list of consumers in a consumer group and manages the access control list of Kafka topics.

The first level segregation of events/messages in Kafka occurs through a Kafka object called the topic. The event producer publishes events to a topic which Kafka can subsequently broadcast to interested consumers. Think of a topic as a collection of FIFO (First In First Out) queues. You can either randomly store a message in one of the queues or place related messages on a single queue to guarantee FIFO. Each of the queues within a topic is called a topic partition. Each message in a queue is placed at a unique position number called an offset.

You can combine multiple consumers in a consumer group to scale out the consumption of messages from a topic. A consumer group is identified through a unique group id. Kafka balances the allocation of partitions between individual consumers of a consumer group to avoid the duplicate processing of messages.

After a consumer consumes a message stored at an offset, it commits the message to inform Kafka that it is done processing it. On the subsequent request, the consumer will receive the message at the next offset and so on.

Local Environment Setup

Setting up a development environment to work with Kafka is reasonably easy with Docker Compose. You can share Docker Compose specifications with other developers in your team to ensure environment consistency. We will use Docker Compose to set up a Kafka cluster that consists of the following components:

1. Apache Zookeeper: The Zookeeper dependency will be removed from Kafka in the future by some vendors such as Confluent. Read the latest documentation from the vendor that you intend to use for Kafka.
2. Kafka
3. Kafdrop: Kafdrop is a popular web-based user interface for viewing Kafka topics and browsing consumer groups. It makes your Kafka cluster observable, which helps you diagnose issues and helps you with development.
4. Schema Registry: Schema Registry is a service that lives outside of your cluster and allows the developers to manage the message schemas. Kafka supports messages in Avro, JSON, and Protobuf formats, and the Schema Registry supports the storage and retrieval of versioned schemas in all those formats. You can read more about Schema Registry on the Confluent docs website.

Several vendors publish Zookeeper and Kafka Docker images with slight differences in behavior and configuration. I typically use the distributions from Bitnami. However, you can also use the distributions from Confluence, Spotify, and Wurstmeister. Bitnami and Confluence build and test the images nightly, and they are also compatible with each other, so I recommend using them.

Create a file named docker-compose.yml and populate the file with the contents of the following listing:

YAML

1

version: "2"

2

3

networks:

4

  kafka-net:

5

    driver: bridge

6

7

services:

8

  zookeeper-server:

9

    image: bitnami/zookeeper:latest

10

    networks:

11

      - kafka-net

12

    ports:

13

      - 2181:2181

14

    environment:

15

      - ALLOW_ANONYMOUS_LOGIN=yes

16

  kafdrop:

17

    image: obsidiandynamics/kafdrop

18

    networks:

19

      - kafka-net

20

    restart: "no"

21

    ports:

22

      - 9000:9000

23

    environment:

24

      KAFKA_BROKERCONNECT: PLAINTEXT://kafka-server:29092

25

      JVM_OPTS: -Xms16M -Xmx48M -Xss180K -XX:-TieredCompilation -XX:+UseStringDeduplication -noverify

26

      SCHEMAREGISTRY_CONNECT: http://schema-registry:8081

27

    depends_on:

28

      - kafka-server

29

  kafka-server:

30

    image: bitnami/kafka:latest

31

    networks:

32

      - kafka-net

33

    ports:

34

      - 9092:9092

35

    environment:

36

      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper-server:2181

37

      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka-server:29092,PLAINTEXT_HOST://127.0.0.1:9092

38

      - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT

39

      - KAFKA_CFG_LISTENERS=PLAINTEXT://:29092,PLAINTEXT_HOST://:9092

40

      - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=PLAINTEXT

41

      - ALLOW_PLAINTEXT_LISTENER=yes

42

    depends_on:

43

      - zookeeper-server

44

  schema-registry:

45

    image: confluentinc/cp-schema-registry:latest

46

    networks:

47

      - kafka-net

48

    ports:

49

      - 8081:8081

50

    environment:

51

      - SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS=PLAINTEXT://kafka-server:29092

52

      - SCHEMA_REGISTRY_HOST_NAME=localhost

53

      - SCHEMA_REGISTRY_LISTENERS=http://0.0.0.0:8081

54

    depends_on:

55

      - kafka-server

#cloud #tutorial #devops #aws #azure #web #.net core