Diving Deep into Kafka

Diving Deep into Kafka

The objective of this blog is to build some more understanding of Apache Kafka concepts such as Topics, Partitions, Consumer, and Consumer Groups. Kafka's basic concepts have been covered in my previous article. Kafka Topic & Partitions As we know, messages in Kafka are categorized or stored inside Topics. In simple terms, Topic can be construed as a Database table.

The objective of this blog is to build some more understanding of Apache Kafka concepts such as Topics, Partitions, Consumer, and Consumer Groups. Kafka's basic concepts have been covered in my previous article.

Kafka Topic & Partitions

As we know, messages in Kafka are categorized or stored inside Topics. In simple terms, Topic can be construed as a Database table. Kafka Topics inside is broken down into partitions. Partitions allow us to parallelize a topic by splitting the data of a topic across multiple brokers, thus adding an essence of parallelism to the ecosystem.

Behind the scenes

Messages are written to a partition in an append-only manner, and messages are read from a partition from beginning to end, FIFO mannerism. Each message within a partition is identified by an integer value called offset. _An _offset is an immutable sequential ordering of messages, maintained by Kafka. Anatomy of a Topic with multiple partitions:

Image for post

Partitioned Topic

Sequential number in array fashion is the offset value maintained by Kafka

Some key points:

  1. Ordering of messages is maintained at the partition level, not across the topic.
  2. Data written to partition is immutable and can’t be updated.
  3. Each message in the Kafka broker is a collection of message topics, partition, offset, key, and value.
  4. Each partition will have a leader that will take care of Read/Write operations in the partition.

kafka-python kafka streaming apache-kafka

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

Wondering how to upgrade your skills in the pandemic? Here's a simple way you can do it.

Corona Virus Pandemic has brought the world to a standstill. Countries are on a major lockdown. Schools, colleges, theatres, gym, clubs, and all other public

Kafka Spark Streaming | Kafka Tutorial

🔥Intellipaat Kafka training: https://intellipaat.com/kafka-training-online/ 👉In this kafka spark streaming tutorial you will learn what is apache kafka, arch...

Apache Kafka: Docker Container and examples in Python

How to install Kafka using Docker and produce/consume messages in Python. Apache Kafka is a stream-processing software platform originally developed by LinkedIn, open sourced in early 2011 and currently developed by the Apache Software Foundation. It is written in Scala and Java.

Apache Kafka Tutorial - Kafka Tutorial for Beginners

This Apache Kafka Tutorial - Kafka Tutorial for Beginners will help you understand what is Apache Kafka & its features. It covers different components of Apache Kafka & it’s architecture. You'll learn: What is Kafka? Kafka Features, Kafka Components, Kafka architecture, Installing Kafka, Working with Single Node Single Broker Cluster

Python Tricks Every Developer Should Know

In this tutorial, you’re going to learn a variety of Python tricks that you can use to write your Python code in a more readable and efficient way like a pro.