Spring Boot and Kafka Configuration Tuning

Spring Boot and Kafka Configuration Tuning

Spring Boot and Kafka Configuration Tuning. See a setup for configuration tuning in an isolated environment and to determine the Spring Boot, Kafka configuration and best practices for moderate uses.

Summary

The goal of this exercise is to provide a setup for configuration tuning in an isolated environment and to determine the Spring Boot, Kafka configuration, and best practices for moderate uses. The high-level observations are:

  1. Partition your Kafka topic and design system stateless for higher concurrency. For most of the moderate use cases (we have 100,000 messages per hour) you won't need more than 10 partitions. As a rule of thumb; concurrency direction is directly proportional to the times the consumer spends processing a single message. 
  2. The default Spring boot configuration are very reasonable for moderate uses. Avoid changing them without a proper root cause analysis. 
  3. Although it differs from use case to use case, it is recommended to have the producer receive acknowledgment from at least one Kafka Partition leader and manual acknowledgment at the consumer side. 
  4. Size Kafka topics appropriately, by using small messages that are less than 0.5 MB, to avoid Kafka broker timeout errors. If required, chunk the large data into small messages, send them through a single partition, and reconstruct the message at the consumer side. 

Challenge

I recently got involved in an initiative where we had a need to hydrate two or more clusters of an In Memory Data Grid (IMDG) with transient data from a combination of data sources. [Please don’t ask me why? :)]

Given the fact that All IMDG clusters must be always in an identical state, the option of sourcing the data directly from each IMDG was out of the question for obvious reasons. I needed a common memory space that is accessible or can flow the data to all IMDGs. In our infrastructure, Kafka was the only available system that was accessible from all IMDGs. Hence, it was decided to fill a Kafka topic once from all data sources, and then use it as a single data source to hydrate all IMDGs.

java tutorial big data spring boot kafka perforamnce

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

Silly mistakes that can cost ‘Big’ in Big Data Analytics

‘Data is the new science. Big Data holds the key answers’ - Pat Gelsinger The biggest advantage that the enhancement of modern technology has brought

Big Data can be The ‘Big’ boon for The Modern Age Businesses

We need no rocket science in understanding that every business, irrespective of their size in the modern-day business world, needs data insights for its expansion. Big data analytics is essential when it comes to understanding the needs and wants of a significant section of the audience.

Role of Big Data in Healthcare - DZone Big Data

In this article, see the role of big data in healthcare and look at the new healthcare dynamics. Big Data is creating a revolution in healthcare, providing better outcomes while eliminating fraud and abuse, which contributes to a large percentage of healthcare costs.

How you’re losing money by not opting for Big Data Services?

Big Data Analytics is the next big thing in business, and it is a reality that is slowly dawning amongst companies. With this article, we have tried to show you the importance of Big Data in business and urge you to take advantage of this immense...

Data Lakes Are Not Just For Big Data - DZone Big Data

A data expert discusses the three different types of data lakes and how data lakes can be used with data sets not considered 'big data.'