Zakary  Goyette

Zakary Goyette

1596765780

How to Consume Kafka Efficiently in Golang?

If you often need to process data from Message Queue Systems like Kafka, you may often wonder how to consume data efficiently. Specifically, how could you process 1 billion or more messages within an hour? Then this story is just for you.

In this story, I proposed four consuming patterns using the well-known golang package Sarama and performed several experiments to show how different patterns affect the final performance. Let’s go.

Setup

To do this test, I established a simple cluster with three nodes, one for Kafka broker, one for the producer server and the final as the consumer server, as shown in Fig 1:

Image for post

Fig 1 Cluster

To ensure fairness, all of the three nodes have the same configurations(4 Cpu cores, 8GB memory and 500GB SSD disk). We defined a struct _Message _and let producer/consumer simply do json marshal/unmarshal things.

type Message struct {
   Id int `json:"id"`
}

The producer code is fairly simple:

func (p *Producer) StartProduce(done chan struct{}, topic string) {
   start := time.Now()
   for i := 0; ; i++ {
      msg := Message{i}
      msgBytes, err := json.Marshal(msg)
      if err != nil {
         continue
      }
      select {
      case <-done:
         return
      case p.p.Input() <- &sarama.ProducerMessage{
         Topic: topic,
         Value: sarama.ByteEncoder(msgBytes),
      }:
         if i % 5000 == 0 {
            fmt.Printf("produced %d messages with speed %.2f/s\n", i, float64(i) / time.Since(start).Seconds())
         }
      case err := <-p.p.Errors():
         fmt.Printf("Failed to send message to kafka, err: %s, msg: %s\n", err, msgBytes)
      }
   }
}

With this configuration, we achieved a producing speed of 570k/s:

produced 63445000 messages with speed 570853.28/s

Could our patterns handle data with such high load? Let’s wait and see.

#golang #sarama #kafka

What is GEEK

Buddha Community

How to Consume Kafka Efficiently in Golang?
Zakary  Goyette

Zakary Goyette

1596765780

How to Consume Kafka Efficiently in Golang?

If you often need to process data from Message Queue Systems like Kafka, you may often wonder how to consume data efficiently. Specifically, how could you process 1 billion or more messages within an hour? Then this story is just for you.

In this story, I proposed four consuming patterns using the well-known golang package Sarama and performed several experiments to show how different patterns affect the final performance. Let’s go.

Setup

To do this test, I established a simple cluster with three nodes, one for Kafka broker, one for the producer server and the final as the consumer server, as shown in Fig 1:

Image for post

Fig 1 Cluster

To ensure fairness, all of the three nodes have the same configurations(4 Cpu cores, 8GB memory and 500GB SSD disk). We defined a struct _Message _and let producer/consumer simply do json marshal/unmarshal things.

type Message struct {
   Id int `json:"id"`
}

The producer code is fairly simple:

func (p *Producer) StartProduce(done chan struct{}, topic string) {
   start := time.Now()
   for i := 0; ; i++ {
      msg := Message{i}
      msgBytes, err := json.Marshal(msg)
      if err != nil {
         continue
      }
      select {
      case <-done:
         return
      case p.p.Input() <- &sarama.ProducerMessage{
         Topic: topic,
         Value: sarama.ByteEncoder(msgBytes),
      }:
         if i % 5000 == 0 {
            fmt.Printf("produced %d messages with speed %.2f/s\n", i, float64(i) / time.Since(start).Seconds())
         }
      case err := <-p.p.Errors():
         fmt.Printf("Failed to send message to kafka, err: %s, msg: %s\n", err, msgBytes)
      }
   }
}

With this configuration, we achieved a producing speed of 570k/s:

produced 63445000 messages with speed 570853.28/s

Could our patterns handle data with such high load? Let’s wait and see.

#golang #sarama #kafka

Hire Dedicated Golang Developers | Golang Web Development Company

Does your business need a robust system across large-scale network servers then developing your app with a Golang programming language is the way to go. Golang is generally used for the development of highly secured, High Speed and High Modularity apps such as a FinTech Industry.

Want to develop a Highly secured app for your business?

Then hire a dedicated Golang developer from WebClues Infotech that are highly skilled in carrying out the work in a timely and qualitative output. With WebClues Infotech you get the assurance that we know what are the customers’ expectations and how to deliver on them on time.

Get your desired Golang Developer based on your project requirement!!

Share your requirements here https://www.webcluesinfotech.com/contact-us/

Book Free Interview with Golang developer: https://bit.ly/3dDShFg

#hire golang developer #hire go language developer #dedicated golang app developers #golang web development company #hire golang developers india #hire expert golang developers

akshay L

akshay L

1572344038

Kafka Spark Streaming | Kafka Tutorial

In this kafka spark streaming tutorial you will learn what is apache kafka, architecture of apache kafka & how to setup a kafka cluster, what is spark & it’s features, components of spark and hands on demo on integrating spark streaming with apache kafka and integrating spark flume with apache kafka.

# Kafka Spark Streaming #Kafka Tutorial #Kafka Training #Kafka Course #Intellipaat

Golang Web Development:Th Best Programming Language in 2020

https://www.mobinius.com/blogs/golang-web-development-company

#golang web development #golang-app-development-company #golang-development-solutions #hire-golang-developers #golang-development-services

Ajay Kapoor

1624960485

Golang vs. Node.JS: Who Trumps the Battle of Backend Frameworks?

Full blog here

The backend of your application is truly the essential part of your product. No matter how much you appreciate the design, the application’s success lies in its backend. A scalable backend that effectively implements the required business logic is the primary goal of programmers.

Therefore, it is crucial to choose the most powerful and scalable technology. There are plenty of languages ​​in the market that can form the backend of any application, Node.js and Golang are the two most popular technologies among them.

They are real and developed languages ​​that have recently been used in various outstanding projects. Golang is an open-source programming language, whereas Node.js is an open-source server framework. They both are gaining popularity for various reasons.

According to a development stat, it is observed that almost 50% out of 58,543 respondents use Node.js as their preferred app development tool.

Golang, on the other hand, has overtaken other programming languages in the application development market and has gained huge recognition over the past few years.

But, which backend framework is best for you? In this article, I’ll make a healthy comparison of two of Google’s most popular backend development tools based on several essential features and various other factors.

Golang developers for hire

#best backend frameworks #node or golang #golang or nodejs #nodejs vs golang #golang vs nodejs #top backend frameworks