One of the must challenging aspects of micro-services is the communication. But there is an easy way to communicate with greatest benefits. One of the most remarkable benefits about using Apache Kafka to communicate micro-services are:
- Zero packets loss.
- Very high throughput (A single microservice can process 10 MB/sec using only one CPU).
- Real time processing.
- CPU bound scalability (it means that if you have two instances of one microservice running in one CPU each one, your application will scale to process 20 MB/sec, if you have 3 you will scale to process 30 MB/sec, end so on).
For this benchmark I got a throughput of 10 MB/sec, but processing rate will depend mainly on CPU capacity, business logic complexity, data, etc.
The complete code and projects I used to generate benchmarks is located in this GitHub repo: kafka-reactive-app.
Case study
Our case study represents a data pipeline, which receives stock movements, and sends these movements to a reactive Angular application.
We will focus on how data is flowing along our data pipeline.
Producer: This is a mock micro-service to generates CSV random values to simulate a source. All records produced by this component are stored in a topic.
Listener: Takes every record and performs a data type validation. Invalid records are stored in a different topic.
Landing: Converts CSV into JSON.
Processor: Every record is branched to store stock rates in one topic and stock information in a different topic.
Messenger: This is a web socket to send data to an Angular application.
Setup your environment:
To setup your environment just run “docker-compose up” using this docker-compose.yml file.
This will start a Confluent environment with a default cluster. Once your environment is started up, go to http://localhost:9021/ and create following topics:
- input-topic
- landing-area
- landing-error
- staging-area
- stock-value
- stock-ratio
- stock-ratio-out
- stock-value-out
#kafka #development #software-development #cloud-computing #microservices
