Before you get the wrong idea, I’m a big fan of Kafka. It does a great job but all too often it’s setting your architecture up for failure.

So, what is Kafka good at?

**High performance and throughput **— Kafka is great at handling large volumes of messages concurrently and dealing with disparate consumers sending and receiving from it

**Data persistence **— replication and automatic failover of partitions is ideal for ensuring your data is safe and can provide accurate feedback on whether messages are persisted

**Asynchronous messaging **— the ability for one service to fire and forget a message, and another to pick it up when it’s ready is well catered for, this isn’t specific to Kafka, but in its comfort zone

(and more)

This sets Kafka up to be at the core of your system, your services can plug into it, they can offload messages, and then they’re stored and readable as needed.

But what have you ultimately done? You’ve turned Kafka into an Enterprise Service Bus.

Don’t use Kafka as an Enterprise Service Bus

An Enterprise Service Bus is an older term now, Wikipedia will tell you it’s from around the turn of the century. The relevance is that it’s exactly this model of building your services around a central service, lots of input and output from that one service, and using it as a routing and processing engine. The key issue here is that whilst Kafka is a redundant, reliable system, you are still building it into the all-seeing, all-knowing centralized service that everything hangs off.

As with all services, failure is inevitable, things can be well built, hugely scalable, nicely distributed, but eventually, be it by strange accident or human failure, something will go wrong. This is the point that your microservice architecture will be there to save you.

Microservices will save you because they’re built around point to point communications, they use distributed architecture to remove central points of failure. Still working on the premise that failure is inevitable, you should be looking at your services thinking if a particular service fails, it will affect this defined segment of functionality, but all the other services that don’t require it will continue to work as normal.

If all your services depend on a central Kafka, Kafka failure could cause you a complete systems failure, so from a reliability point of view, you’ve built your typical distributed monolith. A complete failure isn’t necessarily the only outcome here, but if all your systems rely on a single component, then you have a large and central dependency that could be detrimental to your whole platform.

#streaming #events #kafka #microservices #architecture