Recently I wrote about the challenges of event sourcing and CQRS in this article “What they don’t tell you about event sourcing”. A manager once told me “If you bring me a problem and don’t bring me the solution then you are the problem”. In light of this, I will address most of the issues I raised in the previous article and share my experience on how to tackle them. The topic of this article it’s arguably the hardest and most frustrating characteristic of distributed systems, eventual consistency.
The silver bullet
Ben Horowitz once said that there are no silver bullets. But to eventual consistency there is, the holy grail to deal with eventual consistency, the one fits all solution, perhaps the greatest secret of our time:
…
drum roll
…
Do not use an eventually consistent model. Yeah, that’s it. If I was sitting next to you I’m sure I could hear your eyes roll, but bear with me.
One thing people often do is decide the technical implementation disregarding the business implications. As long as the solution complies with the requirements it is fine to use the sexiest, most hyped pattern or technology in the market.
Before jumping the gun on a decision like this you should sit down with the business and discuss the implications on the people using the system. Get first an idea of the quantity of the data, the performance and availability requirements, and whether or not your business can be eventually consistent. Be discerning and rigorous about the decision. Do you need to be eventually consistent? The quantity of data is often an excuse to use it. If your database fits inside a pen you don’t need big data. Just use your friendly neighborhood SQL or any other ACID technology. You are not just making the business life easier, you are making it easier by avoiding the extra complexity for every other developer who will have to maintain and increment on that system.
The first step for dealing with eventual consistency is not dealing with it at all.
It is crucial to understand where introducing eventual consistency will have an impact and where it won’t. Another important factor to take into account besides the users using the functionality are the consumers of that information. If another service depends on it to process a business rule, relying on an eventual consistency model will likely lead to erroneous results if the correct measures aren’t in place. Even when they are, the complexity of the consumer system will increase substantially.
Having said that, if your business is large enough the time will come when you will need eventual consistency. Sooner or later, when walking towards a microservice architecture, for example, eventual consistency will be needed due to all the distributed data models or due to availability constraints. It’s also fairly common when using an event-driven architecture due to the services reacting to those events which are intrinsically asynchronous.
Then the Sword of Damocles will hang over your head. How do you keep that horse’s tail hair from breaking you ask? Well here are some pointers:
End to End Argument
The end to end argument is commonly mentioned in the context of computer networking and states that:
“…functions placed at low levels of a system may be redundant or of little value when compared with the cost of providing them at that low level.”
The same reasoning can be applied to a microservice architecture and can be especially useful when dealing with eventual consistency. Typically eventual consistency comes out due to the sheer scale of the data and due to performance reasons. This, however, doesn’t mean every single component in the architecture has to be eventually consistent, e.g. an eCommerce platform might have hundreds of millions of stock units, millions of products, and a handful of product categories. This implies the system to manage stock units would indeed benefit from an eventually consistent model or of an asynchronous message queue of some kind, on the other hand, the system that manages categories would hardly benefit from it. The user order flow will use all three of them along with several others. If the choice is made to make the stock information eventually consistent, the platform’s product search will also likely be eventually consistent, thus the stock information will be eventually accurate, e.g. a user can see stock on a product that doesn’t have stock. Nevertheless, as long as the checkout process has a way to validate the correct stock on the moment of the purchase it is possible to benefit from the eventually consistent model in that specific part of the architecture and the performance boost that comes with it.
The business process will most likely flow through several microservices and we can choose to make specific systems inside that flow eventually consistent, prioritizing performance where needed as long as there is a way to guarantee the correctness of the end to end flow.
Event? I don’t think I was invited to that.
Events are our most popular answer for reactive solutions and are the foundation for many microservice architectures, typically event-driven ones. However, they imply eventual consistency of some sort and the worst kind of eventual consistency: the one you choose to create. Why it is the worst kind? It’s the worst kind simply because you can’t blame it on some else obviously or on some shady database technology you are using for the first time.
The meaning of an event and how it is used can vary from system to system. The interaction between the publishing system and how the consumers handle the event has consequences throughout the whole microservice architecture. They usually imply eventual consistency due to the asynchronous queues between the domains. How to deal with this eventual consistency inside the system can be challenging. These usually patterns stand out:
Town Crier Events
This pattern uses events as simple notifications. The source system publishes an event notifying the consumers that something changed in its domain. Then the consumers will request additional information to the source system. E.g. if a user changes the door number of his address the system managing the user domain will publish a user address changed event with perhaps the id of the user and with the new door number. Using an approach with DDD in mind the events should reflect the intent of the user. But how many systems will be able to act on the door number alone without for example the remaining of the address? The consumers of the event will then have to request the source system for further information.
This approach provides a high level of decoupling between the domains. The user’s address information will only persist in the user domain. This allows for each domain to evolve seamlessly and independently from each other, along with the business needs with little impact to the consumers.
#event-driven-architecture #software-architecture #software-development #eventual-consistency #microservices #microservice