Saga Pattern: Compensating Distributed Transactions

In today’s web world with an increase in the load to monolithic web application a new system design came into existence that is called microservice architecture design. Microservice architecture design helps us to structure an application as a collection of services that are loosely coupled and independently deployable. As there are many benefits of using microservice architecture but it too comes with some challenges. One of the main challenges is to handle the distributed transactions.

In this tutorial, we will discuss about the distributed transactions and how to handle it using the Saga pattern.

Assuming we have three microservices (service, service2, and service3) and based on some transactions that happened in service1, it will trigger an event which calls the next microservice i.e. step 2 and similarly service2 will call service3 microservice.

1. service1

2. service2

3. service3

and we have a distributed transaction use case.

1. If service1 transaction succeeds call service2 and if service2 transaction succeeds call service3 and if service3 transaction succeeds mark the transaction as completed.

2.If service1 transaction succeeds call service2 and if service2 transaction fails mark the transaction as failed and rollback the service1 transaction.

3.If service1 transaction succeeds call service2 and if service2 transaction succeeds call service3 and if service3 transaction fails mark the transaction as fail and rollback the service2 and service1 transaction.

4.If service1 transaction fails mark the transaction as fail and rollback the transaction.

This is where Saga pattern comes to rescue.

Saga pattern is an asynchronous and event-driven design pattern which consists of a sequential/parallel chain of microservices (steps) which are been called based on some events(transactions) that occur from within a single microservice.

We can also say Saga pattern is a fail-over and compensating handling pattern, so whenever there is any failure, it will execute the corresponding compensating action to return the initial state from where the Saga flow started.

So to implement the saga pattern one of the best solution is to use AWS step function.

Lets understand what is Step function and how it can fit to our use case.

Step functions are basically an orchestration service for AWS Lambda and activity-based tasks.It makes easy to coordinate microservices using visual workflow.

• State machine: In simple word the workflow which we build using step function is called a state machine. It describes how to propagate inputs from one state to the next
• **AWS States: **It’s a way to tell the state machine to execute some task and there are seven types of AWS states you can have:

1. Task

2. Pass

3. Wait

4. Succeed

5. Fail

6. Choice

7. Input and Output

For more details about the task please refer the AWS - AWS Step Function

Now as we understand state machine consist of different states and states consist of task.

Let’s develop the state machine for our use case.

Firstly we will create Service1 as one of the state as mention in below code

#cloud #aws #design patterns #saga pattern #aws step functions

What is GEEK

Buddha Community

Saga Pattern: Compensating Distributed Transactions

In today’s web world with an increase in the load to monolithic web application a new system design came into existence that is called microservice architecture design. Microservice architecture design helps us to structure an application as a collection of services that are loosely coupled and independently deployable. As there are many benefits of using microservice architecture but it too comes with some challenges. One of the main challenges is to handle the distributed transactions.

In this tutorial, we will discuss about the distributed transactions and how to handle it using the Saga pattern.

Assuming we have three microservices (service, service2, and service3) and based on some transactions that happened in service1, it will trigger an event which calls the next microservice i.e. step 2 and similarly service2 will call service3 microservice.

1. service1

2. service2

3. service3

and we have a distributed transaction use case.

1. If service1 transaction succeeds call service2 and if service2 transaction succeeds call service3 and if service3 transaction succeeds mark the transaction as completed.

2.If service1 transaction succeeds call service2 and if service2 transaction fails mark the transaction as failed and rollback the service1 transaction.

3.If service1 transaction succeeds call service2 and if service2 transaction succeeds call service3 and if service3 transaction fails mark the transaction as fail and rollback the service2 and service1 transaction.

4.If service1 transaction fails mark the transaction as fail and rollback the transaction.

This is where Saga pattern comes to rescue.

Saga pattern is an asynchronous and event-driven design pattern which consists of a sequential/parallel chain of microservices (steps) which are been called based on some events(transactions) that occur from within a single microservice.

We can also say Saga pattern is a fail-over and compensating handling pattern, so whenever there is any failure, it will execute the corresponding compensating action to return the initial state from where the Saga flow started.

So to implement the saga pattern one of the best solution is to use AWS step function.

Lets understand what is Step function and how it can fit to our use case.

Step functions are basically an orchestration service for AWS Lambda and activity-based tasks.It makes easy to coordinate microservices using visual workflow.

• State machine: In simple word the workflow which we build using step function is called a state machine. It describes how to propagate inputs from one state to the next
• **AWS States: **It’s a way to tell the state machine to execute some task and there are seven types of AWS states you can have:

1. Task

2. Pass

3. Wait

4. Succeed

5. Fail

6. Choice

7. Input and Output

For more details about the task please refer the AWS - AWS Step Function

Now as we understand state machine consist of different states and states consist of task.

Let’s develop the state machine for our use case.

Firstly we will create Service1 as one of the state as mention in below code

#cloud #aws #design patterns #saga pattern #aws step functions

The Saga Pattern in Microservices

Handling distributed transactions.

When everything is normal, the API composition and CQRS patterns can provide appropriate solutions for distributed queries.

However, maintaining the integrity of distributed data is complex. If you store all data in a single relational database and specify the appropriate constraints in the schema, you can rely on the database engine to maintain data integrity.

The situation is very different when multiple microservices keeps the data in isolated data stores (relational or non-relational). Data integrity is critical, but it must be maintained by code. The saga pattern addresses this concern.

A common measure of data integrity is that all transactions that modify data have the ACID properties:

• Atomic: All operations in the transaction succeed or all fail.
• Consistent: The data state meets all constraints before and after the transaction
• Isolated: Concurrent transactions behave as if they are serialized.
• Durable: When a transaction completes successfully, the results are persisted.

#saga #transactions #patterns #microservices

Builder Design Pattern

What is Builder Design Pattern ? Why we should care about it ?

Starting from **Creational Design Pattern, **so wikipedia says “creational design pattern are design pattern that deals with object creation mechanism, trying to create objects in manner that is suitable to the situation”.

The basic form of object creations could result in design problems and result in complex design problems, so to overcome this problem Creational Design Pattern somehow allows you to create the object.

Builder is one of the** Creational Design Pattern**.

When to consider the Builder Design Pattern ?

Builder is useful when you need to do lot of things to build an Object. Let’s imagine DOM (Document Object Model), so if we need to create the DOM, We could have to do lot of things, appending plenty of nodes and attaching attributes to them. We could also imagine about the huge XML Object creation where we will have to do lot of work to create the Object. A Factory is used basically when we could create the entire object in one shot.

As **Joshua Bloch (**He led the Design of the many library Java Collections Framework and many more) – “Builder Pattern is good choice when designing the class whose constructor or static factories would have more than handful of parameters”

#java #builder #builder pattern #creational design pattern #design pattern #factory pattern #java design pattern

Distributed Saga and Resiliency of Microservices

Sagas are typically used for modeling long-lived transactions like those involved in workflows. It is not advisable to use two-phase transaction protocols to control long-lived transactions since the locking of resources for prolonged durations across trust boundaries is not practical, rather is not at all advisable. Sagas are similar to nested transactions. In a nested transaction, atomic transactions are embedded in other transactions. In sagas, each of these transactions has a corresponding compensating transaction. While a Saga proceeds with its steps, if any of the transactions in a saga fails, the compensating actions for each transaction that was successfully run previously will be invoked so as to nullify the effect of the previously successful transactions.

Modeling a Saga and setting up a Saga Infrastructure is rather straight forward in Local and simple deployments, however when we want to scale out in public cloud environments, and that too with multiple instances of the same type of microservice, there comes a new list of challenges. We will look at few of them in this discussion.

Setting the Context for Saga

We will not attempt to explain what a Saga is in more details here since it’s covered well in the DZone article titled “Distributed Sagas for Microservices.” The sample described there is an apt business case for our discussion, and we will use that in our explanation here. The scenario is that of doing a Travel booking. Many Travel agent booking systems provide this feature where they aggregate multiple kinds of travel inventories from different enterprises who actually own the inventory. It’s very unlikely that a single enterprise owns inventory for all these resources, but it’s highly likely that the end-user wants to book one or more of these resources in a single transaction because a confirmed hotel booking with a non-confirmed flight booking is not very useful for him!

In our case, The Travel agent enterprise refers to www.makeyourtrip.com. Travelers can come to this web site and reserve a complete travel package, which involves booking inventories of multiple types.

Let’s assume that makeyourtrip has got a partnership with 3 other enterprises for retrieving inventories, as follows:

• www.yertz.com - Can Rental Services
• www.jilton.com - Hotel Room Rental Services
• www.selta.com - Flight Booking Services

The microservices for business operations for above enterprises can be portrayed as illustrated in Figure 01.

We will use a variant of the Hexagonal Architecture representation to depict each microservice. Figure 01 shows such a representation, and we have 4 such microservices each corresponding to the 4 enterprises mentioned earlier. A booking request from a traveler will come through the internet and hits the travel agent application first. We will assume that all that is required as input parameters to carry out a complete travel booking is captured in a single request from the traveler’s user agent device by the travel agent’s microservice, which is Trip microservice.

Referring to Figure 01, let us look at a typical sequence of actions:

1. Traveler sends a “Book Trip” request from the browser, which will hit the Trip microservice.
2. The Trip microservice is responsible for starting Saga. It calls on what is called as a Saga coordinator endpoint, starting a Saga. The coordinator announces a Saga identifier in response. The Trip microservice enlists itself with the created Saga by calling the Saga coordinator providing the Saga identifier and handing over addresses of REST endpoints for compensation (optionally confirmation) callbacks. Those are endpoint the coordinator can call back in response to the outcome of the execute action on any participating microservices.
3. The Trip microservice takes the Saga ID and adds it as an HTTP header to the REST call to the 3 other fulfilling microservices, Cab Microservice, Hotel Microservice and Flight Microservice.
4. The called microservices distinguishes the Saga and they can enlist themselves (by announcing REST endpoints for compensation/confirmation callbacks) to Saga coordinator.
5. The participant microservices, viz. Cab Microservice, Hotel Microservice and Flight Microservice executes the business process
6. Any of the participant microservices could fail the Saga by calling “Execution Failure” on the Saga coordinator.
7. Saga coordinator sends commands either to confirm or to compensate, to all participants
8. On way back, the initiator microservice is responsible for finishing the Saga by calling complete (with a success or failure flag) on the Saga coordinator with the saga identifier.

#travel booking #microservice architecture #orchestration #distributed transactions #saga pattern #kafka partitions #choreography #cqrs in php

Saga Orchestration for Microservices Using the Outbox Pattern

• Sagas allow for the implementation of long-running, distributed business transactions, executing a set of operations across multiple microservices, applying consistent all-or-nothing semantics.
• For the sake of decoupling, communication between microservices should preferably happen asynchronously, for instance using distributed commit logs like Apache Kafka.
• The outbox pattern provides a solution for service authors to perform writes to their local database and send messages via Apache Kafka, without relying on unsafe “dual writes.”
• Debezium, a distributed open-source change data capture platform, provides a robust and flexible foundation for orchestrating Saga flows using the outbox pattern.

When moving to microservices, one of the first things to realize is that individual services don’t exist in isolation. While the goal is to create loosely coupled, independent services with as little interaction as possible, chances are high that one service needs a particular data set owned by another service, or that multiple services need to act in concert to achieve a consistent outcome of an operation in the domain of our business.

The outbox pattern,  implemented via change data capture, is a proven approach for addressing the concern of data exchange between microservices; avoiding any unsafe “dual writes” to multiple resources, e.g., a database and a message broker, the outbox pattern achieves eventually consistent data exchange, without depending on the synchronous availability of all participants, and not requiring complex protocols such as XA (a widely  used standard for distributed transaction processing defined by  The Open Group) either.

In this article, I’d like to explore how to take the outbox pattern to the next level and use it for implementing Sagas—potentially long-running business transactions that span across multiple microservices. One common example is that of booking a trip comprising multiple parts: either all flight legs and accommodation should be booked together or none of them. Sagas split up one such overarching business transaction into a series of multiple local database transactions, that are executed by the participating services.

#microservices #saga pattern #transactions #debezium #article