How to Sagas in a Distributed World: Utopia or Dystopia

In my previous article, Legacy Modernization or Chasing Rainbows, I discussed the myriad complexities and challenges we encounter when migrating our legacy monolith systems to microservices. In the course of this journey, we often arrive at the Data-Decomposition phase, where we encounter what we had taken for granted in our monolith: ACID Transactions of RDBMS.

To elaborate, a Relational Database (RDBMS) must be Atomic, Consistent, Isolated, and Durable (ACID) to ensure the correctness of a transaction. The Atomicity ensures that either all operations in the transaction succeed or all fail. Consistency takes data from one valid state to another valid state. Isolation guarantees that concurrent transactions behave as if they are running sequentially and not step on each other. Lastly, Durability means that committed transactions are persisted irrespective of any type of system failure. In a distributed transaction scenario, as the transaction spans several services, it always remains a key concern to ensure ACID.

#microservices 

What is GEEK

Buddha Community

How to Sagas in a Distributed World: Utopia or Dystopia
August  Larson

August Larson

1624298520

Sampling Distributions with Python

College Statistics with Python

Introduction

In a series of weekly articles, I will be covering some important topics of statistics with a twist.

The goal is to use Python to help us get intuition on complex concepts, empirically test theoretical proofs, or build algorithms from scratch. In this series, you will find articles covering topics such as random variables, sampling distributions, confidence intervals, significance tests, and more.

At the end of each article, you can find exercises to test your knowledge. The solutions will be shared in the article of the following week.

Articles published so far:

As usual, the code is available on my GitHub.

#statistics #distribution #python #machine-learning #sampling distributions with python #sampling distributions

How to Sagas in a Distributed World: Utopia or Dystopia

In my previous article, Legacy Modernization or Chasing Rainbows, I discussed the myriad complexities and challenges we encounter when migrating our legacy monolith systems to microservices. In the course of this journey, we often arrive at the Data-Decomposition phase, where we encounter what we had taken for granted in our monolith: ACID Transactions of RDBMS.

To elaborate, a Relational Database (RDBMS) must be Atomic, Consistent, Isolated, and Durable (ACID) to ensure the correctness of a transaction. The Atomicity ensures that either all operations in the transaction succeed or all fail. Consistency takes data from one valid state to another valid state. Isolation guarantees that concurrent transactions behave as if they are running sequentially and not step on each other. Lastly, Durability means that committed transactions are persisted irrespective of any type of system failure. In a distributed transaction scenario, as the transaction spans several services, it always remains a key concern to ensure ACID.

#microservices 

Ian  Robinson

Ian Robinson

1623263280

Data Distribution in Apache Ignite

This blog is an abridged version of the talk that I gave at the Apache Ignite community meetup. You can download the slides that I presented at the meetup here. In the talk, I explain how data in Apache Ignite is distributed.

Why Do You Need to Distribute Anything at all?

Inevitably, the evolution of a system that requires data storage and processing reaches a threshold. Either too much data is accumulated, so the data simply does not fit into the storage device, or the load increases so rapidly that a single server cannot manage the number of queries. Both scenarios happen frequently.

Usually, in such situations, two solutions come in handy—sharding the data storage or migrating to a distributed database. The solutions have features in common. The most frequently used feature uses a set of nodes to manage data. Throughout this post, I will refer to the set of nodes as “topology.”

The problem of data distribution among the nodes of the topology can be described in regard to the set of requirements that the distribution must comply with:

  1. Algorithm. The algorithm allows the topology nodes and front-end applications to discover unambiguously on which node or nodes an object (or key) is located.
  2. Distribution uniformity. The more uniform the data distribution is among the nodes, the more uniform the workloads on the nodes is. Here, I assume that the nodes have approximately equal resources.
  3. Minimal disruption. If the topology is changed because of a node failure, the changes in distribution should affect only the data that is on the failed node. It should also be noted that, if a node is added to the topology, no data swap should occur among the nodes that are already present in the topology.

#tutorial #big data #distributed systems #apache ignite #distributed storage #data distribution #consistent hashing

Ray  Patel

Ray Patel

1623896372

Home Assistant custom integration for e-distribution with python

edistribucion

e-Distribución is an energy distribution company that covers most of South Spain area. If you live in this area, you probably are able to register into their website to get some information about your power demand, energy consumption, or even cycle billing (in terms of consumptions).

Although their application is great, this integration enables you to add a sensor to Home Assistant and getting updated automatically. However, it has some limitations yet, and no front-end support is being provided at the moment.

How to install

  1. Install HACS
  2. Add this repo (https://github.com/uvejota/edistribucion) to the custom repositories in HACS
  3. Install the integration. Please consider that alpha/beta versions are untested, and they might cause bans due to excesive polling.
  4. Add this basic configuration at Home Assistant configuration files (e.g., configuration.yml)
sensor:
  - platform: edistribucion
    username: !secret eds_user ## this key may exist in secrets.yaml!
    password: !secret eds_password ## this key may exist in secrets.yaml!

YAML

At this point, you got an unique default sensor for the integration, namely sensor.edistribucion, linked to those credentials in the e-Distribución platform. This default sensor assumes the first CUPS that appears in the fetched list of CUPS, which frequently is the most recent contract, so this configuration may be valid for most users. If you need a more detailed configuration, please check the section below “What about customisation?”.

#machine learning #distribution #python #home assistant custom integration for e-distribution with python #home assistant #e-distribution with python

Seamus  Quitzon

Seamus Quitzon

1599094027

Distributed Saga and Resiliency of Microservices

Sagas are typically used for modeling long-lived transactions like those involved in workflows. It is not advisable to use two-phase transaction protocols to control long-lived transactions since the locking of resources for prolonged durations across trust boundaries is not practical, rather is not at all advisable. Sagas are similar to nested transactions. In a nested transaction, atomic transactions are embedded in other transactions. In sagas, each of these transactions has a corresponding compensating transaction. While a Saga proceeds with its steps, if any of the transactions in a saga fails, the compensating actions for each transaction that was successfully run previously will be invoked so as to nullify the effect of the previously successful transactions.

Modeling a Saga and setting up a Saga Infrastructure is rather straight forward in Local and simple deployments, however when we want to scale out in public cloud environments, and that too with multiple instances of the same type of microservice, there comes a new list of challenges. We will look at few of them in this discussion.

Setting the Context for Saga

We will not attempt to explain what a Saga is in more details here since it’s covered well in the DZone article titled “Distributed Sagas for Microservices.” The sample described there is an apt business case for our discussion, and we will use that in our explanation here. The scenario is that of doing a Travel booking. Many Travel agent booking systems provide this feature where they aggregate multiple kinds of travel inventories from different enterprises who actually own the inventory. It’s very unlikely that a single enterprise owns inventory for all these resources, but it’s highly likely that the end-user wants to book one or more of these resources in a single transaction because a confirmed hotel booking with a non-confirmed flight booking is not very useful for him!

In our case, The Travel agent enterprise refers to www.makeyourtrip.com. Travelers can come to this web site and reserve a complete travel package, which involves booking inventories of multiple types.

Let’s assume that makeyourtrip has got a partnership with 3 other enterprises for retrieving inventories, as follows:

The microservices for business operations for above enterprises can be portrayed as illustrated in Figure 01.

Figure 01 Orchestration based Saga using HTTP

We will use a variant of the Hexagonal Architecture representation to depict each microservice. Figure 01 shows such a representation, and we have 4 such microservices each corresponding to the 4 enterprises mentioned earlier. A booking request from a traveler will come through the internet and hits the travel agent application first. We will assume that all that is required as input parameters to carry out a complete travel booking is captured in a single request from the traveler’s user agent device by the travel agent’s microservice, which is Trip microservice.

Referring to Figure 01, let us look at a typical sequence of actions:

  1. Traveler sends a “Book Trip” request from the browser, which will hit the Trip microservice.
  2. The Trip microservice is responsible for starting Saga. It calls on what is called as a Saga coordinator endpoint, starting a Saga. The coordinator announces a Saga identifier in response. The Trip microservice enlists itself with the created Saga by calling the Saga coordinator providing the Saga identifier and handing over addresses of REST endpoints for compensation (optionally confirmation) callbacks. Those are endpoint the coordinator can call back in response to the outcome of the execute action on any participating microservices.
  3. The Trip microservice takes the Saga ID and adds it as an HTTP header to the REST call to the 3 other fulfilling microservices, Cab Microservice, Hotel Microservice and Flight Microservice.
  4. The called microservices distinguishes the Saga and they can enlist themselves (by announcing REST endpoints for compensation/confirmation callbacks) to Saga coordinator.
  5. The participant microservices, viz. Cab Microservice, Hotel Microservice and Flight Microservice executes the business process
  6. Any of the participant microservices could fail the Saga by calling “Execution Failure” on the Saga coordinator.
  7. Saga coordinator sends commands either to confirm or to compensate, to all participants
  8. On way back, the initiator microservice is responsible for finishing the Saga by calling complete (with a success or failure flag) on the Saga coordinator with the saga identifier.

#travel booking #microservice architecture #orchestration #distributed transactions #saga pattern #kafka partitions #choreography #cqrs in php