Eleo Nona

Eleo Nona

1572236357

Microservices: Implementing the Outbox Pattern

The Problem Statement

Microservices often publish events after performing a database transaction. Writing to the database and publishing an event are two different transactions and they have to be atomic. A failure to publish an event can mean critical failure to the business process.

To explain the problem statement better, let’s consider a Student microservice that helps Enroll the student. After enrollment, the “Course Catalog” service, emails the student all the available courses. Assuming an Event-Driven application, the Student microservice enrolls the student by inserting a record in the database and publishes an event stating that the enrollment for the student is complete. The “Course Catalog” service listens to this event and performs its actions. In a failure scenario, if the Student microservice goes down after inserting a record, the system would be left in an inconsistent state.

Image 1. Failure while publishing an event after updating/inserting the database

The OutBox Pattern

This pattern provides an effective solution to publish events reliably. The idea of this approach is to have an “Outbox” table in the service’s database. When receiving a request for enrollment, not only an insert into the Student table is done, but a record representing the event is also inserted into the Outbox table. The two database actions are done as part of the same transaction.

An asynchronous process monitors the Outbox table for new entries and if there are any, it publishes the events to the Event Bus. The pattern merely splits the two transactions over different services, increasing reliability.

Image 2. Two separate transactions using the outbox pattern.

A description of this pattern can be found on Chris Richardson’s excellent microservices.io site. As described on the site there are two approaches to implementing the Outbox pattern (Transaction log tailing and Polling publisher). We will be using the log tailing approach in the solution below.

Transaction log tailing can be implemented in a very elegant and efficient way using Change Data Capture (CDC) with Debezium and Kafka-Connect.

Outbox Pattern With Kafka Connect

Solution Design

The Student microservice exposes endpoints to perform database operations on the domain. The microservice uses a Postgres database, which houses two tables “Student” and “Outbox”. The transactional operations, modify/insert into the “Student” table and adds a record in the “Outbox” table.

The Kafka-Connect framework runs as a separate service besides the Kafka broker. The Debezium connector for Postgres is deployed on the Kafka-Connect runtime, to capture the changes on the database. In our example, a custom connector is also deployed within Kafka-Connect to help identify the right Kafka topics for an event.

The Debezium connector tails the database transaction logs (write-ahead log) from the ‘‘Outbox’’ table and publishes an event to the topics defined by the custom connector.

Image 3. Solution Design

This solution guarantees at-least-once delivery, since Kafka Connect services ensure that each connector is always running; however, there is a chance the solution can publish the same event multiple times between connectors going down and starting up. To ensure exactly-once delivery, the consuming client must be Idempotent, making sure the duplicate events aren’t processed again.

Understanding the Code

You can find the code here. I would encourage you to read through the story – since I have walked through some key implementation details and the limitations of this pattern.

Student Microservice

This is a simple Spring-Boot microservice, which exposes three endpoints via the REST controller and uses Spring-JPA for database actions. The endpoints exposed are a GET for fetching student information, POST for creating or enrolling a student and a PUT for updating the student email address. The POST and the PUT generate events ‘Student Enrolled’ and ‘Student Email Changed’. The change to invoke the Database actions and inserting the event is handled in the Service class.

@Transactional
    public StudentDTO enrollStudent(EnrollStudentDTO student) 
      throws Exception {
        log.info("Enroll Student details for StudentId: {}", 
                 student.getName());

        StudentEntity studentEntity = StudentMapper.
          INSTANCE.studentDTOToEntity(student);
        studentRepository.save(studentEntity);

        //Publish the event
        event.fire(EventUtils.createEnrollEvent(studentEntity));

        return StudentMapper.INSTANCE.studentEntityToDTO(studentEntity);
    }

    ...

    public static OutboxEvent createEnrollEvent(StudentEntity studentEntity) 
    {
      ObjectMapper mapper = new ObjectMapper();
      JsonNode jsonNode = mapper.convertValue(studentEntity, JsonNode.class);

      return new OutboxEvent(
              studentEntity.getStudentId(),
              "STUDENT_ENROLLED",
              jsonNode
      );
    }

The method needs the Transactional annotation so that database action and the event write is bound by a single transaction. The enrollStudent() creates a new record on the Student table and then fires an event using Spring’s [ApplicationEventPublisherAware](https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/context/ApplicationEventPublisherAware.html) support. The method createEnrollEvent(), helps build the data to be inserted into the OutBox. Inserting the event into the ‘Outbox’ table is handled in the EventService class which uses a Spring-JPA Repository to handle the database interactions.

@EventListener
    public void handleOutboxEvent(OutboxEvent event) {

        UUID uuid = UUID.randomUUID();
        OutBoxEntity entity = new OutBoxEntity(
                uuid,
                event.getAggregateId(),
                event.getEventType(),
                event.getPayload().toString(),
                new Date()
        );

        log.info("Handling event : {}.", entity);

        outBoxRepository.save(entity);

        /*
         * Delete the event once written, so that the outbox doesn't grow.
         * The CDC eventing polls the database log entry and not the table in the database.
         */
        outBoxRepository.delete(entity);
    }

A key thing to note here is the code deletes the record on the ‘Outbox’ Table once it has been written so that the outbox table doesn’t grow. Also, Debezium doesn’t examine the actual contents of the database table, but instead it tails the write-ahead transaction log. The calls to save() and delete() will make a CREATE and a DELETE entry in the log, once the transaction commits. The Kafka-Connect custom transformer can be programmed not to perform any action on the DELETE entry.

Custom Debezium Transformer

This component determines the Kafka topic to which the event needs to be published. This is done by using the EVENT_TYPE column of the payload from the ‘Outbox’ table. The component is built as a JAR and will be placed in the Kafka-Connect runtime. The setup of placing the JAR in the Kafka-Connect runtime is handled by the DockerFile.

FROM debezium/connect
ENV DEBEZIUM_DIR=$KAFKA_CONNECT_PLUGINS_DIR/debezium-transformer

RUN mkdir $DEBEZIUM_DIR
COPY target/custom-debezium-transformer-0.0.1.jar $DEBEZIUM_DIR

We use the image debezium/connect, since it comes preloaded with all available connectors. The component consists of just one class that helps determine the topic before the message is published.

public class CustomTransformation<R extends ConnectRecord<R>> implements Transformation<R> {

    /**
     * This method is invoked when a change is made on the outbox schema.
     *
     * @param sourceRecord
     * @return
     */
    public R apply(R sourceRecord) {

        Struct kStruct = (Struct) sourceRecord.value();
        String databaseOperation = kStruct.getString("op");

        //Handle only the Create's
        if ("c".equalsIgnoreCase(databaseOperation)) {

            // Get the details.
            Struct after = (Struct) kStruct.get("after");
            String UUID = after.getString("uuid");
            String payload = after.getString("payload");
            String eventType = after.getString("event_type").toLowerCase();
            String topic = eventType.toLowerCase();

            Headers headers = sourceRecord.headers();
            headers.addString("eventId", UUID);

            // Build the event to be published.
            sourceRecord = sourceRecord.newRecord(topic, null, Schema.STRING_SCHEMA, UUID,
                    null, payload, sourceRecord.timestamp(), headers);
        }

        return sourceRecord;
    }

The transformer extends the Kafka-Connect Transformation class. The [apply()](https://docs.spring.io/spring-kafka/api/org/springframework/kafka/core/KafkaTemplate.html) method, filters the CREATE operation (‘c’) skipping the DELETE, as explained above.

For every CREATE the topic name is identified and the payload is returned. For simplicity in this example, the topic name is the lowercase value of the EVENT_TYPE column, inserted into the “Outbox” table by the Student Microservice.

Installation of the Needed Images and Frameworks

The guide assumes the user has docker pre-installed. Creating the Debezium Connect Image is done by triggering a maven build on the custom-debezium-connect project and building the docker image.

mvn clean install
docker build -t custom-debezium-connect .

Running the Docker Compose under the project folder installs all the pre-requisites: Zookeeper, Kafka, Postgres, and Kafka-Connect. The Docker Compose file:

version: "3.5"

services:
  # Install postgres and setup the student database.
  postgres:
    container_name: postgres
    image: debezium/postgres
    ports:
      - 5432:5432
    environment:
      - POSTGRES_DB=studentdb
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=password

  # Install zookeeper. 
  zookeeper:
    container_name: zookeeper
    image: zookeeper
    ports:
      - 2181:2181

  # Install kafka and create needed topics. 
  kafka:
    container_name: kafka
    image: confluentinc/cp-kafka
    hostname: kafka
    ports:
      - 9092:9092
      - 29092:29092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_HOST://kafka:29092
      LISTENERS: PLAINTEXT://0.0.0.0:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_CREATE_TOPICS: "student_email_changed:1:1,student_enrolled:1:1"
    depends_on:
      - zookeeper

  # Install debezium-connect. 
  debezium-connect:
    container_name: custom-debezium-connect
    image: custom-debezium-connect
    hostname: debezium-connect 
    ports:
      - '8083:8083'
    environment:
      GROUP_ID: 1
      CONFIG_STORAGE_TOPIC: debezium_connect_config
      OFFSET_STORAGE_TOPIC: debezium_connect_offsets
      STATUS_STORAGE_TOPIC: debezium_connect_status
      BOOTSTRAP_SERVERS: kafka:29092
    depends_on:
      - kafka
      - postgres

We use the image debezium/postgres, because it comes prebuilt with the logical decoding feature. This is a mechanism that allows the extraction of the changes, that were committed to the transaction log making the CDC possible. The documentation for installing the plugin to Postgres can be found here.

Setting Up the Kafka Topics

Execute the below commands to create the two Kafka topics: “student_enrolled” and “student_email_changed

docker exec -t kafka /usr/bin/kafka-topics \
      --create --bootstrap-server :9092 \
      --topic student_email_changed \
      --partitions 1 \
      --replication-factor 1

docker exec -t kafka /usr/bin/kafka-topics \
      --create --bootstrap-server :9092 \
      --topic student_enrolled \
      --partitions 1 \
      --replication-factor 1

Linking the Debezium Kafka Connect With the Outbox Table

Execute the below curl command to create a connector in the Kafka-Connect server. This connector points to the Postgres installation and also specifies the table and the custom transformer class we built earlier.

curl -X POST \
  http://localhost:8083/connectors/ \
  -H 'content-type: application/json' \
  -d '{
   "name": "student-outbox-connector",
   "config": {
      "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
      "tasks.max": "1",
      "database.hostname": "postgres",
      "database.port": "5432",
      "database.user": "user",
      "database.password": "password",
      "database.dbname": "studentdb",
      "database.server.name": "pg-outbox-server",
      "tombstones.on.delete": "false",
      "table.whitelist": "public.outbox",
      "transforms": "outbox",
      "transforms.outbox.type": "com.sohan.transform.CustomTransformation"
   }
}'

That completes the setup needed, where we have Zookeeper running on port 2181, Kafka running on port 9092 with all the needed topics, Postgres running on port 5432 having a ‘StudentDB’ pre-created and finally the Kafka-Connect with Debezium and our custom transformer running on port 8083.

Running the Solution

Once the Student Microservice isstarted we can see the pattern in action. To simulate a Student enrollment, we can execute the below curl.

curl -X POST \
  'http://localhost:8080/students/~/enroll' \
  -H 'content-type: application/json' \
  -d '{ 
    "name": "Megan Clark",
    "email": "mclark@gmail.com",
    "address": "Toronto, ON"
}'

We see that a new student record is inserted into the database for ‘Megan Clark’.

Image 4. Student Enrolled inserted into the database

And we see an event published into the topic student_enrolled, notifying the downstream systems that ‘Megan Clark’ has enrolled.

Image 5. Console consumer to verify data being published to Kafka

To simulate a student updating the email address, we can execute the below curl operation.

$ curl -X PUT \  http://localhost:8080/students/1/update-email \  
-H 'content-type: application/json' \  -d '{    "email": "jsmith@gmail.com"}'

We can notice that email has been changed into ‘jsmith@gmail.com’

Image 6. Student Email changed in the database

And we see an event published into the topic student_email_changed, notifying the downstream systems that Student with Student-ID ‘1’ has changed his email id.

Image 6. Console consumer to verify data being published to Kafka

If we comment on the line of code that deletes outbox events after writing them in the EventService (outBoxR epository.delete(entity)), we can view the events inserted in the outbox table.

Image 7. Events in the OutBox Table.

Summary

In a microservice architecture, system failure is inevitable. Adapting this architecture style forces us to design for failures. The Outbox Pattern gives us a robust method of reliable messaging in the face of failure.

The above solution makes the implementation of the pattern simple. But to make the system highly available, we must run multiple instances (clusters) of Zookeeper, Apache Kafka, and Kafka Connect.

Finally, I would like to point out this isn’t the only way to tackle the problem of reliable messaging. But it is an invaluable pattern to have at your disposal.

Thank for reading!

#Microservices #c #cpringboot #kafka #database

What is GEEK

Buddha Community

Microservices: Implementing the Outbox Pattern
Einar  Hintz

Einar Hintz

1599055326

Testing Microservices Applications

The shift towards microservices and modular applications makes testing more important and more challenging at the same time. You have to make sure that the microservices running in containers perform well and as intended, but you can no longer rely on conventional testing strategies to get the job done.

This is where new testing approaches are needed. Testing your microservices applications require the right approach, a suitable set of tools, and immense attention to details. This article will guide you through the process of testing your microservices and talk about the challenges you will have to overcome along the way. Let’s get started, shall we?

A Brave New World

Traditionally, testing a monolith application meant configuring a test environment and setting up all of the application components in a way that matched the production environment. It took time to set up the testing environment, and there were a lot of complexities around the process.

Testing also requires the application to run in full. It is not possible to test monolith apps on a per-component basis, mainly because there is usually a base code that ties everything together, and the app is designed to run as a complete app to work properly.

Microservices running in containers offer one particular advantage: universal compatibility. You don’t have to match the testing environment with the deployment architecture exactly, and you can get away with testing individual components rather than the full app in some situations.

Of course, you will have to embrace the new cloud-native approach across the pipeline. Rather than creating critical dependencies between microservices, you need to treat each one as a semi-independent module.

The only monolith or centralized portion of the application is the database, but this too is an easy challenge to overcome. As long as you have a persistent database running on your test environment, you can perform tests at any time.

Keep in mind that there are additional things to focus on when testing microservices.

  • Microservices rely on network communications to talk to each other, so network reliability and requirements must be part of the testing.
  • Automation and infrastructure elements are now added as codes, and you have to make sure that they also run properly when microservices are pushed through the pipeline
  • While containerization is universal, you still have to pay attention to specific dependencies and create a testing strategy that allows for those dependencies to be included

Test containers are the method of choice for many developers. Unlike monolith apps, which lets you use stubs and mocks for testing, microservices need to be tested in test containers. Many CI/CD pipelines actually integrate production microservices as part of the testing process.

Contract Testing as an Approach

As mentioned before, there are many ways to test microservices effectively, but the one approach that developers now use reliably is contract testing. Loosely coupled microservices can be tested in an effective and efficient way using contract testing, mainly because this testing approach focuses on contracts; in other words, it focuses on how components or microservices communicate with each other.

Syntax and semantics construct how components communicate with each other. By defining syntax and semantics in a standardized way and testing microservices based on their ability to generate the right message formats and meet behavioral expectations, you can rest assured knowing that the microservices will behave as intended when deployed.

#testing #software testing #test automation #microservice architecture #microservice #test #software test automation #microservice best practices #microservice deployment #microservice components

Samanta  Moore

Samanta Moore

1623835440

Builder Design Pattern

What is Builder Design Pattern ? Why we should care about it ?

Starting from **Creational Design Pattern, **so wikipedia says “creational design pattern are design pattern that deals with object creation mechanism, trying to create objects in manner that is suitable to the situation”.

The basic form of object creations could result in design problems and result in complex design problems, so to overcome this problem Creational Design Pattern somehow allows you to create the object.

Builder is one of the** Creational Design Pattern**.

When to consider the Builder Design Pattern ?

Builder is useful when you need to do lot of things to build an Object. Let’s imagine DOM (Document Object Model), so if we need to create the DOM, We could have to do lot of things, appending plenty of nodes and attaching attributes to them. We could also imagine about the huge XML Object creation where we will have to do lot of work to create the Object. A Factory is used basically when we could create the entire object in one shot.

As **Joshua Bloch (**He led the Design of the many library Java Collections Framework and many more) – “Builder Pattern is good choice when designing the class whose constructor or static factories would have more than handful of parameters

#java #builder #builder pattern #creational design pattern #design pattern #factory pattern #java design pattern

Tia  Gottlieb

Tia Gottlieb

1597438200

What Is a Microservice Architecture? Why Is It Important Now?

We have been building software applications for many years using various tools, technologies, architectural patterns and best practices. It is evident that many software applications become large complex monolith over a period for various reasons. A monolith software application is like a large ball of spaghetti with criss-cross dependencies among its constituent modules. It becomes more complex to develop, deploy and maintain monoliths, constraining the agility and competitive advantages of development teams. Also, let us not undermine the challenge of clearing any sort of technical debt monoliths accumulate, as changing part of monolith code may have cascading impact of destabilizing a working software in production.

Over the years, architectural patterns such as Service Oriented Architecture (SOA) and Microservices have emerged as alternatives to Monoliths.

SOA was arguably the first architectural pattern aimed at solving the typical monolith issues by breaking down a large complex software application to sub-systems or “services”. All these services communicate over a common enterprise service bus (ESB). However, these sub-systems or services are actually mid-sized monoliths, as they share the same database. Also, more and more service-aware logic gets added to ESB and it becomes the single point of failure.

Microservice as an architectural pattern has gathered steam due to large scale adoption by companies like Amazon, Netflix, SoundCloud, Spotify etc. It breaks downs a large software application to a number of loosely coupled microservices. Each microservice is responsible for doing specific discrete tasks, can have its own database and can communicate with other microservices through Application Programming Interfaces (APIs) to solve a large complex business problem. Each microservice can be developed, deployed and maintained independently as long as it operates without breaching a well-defined set of APIs called contract to communicate with other microservices.

#microservice architecture #microservice #scaling #thought leadership #microservices build #microservice

Autumn  Blick

Autumn Blick

1595338835

Microservices and Data Management - DZone Microservices

Introduction

For pure frontend developers who doesn’t have much exposure to backend or middleware technology, microservices are a vague thing. They might have high-level introduction. So, let us have some deep understanding of what microservices are, and how it is different from monolithic application data management.

Monolithic and Microservice

In a monolithic application, all the stakeholders like all the business logic, routing features, middle-wares and Database access code get used to implement all the functionalities of the application. It is basically a single unit application. It has a lot of challenges in terms of scalability and agility. On the other side, in a microservice, all the business logic, routing features, middle-wares, and database access code get used to implement a single functionality of the application. We break down the functionalities to the core level and then connect to related services. So, the functionalities are actually dependent on related services only and does not get affected if there is an issue with other services. This helps to make the application agile, flexible, and highly scalable.

Monolithic architecture

Microservices Architecture

Why Microservices

Independent DB for the Services

The very first important thing associated with microservices is that each functionality requires its own database and never connects to the database of other services. In a monolithic service, since you have a single database. if something goes wrong with it then the whole application gets crashed. But in microservice, since we have an independent database for each service, in case of any problem with any particular database, it certainly does not affect other services and your application does not crash as a whole.

No Dependency on Schema

We have many services in our application and each service requires its own database. Hence, each database has its own schema or structure. But, if any service is connected to other service and shares the data and during development, the source database changes its schema and does not update the dependent services, then the service will not function correctly and may crash. So, there should be no dependency on databases.

Performance

Depending on the nature of service, we choose the appropriate type of DB. Some services are more efficient in specific database. So, creating a single database for all the services in the application might affect performance. In Microservice, since we have individual DB for each of the service, it is quite flexible, independent, and functions efficiently.

Data Management

Unlike the monolithic approach, in microservice, each functionality or service connects to its own database and never gets connected to other database. So, the big question arises of how we communicate between two services. It is quite generic in an application that we require to get some information based on the combination of many service outputs. But as a thumb rule, services dont communicate. Then what is the solution to this issue? Let us see, how data communicates between the services.

#data management #monolith vs microservice #microservices benefits #microservices communication #microservices archiecture

Autumn  Blick

Autumn Blick

1595335187

Microservices and Data Management - DZone Microservices

Introduction

For pure frontend developers who doesn’t have much exposure to backend or middleware technology, microservices are a vague thing. They might have high-level introduction. So, let us have some deep understanding of what microservices are, and how it is different from monolithic application data management.

Monolithic and Microservice

In a monolithic application, all the stakeholders like all the business logic, routing features, middle-wares and Database access code get used to implement all the functionalities of the application. It is basically a single unit application. It has a lot of challenges in terms of scalability and agility. On the other side, in a microservice, all the business logic, routing features, middle-wares, and database access code get used to implement a single functionality of the application. We break down the functionalities to the core level and then connect to related services. So, the functionalities are actually dependent on related services only and does not get affected if there is an issue with other services. This helps to make the application agile, flexible, and highly scalable.

Monolithic architecture

Microservices Architecture

Why Microservices

Independent DB for the Services

The very first important thing associated with microservices is that each functionality requires its own database and never connects to the database of other services. In a monolithic service, since you have a single database. if something goes wrong with it then the whole application gets crashed. But in microservice, since we have an independent database for each service, in case of any problem with any particular database, it certainly does not affect other services and your application does not crash as a whole.

No Dependency on Schema

We have many services in our application and each service requires its own database. Hence, each database has its own schema or structure. But, if any service is connected to other service and shares the data and during development, the source database changes its schema and does not update the dependent services, then the service will not function correctly and may crash. So, there should be no dependency on databases.

Performance

Depending on the nature of service, we choose the appropriate type of DB. Some services are more efficient in specific database. So, creating a single database for all the services in the application might affect performance. In Microservice, since we have individual DB for each of the service, it is quite flexible, independent, and functions efficiently.

Data Management

Unlike the monolithic approach, in microservice, each functionality or service connects to its own database and never gets connected to other database. So, the big question arises of how we communicate between two services. It is quite generic in an application that we require to get some information based on the combination of many service outputs. But as a thumb rule, services dont communicate. Then what is the solution to this issue? Let us see, how data communicates between the services.

#data management #monolith vs microservice #microservices benefits #microservices communication #microservices archiecture