If you’re a team developing and maintaining a software monolith, there’s a good chance you’re considering or planning a move to an architecture based around microservices. I’m won’t go into the various trade-offs involved in that decision in this article; rather, I will focus on one specific technique that might help you make the transition: Change Data Capture (CDC)

It’s relatively straightforward to build a system around microservices if you’re starting from scratch. However, it can be difficult to plan and manage a transition from an existing monolith. The kinds of changes involved can be substantial, and it’s hard to keep a live system running smoothly while you fundamentally change how it works.

It’s a big shift from an ACID-compliant database to a distributed architecture based on eventual consistency, and keeping data consistent during a long migration, when different information is held in different parts of your system can be particularly challenging.

Change Data Capture (CDC) enables you to make minimal changes (if any at all) to your production system at first. Rather, you set up a system to observe your database, and create events whenever key data is changed, with your “new architecture” systems responding to these events.

Change Data Capture

Let’s look at an example. Say you want to add an onboarding email flow to your application, so that new users receive helpful emails over the course of several days after they create an account. Using CDC, you can create this new flow as a microservice. Whenever a new user record is added to your main userstable, a new event is created. Then, your new microservice would consume that event and manage the onboarding process, without requiring any further changes to your main legacy application. Another example would be to send users an exit survey after they deleted their account, to capture data on why your service no longer meets their requirements.

I’m going to walk through one technique for achieving this, which requires literally no changes whatsoever to the “main” application; Heroku’s recently-launched Streaming Data Connectors Beta.

The way this works is that you add a managed Kafka and a “data connector” to your Heroku application, defining the tables and columns where changes should generate events. Then, you can set up your new microservices to consume events from Kafka topics.

In the rest of this article, I’m going to walk you through how to set this up. We’ll be using a trivial database-backed web application to represent our monolith, and a separate application subscribed to a Kafka topic, which will consume the events we generate by making changes to our database.

#microservices #kafka #heroku #software developent #event streaming #software architec...