As a back-end developer, you are often asked to build some structured orchestration of tasks such as microservices orchestrations, distributed transactions, data pipelines or some business logic. In all those examples you must ensure that the different types of tasks are processed according to a given scenario. This scenario can be simple such as sequential tasks, or a lot more complex with scenarios depending on time, with a complex decision diagram, or depending on external signals for example.

As soon as you are facing moderately complex workflows, you may be tempted to use a DSL, coupled with a dedicated engine to operate those workflows, such as Airflow, Luigi or Conductor.

In this article, I will show how a programming language can be used as a DSL by a “workflow as code” engine, and why it’s probably your best long-term option for building reliable automation at any scale.

The examples provided below will use Kotlin, as this is the language I use in Infinitic, the “workflow as code” engine I work on. Kotlin was chosen for a great integration with Apache Pulsar as described here.

Context

Even if you can benefit from a “workflow as code” engine in simple infrastructure, I am considering the more complex situation where you have distributed stateless workers in different locations. You are able to distribute tasks to those workers through some mechanisms (for example a queuing system) — each task being usually described by their name and some data.

The problem we try to solve is: how to orchestrate the processing of those tasks on different servers — in a way both reliable, scalable, and easy to manage and monitor?

Example

Let’s consider the sequential workflow below as an example :

• The workflow is starting with an url pointing to an image and a target size
• The first task is downloading the image
• The second task is resizing the image according to target size
• And the last task is uploading the resized image and returning an url pointing to it:

Using Infinitic, this workflow could be coded like this:

#pulsar #infinitic #kotlin #workflow #task-management