Code Is the Best DSL for Building Workflows

Code Is the Best DSL for Building Workflows

In this article, I will show how a programming language can be used as a DSL by a “workflow as code” engine, and why it’s probably your best long-term option for building reliable automation at any scale.

As a back-end developer, you are often asked to build some structured orchestration of tasks such as microservices orchestrations, distributed transactions, data pipelines or some business logic. In all those examples you must ensure that the different types of tasks are processed according to a given scenario. This scenario can be simple such as sequential tasks, or a lot more complex with scenarios depending on time, with a complex decision diagram, or depending on external signals for example.

As soon as you are facing moderately complex workflows, you may be tempted to use a DSL, coupled with a dedicated engine to operate those workflows, such as AirflowLuigi or Conductor.

In this article, I will show how a programming language can be used as a DSL by a “workflow as code” engine, and why it’s probably your best long-term option for building reliable automation at any scale.

The examples provided below will use Kotlin, as this is the language I use in [Infinitic_](https://github.com/infiniticio/infinitic), the “workflow as code” engine I work on. Kotlin was chosen for a great integration with [Apache Pulsar_](https://pulsar.apache.org/)_ as described [here_](https://medium.com/@gillesbarbier/building-an-event-driven-orchestration-engine-bf62d45aef5d?source=friends_link&sk=22528e7580f94f5835e760ad3556c16f).

Context

Even if you can benefit from a “workflow as code” engine in simple infrastructure, I am considering the more complex situation where you have distributed stateless workers in different locations. You are able to distribute tasks to those workers through some mechanisms (for example a queuing system) — each task being usually described by their name and some data.

The problem we try to solve is: how to orchestrate the processing of those tasks on different servers — in a way both reliable, scalable, and easy to manage and monitor?

Example

Let’s consider the sequential workflow below as an example :

  • The workflow is starting with an url pointing to an image and a target size
  • The first task is downloading the image
  • The second task is resizing the image according to target size
  • And the last task is uploading the resized image and returning an url pointing to it:

Image for post

Using Infinitic, this workflow could be coded like this:

Image for post

pulsar infinitic kotlin workflow task-management

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

Android Coroutines: How to Manage Async Tasks in Kotlin

How to Manage Async Tasks in Kotlin. Take a good look at the current state of coroutines support in Android libraries and tools, explore some interesting and not-so-obvious edge cases when dealing with Kotlin code that includes coroutines and interop with Java, and briefly look at some new developments in Kotlin coroutines, namely StateFlow and its comparison with LiveData, and SharedFlow.

Android 11 Latam: Android Coroutines: How to manage async tasks Kotlin

15:00 hrs. GMT-4 Charla: Android Coroutines: How to manage async tasks in Kotlin Speaker: Florina Muntenescu, Android Developer Advocate, Google

Setting up Your Workflow for Competitive Programming in Kotlin

This is a follow-up to my previous article, where I convince you to try Kotlin for Competitive Programming. But if you're already convinced, then

How to use Kotlin - from the Lead Kotlin Language Designer

In this Kotlin tutorial, the lead Kotlin language designer will show you how you can write more idiomatic Kotlin, what the benefits are, and help you discover some of the most powerful yet lesser known features of Kotlin. Kotlin is similar to the Java programming language, so it's natural that your Kotlin code looks very much like Java code when you are first start to use the language.

Kotlin Programming Fundamentals Tutorial - Learn Kotlin for Beginners

Kotlin Programming Fundamentals Tutorial - Learn Kotlin for Beginners: Learn programming fundamentals using the Kotlin programming language. Kotlin is an excellent language for GUI Architectures, Libraries, and Server Side Applications. This course will start you off the right way, no matter which path you take with the language. The course features hands-on coding exercises to teach you both Functional, Event Driven, and Object Oriented design patterns.