Task orchestration tools and workflows

Recently there’s been an explosion of new tools for orchestrating task- and data workflows (sometimes referred to as “MLOps”). The quantity of these tools can make it hard to choose which ones to use and to understand how they overlap, so we decided to compare some of the most popular ones head to head.

Airflow is the most popular solution, followed by Luigi. There are newer contenders too, and they’re all growing fast. (Source: Author)

Overall Apache Airflow is both the most popular tool and also the one with the broadest range of features, but Luigi is a similar tool that’s simpler to get started with. Argo is the one teams often turn to when they’re already using Kubernetes, and Kubeflow and MLFlow serve more niche requirements related to deploying machine learning models and tracking experiments.

Before we dive into a detailed comparison, it’s useful to understand some broader concepts related to task orchestration.

What is task orchestration and why is it useful?

Smaller teams usually start out by managing tasks manually — such as cleaning data, training machine learning models, tracking results, and deploying the models to a production server. As the size of the team and the solution grows, so does the number of repetitive steps. It also becomes more important that these tasks are executed reliably.

The complex ways these tasks depend on each other also increases. When you start out, you might have a pipeline of tasks that needs to be run once a week, or once a month. These tasks need to be run in a specific order. As you grow, this pipeline becomes a **network **with dynamic branches. In certain cases, some tasks set off other tasks, and these might depend on several other tasks running first.

#data-science #mlops #machine-learning #airflow #luigi #machine learning