In this article, I would like to give you a jump-start tutorial to understand the basic concepts and create a workflow pipeline from scratch.

So let’s get started!

Introduction

Apache Airflow is an orchestration tool that helps you to programmatically create and handle task execution into a single workflow. It then handles monitoring its progress and takes care of scheduling future workflows depending on the schedule defined.

Workflows are created using Python scripts, which define how your tasks are executed. They are usually defined as Directed Acyclic Graphs (DAG).

The workflow execution is based on the schedule you provide, which is as per Unix cron schedule format. Once you create Python scripts and place them in the dags folder of Airflow, Airflow will automatically create the workflow for you.

How difficult are the Python scripts?

Well, not difficult but pretty straightforward. Let me explain this.

#python #airflow