Airflow is an open source platform used to orchestrate workflows. It is perfect for Extract, Transform, Load tasks, data migration and data integration, among other jobs.

In this tutorial we will see how we can leverage Twilio’s Programmable Messaging to set up an alerting system for Airflow jobs.

Dag - Graph view

Airflow DAG (source: Apache Airflow)

Tutorial requirements

To follow this tutorial you need to have:

Set up Airflow to run locally

We are going to start off by cloning a repository that contains the docker image we shall use for this tutorial. The image is based on Python 3.7-slim-buster and will create Postgres and Redis containers. Postgres is used for the backend, while Redis is for the queue.

Installation & Setup

In this tutorial, I am going to use an existing docker-airflow image. If you prefer to install Airflow locally, please find detailed instructions here.

Enter the following commands in your terminal window to clone the docker-airflow repository:

$ git clone
$ cd docker-airflow

Run the following command to start an Airflow instance:

$ docker-compose -f docker-compose-LocalExecutor.yml up

If everything is set up correctly, you should see the Airflow web server UI on visiting the http://localhost:8080/ URL in your browser. Please note that the code base comes with a default tutorial entry that we are not going to use.

Airflow UI

#code #tutorials and hacks #apache

Monitor your Apache Airflow Workflows with Twilio
3.05 GEEK