Airflow is an open source platform used to orchestrate workflows. It is perfect for Extract, Transform, Load tasks, data migration and data integration, among other jobs.
In this tutorial we will see how we can leverage Twilio’s Programmable Messaging to set up an alerting system for Airflow jobs.
Airflow DAG (source: Apache Airflow)
Tutorial requirements
To follow this tutorial you need to have:
- Git. See installation instructions here.
- Docker. You can follow the installation steps provided here.
- A Twilio account. If you are new to Twilio create a free account now. You can review the features and limitations of a free Twilio account.
Set up Airflow to run locally
We are going to start off by cloning a repository that contains the docker image we shall use for this tutorial. The image is based on Python 3.7-slim-buster and will create Postgres and Redis containers. Postgres is used for the backend, while Redis is for the queue.
Installation & Setup
In this tutorial, I am going to use an existing docker-airflow image. If you prefer to install Airflow locally, please find detailed instructions here.
Enter the following commands in your terminal window to clone the docker-airflow repository:
$ git clone https://github.com/puckel/docker-airflow.git
$ cd docker-airflow
Run the following command to start an Airflow instance:
$ docker-compose -f docker-compose-LocalExecutor.yml up
If everything is set up correctly, you should see the Airflow web server UI on visiting the http://localhost:8080/ URL in your browser. Please note that the code base comes with a default tutorial entry that we are not going to use.
#code #tutorials and hacks #apache
