Airflow + Slack
Slack is an increasingly popular chat app used in the workplace. Apache Airflow is an open source platform for orchestrating workflows. One of the biggest advantages to using Airflow is the versatility around its** hooks** and operators. Hooks are interfaces to external platforms, databases and also serve as the basic building blocks of Operators.
The Slack Webhook Operator can be used to integrate Airflow with Slack. This operator is typically used for reporting and alerting purposes by scheduling incoming messages to Slack channels when some trigger condition is met.
I’m going to show you how to leverage these tools to perform some very simple reporting in your Slack workspace: send daily weather forecasts to a channel.
These foundations can be expanded to create more complex Airflow + Slack integrations. Let’s get started!
Setting up a Slack Workspace
**Note: if you are already familiar with setting up apps on Slack and with the Slack webhook, skip to **“Airflow + Docker”
A workspace is a shared hub of channels where teammates and collaborators can communicate together. I created a workspace called weather-enthusiasts.
From our workspace, we need:
- a channel to accept these messages
- webhook url
Let’s have Airflow post to a public channel called #daily-weather-feed. If the message contents are sensitive, consider changing this to a private channel.
Next, we want to create an Airflow app for the workspace. Go to https://api.slack.com/apps and click Create New App
This should lead you to a modal where you can set the App name. Airflow will post messages under the name you select. For simplicity, I named my app airflow.
Exiting the modal leads to a page where you can add “Incoming Webhooks” as an app feature.
Make sure that incoming webhooks are turned on.
Scroll to the bottom of the page and click on “Add New Webhook to Workspace”.
This generates a WebHook URL, which can be used as authentication for Airflow.
The WebHook URL also allows you to programmatically send messages to Slack. Here’s a very simple POST request you can try in the terminal. Don’t forget to replace my redacted URL with yours.
curl -X POST -H 'Content-type: application/json' --data '{"text":"Hi, this is an automated message!"}' https://hooks.slack.com/services/XXXX
Check your channel to see the automated message.
#data-engineering #data #data-science #technology #programming #data analysis