Introduction

Airflow is one of my favorite tools that I frequently use to setup and manage data science pipelines. The Airflow UI gives us a clear picture of the DAGS and its current status. I may be wrong here but from my experience, I have seen that Airflow on a single machine is not scalable. Thus, to scale Airflow, we can use Kubernetes.

When I was trying to deploy Airflow on AWS EKS, I had to go through multiple sources, get things clarified from the community. Thus, I am writing this article to make things as easy as possible to deploy airflow on AWS EKS.

The prerequisites for the article are to have aws-cli, kubectl, and helm installed, setup an EKS cluster in AWS. We will be using helm to deploy the airflow helm chart on AWS EKS.

I found Helm to be very useful in setting up and managing the Kubernetes application. Many helm charts are available in Artifacthub.io.

Airflow also has a helm chart that has very good community support. I have to thank the Airflow helm chart community that helped me out a lot! The helm chart is available at https://github.com/airflow-helm/charts/tree/main/charts/airflow.

#aws #aws-eks #airflow #kubernetes