Airflow — Trigger KubernetesPodOperator using HTTP call

Airflow — Trigger KubernetesPodOperator using HTTP call

Airflow has an Operator called “Kubernetes Pod Operator” which would ideally connect to a cluster, create a pod run the desired program and delete the pod. This is a much-required feature where we will be able to run our post-load/Intermediate jobs or process in the Kubernetes cluster which can scale up and wide.

Airflow has an Operator called “Kubernetes Pod Operator” which would ideally connect to a cluster, create a pod run the desired program and delete the pod. This is a much-required feature where we will be able to run our post-load/Intermediate jobs or process in the Kubernetes cluster which can scale up and wide. This saves a lot in terms of cost as the pods will be up only when we are running the job. This also make our system flexible as we can run any number of jobs concurrently without having to worry about the computing power that we have as Kubernetes auto handles that for us.

Although the Kubernetes pod is perfect to use it has a caveat. It does not have an option to get the data from a DAG http trigger made to airflow. That’s when we will introduce a simple customization to the existing KubernetesPodOperator package to make use of the data coming in from the Http request.

Below is a sample DAG where it uses KubernetesPodOperator to make run a particular program inside the Kubernetes Cluster.

import airflow
from airflow import DAG
from kubernetes_pod_operator import KubernetesPodOperator

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': airflow.utils.dates.days_ago(0),
}

dag = DAG('sampleDAG', default_args=default_args, schedule_interval=None)

DriftTask = KubernetesPodOperator(
    namespace='default',
    image="python",
    cmds=["python", "-c"],
    arguments=["print('This code is running in a Kubernetes Pod')"],
    labels={},
    name="sampleDAG",
    task_id="sampleDAG",
    get_logs=True,
    dag=dag,
    log_events_on_failure=True,
    is_delete_operator_pod=True)

DriftTask

This Dag would use the python image from hub.docker.com to create a pod in the Kubernetes Cluster and run the below command in it.

print('This code is running in a Kubernetes Pod')
  • What if you want to pass dynamic data that needs to be printed from this program?
  • What if the program that you try to trigger needs the dynamic input coming from an HTTP trigger?

To answer these questions we will have to write a custom wrapper around the existing KubernetesPodOperator class. Below is its implementation.

kubernetes kubernetes-operator kubernetes-pods

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

50+ Useful Kubernetes Tools for 2020 - Part 2

Our original Kubernetes tool list was so popular that we've curated another great list of tools to help you improve your functionality with the platform.

What Is a Kubernetes Operator and Why it Matters for SRE

In this blog post, we’ll explain the Kubernetes Operator and discuss how it can evolve your SRE solution. Kubernetes is an open-source project that “containerizes” workloads and services and manages deployment and configurations. Released by Google in 2015, Kubernetes is now maintained by the Cloud Native Computing Foundation.

Writing a Kubernetes Operator — Locust Operator

In my previous article, I discussed how two frameworks can generate the basic structure for a Kubernetes Operator. I would discuss how to build on top of the base generated by the framework, so I would go through how to build an operator with simple functionality. If you are don’t know about Kubernetes operators check out the session where I talk about operators in detail.

Kubernetes in the Cloud: Strategies for Effective Multi Cloud Implementations

This article explains how you can leverage Kubernetes to reduce multi cloud complexities and improve stability, scalability, and velocity.

Kubernetes vs Docker

Get Hands-on experience on Kubernetes and the best comparison of Kubernetes over the DevOps at your place at Kubernetes training