How this Github project can take your local Prefect workflows to the next level by provisioning AWS & Prefect Cloud infrastructure and the important role Github Actions  plays as part of it.

TL;DR: This project sets up a _dataflow management system _powered by  Prefect and  AWS. Its deployment has been fully automated through  Github Actions, which additionally  exposes a reusable interface to register workflows with  Prefect Cloud.

The problem

Prefect is an open-source tool that empowers teams to orchestrate workflows with Python. Its cloud solution -  Prefect Cloud - adds a management layer on top of the framework. This enables teams to manage workflows while operating under a  hybrid model that guarantees the execution layer remains on the user’s side.

By saying that, it is the user’s responsibility to spin up and integrate the execution layer with Prefect Cloud, meaning that some engineering effort is required to get the solution up and running from end-to-end.

The motivation: when the automation needs to be automated

A few months ago I worked on a personal project with Prefect and Python. When I finished the workflow development and went to register it with Prefect Cloud I noticed the registration step didn’t fit well with the workflow definition, becoming a manual and repetitive process for me. I decided then to design something generic and reusable for future pieces of work.

#github-actions #prefect #aws #terraform #github