Within the next 10 minutes, you will learn something that will enrich your data journey altogether. With this post, Data Engineers and Scientists can CICD Infrastructure with ease.

I strongly believe in POCing a new design or code template and get it review by other engineers because you never know what efficient nuances you are missing and it is always good to have a fresh set of eyes reviewing the code. And I tend to make POC as near to MVP (Minimum viable product) as possible to make myself and the team more confident of the design and to not waste any time in Regression testing later. It also helps in estimating my delivery task better and more accurately. But issues arise when I have to be dependent upon the DevOps team to ‘Infrastructure as code’ (IAC) my project in the dev environment. In the Prod environment, it is desirable to involve them in DevOps the infrastructure based on the best practices they have learned but in Dev, it can derail your MVP by just waiting for them to prioritize your task.

So a couple of years ago I started learning DevOps/DataOps and I started with Cloudformation (CFN) and Atlassian Bamboo since I was mostly working on AWS and the organization was using Bamboo. But lately, I got the chance to get my hands dirty in Terraform (TF) and Github Actions, because I was required to work on GCP, and dear oh dear it is way too easy to grasp and good to learn because with TF and Actions you can deploy in any cloud. And for a Data Engineer or Scientist or Analyst, it becomes really handy if you know an IAC tool. Since Github Actions sit closer to your code, it becomes all the more convenient for me.

So, I will break this down into 3 easy sections:

1. Integrating TF cloud to Github
2. Github Actions workflow to run TF steps
3. Overview of TF files based on Best Practices

#gcp #terraform #github-actions #dataops #ci-cd-pipeline