Data Ops – Git Actions & Terraform for Data Engineers & Scientists — GCP/AWS/Azure. With this post, Data Engineers and Scientists can CICD Infrastructure with ease. Github Actions workflow to run TF steps. Integrating Terraform cloud to Github.
I strongly believe in POCing a new design or code template and get it review by other engineers because you never know what efficient nuances you are missing and it is always good to have a fresh set of eyes reviewing the code. And I tend to make POC as near to MVP (Minimum viable product) as possible to make myself and the team more confident of the design and to not waste any time in Regression testing later. It also helps in estimating my delivery task better and more accurately. But issues arise when I have to be dependent upon the DevOps team to ‘Infrastructure as code’ (IAC) my project in the dev environment. In the Prod environment, it is desirable to involve them in DevOps the infrastructure based on the best practices they have learned but in Dev, it can derail your MVP by just waiting for them to prioritize your task.
So a couple of years ago I started learning DevOps/DataOps and I started with Cloudformation (CFN) and Atlassian Bamboo since I was mostly working on AWS and the organization was using Bamboo. But lately, I got the chance to get my hands dirty in Terraform (TF) and Github Actions, because I was required to work on GCP, and dear oh dear it is way too easy to grasp and good to learn because with TF and Actions you can deploy in any cloud. And for a Data Engineer or Scientist or Analyst, it becomes really handy if you know an IAC tool. Since Github Actions sit closer to your code, it becomes all the more convenient for me.
So, I will break this down into 3 easy sections:
GitHub launched GitHub Actions in beta which enabled developers to create automated workflows to build, test, and deploy their source code on GitHub. In this article, we will discuss about GitHub Actions and how it can be used to build an automated software development life-cycle workflow.
Even if you’re the only developer in a project, you still have to make sure you automate your deployment process and follow certain steps in order to prevent bugs sneak into production code. Writing tests is the first step of it. But then, running those tests everytime you make a change in the code is a time consuming process where you can benefit from some automation. This is where Github Actions come into play: How about creating a workflow in Github Actions which will run the tests for you when you make a pull request and won’t allow merging to master and deploying to production if the tests fail? That would really help with avoiding unintentional deployment of buggy code. Another important point is not forgetting to deploy your code into production when a pull request is merged to master. The Github workflow that we are going to create will also handle that by automatically deploying to Firebase Hosting when a pull request is completed. CI/CD with Vue, Firebase Hosting and Github Actions
To set up a serverless CI/CD pipeline in your AWS environments, there are several key services that you need to use. Find out more here.
In this post we’ll learn how to add a simple GitHub actions CI/CD flow to a basic Altostra API service project.
Terraform for GitHub Repos & Quality Tests with Github Actions. A walkthrough on setting up GitHub repositories with Terraform and a workflow to validate your Terraform files with GitHub Actions. Then we are going to use Github Actions and specifically the Hashicorp setup-terraform GitHub.