What is continuous integration and continuous development? How is it used in terms of Azure Data Factory? I will walk you through the process of implementing continuous integration and continuous delivery for pipelines created in the Azure data factory.
What is continuous integration and continuous development? How is it used in terms of Azure Data Factory?
Continuous Integration (CI) is a practice that allows developers to seamlessly merge code in a common repository whereas Continuous Delivery (CD) is a practice that adds a layer to CI by providing multi-stage infrastructure provisioning and deployment which helps in automating the entire software release process.
For Azure data pipelines, CI/CD means nothing but moving data factory pipelines from one environment to another.
In this blog, I will walk you through the process of implementing continuous integration and continuous delivery for pipelines created in the Azure data factory.
To begin with, let us first take a look at the high-level architecture:
From this, we can concur that the Azure Data factory is used to perform ETL/ELT operations on data. To implement CI/CD in ADF, the Azure DevOps repository needs to be configured first. As shown in the above flow, once the pipeline is developed and published, changes can be pushed in the master branch through Pull Request following which a Release pipeline is created to deploy these changes on Test and Production environments. This is done by configuring environment-specific connections, which is elaborated later.
Before we move any further, let me tell you that an Azure DevOps account is a mandatory stepping stone if you want to successfully implement CI/CD in ADF.
Now, since implementing CI/CD is the main target of this blog, we will create and use a simple ADF pipeline.
Here are the steps for it:
Azure DevOps CI/CD Pipeline will give you a brief introduction on how you can implement DevOps practices on Microsoft Azure.
Learn how to building an Azure DevOps-based ARM CI/CD for Azure Cloud. This blog series focuses on presenting complex DevOps projects as simple and approachable via plain language and lots of pictures.
This tutorial will drive you through setting up a great CI/CD pipeline for Xamarin Android in a fully hosted Azure DevOps enviroment.
This Edureka live video on "Azure DevOps CI/CD Pipeline" will give you a brief introduction on how you can implement DevOps practices on Microsoft Azure.
CI/CD Pipeline with Azure DevOps for Data Science project.: A CI/CD Pipeline implementation, or Continuous Integration/Continuous Deployment for Data science.