How to Build an Azure Pipeline (Build/Release) from Scratch

Software must be written, tested and deployed into a production environment. To do so, in an automated fashion, developers and DevOps build CI/CD pipelines or build/release pipelines that automate much of this activity.

A pipeline allows developers, DevOps teams and others to produce and deploy reliable code. In the Microsoft realm, the way to build a pipeline is with Azure DevOps with a feature called Azure Pipelines.

Get practical knowledge of Azure DevOps from real time experts at DevOps Online Training

reating an Azure DevOps Organization

Because we’re starting from absolute scratch, you’ll first create the “base” object that all Azure DevOps features and services are contained within called the organization. The organization is where all projects are stored which you’ll learn how to build in the next section.

From the Azure Portal, browse to All services, search for DevOps and select Azure DevOps. You can see what to expect below. This will open the Azure DevOps blade as you can see below.
Azure DevOps dashboard
2. Next, click My Azure DevOps Organizations and provide your Azure credentials. You will be brought to a page where you can create a new organization as shown below. Once here, click on the blue Create new organization button.
Create new Azure DevOps organization page
3. On the next page, provide a name for the organization. If you run into problems with using a specific name, be sure to check the Azure DevOps organization naming conventions. On this screen, also specify an Azure region closest to you. Once done, click Continue.
Azure DevOps organization creation page
At this point, your Azure DevOps organization is created!

Creating an Azure DevOps Project

The next step is creating a project. A project is a container for the pipeline you’ll be created, various artifacts and any other information pertaining to a particular service or piece of software.

Azure DevOps gives you the ability to create a project during the same workflow as creating an organization as performed above. If you followed along, you should now be presented with a project creation page as shown below.

On the project creation page, provide a name for your project in the Project name field. For this project, you’ll be using an Azure DevOps project called devopsdemo.
Confirm the creation of the project by clicking the + Create Project button.
For now, leave the Visibility set to Private. This ensures your project isn’t exposed to the Internet. A public project allows non-members of a project and non-signed-in users read-only access to the project’s artifacts.

Get hands on experience on building the DevOps project on Azure at DevOps Online Training hyderabad

**Azure DevOps project creation page

At this point, your Azure DevOps workspace (project + organization) has been created. If you’ve been following along, you will now have an Azure DevOps organization created called pdtitws123 and a project inside of that organization called devopsdemo.

Azure DevOps project overview
**Building an Azure DevOps Build Pipeline

Now that you have an Azure DevOps organization and project set up, you can now create a build pipeline inside. The pipeline is where all the magic happens. It’s where you will create builds to perform various tasks like compiling code, bringing in dependencies and more.

1.From the dashboard, select Pipelines then on Builds.
Azure DevOps Builds menu
2. You will see a message telling you there are no build pipelines found yet. Click on New pipeline to begin creating the build pipeline.

  1. You will then be prompted for where your code is stored. In this Project, the code is stored in a GitHub repository. Select GitHub. This is the place where code is stored and the triggers for invoking the build will come from.

Selecting version control source
4. Once you click on GitHub, you’ll be prompted to provide your GitHub account credentials as shown below. Before you begin this step, remember to have an empty GitHub repo created as defined in the Prerequisites section of this article!

  1. Next, confirm the step where you are asked to authorize Azure Pipelines. This ensures Azure DevOps has permission to access your GitHub repos.

Authorizing Azure Pipelines to access GitHub
Linking a GitHub Repo to the Build Pipeline

Once you’ve provided Azure DevOps permission to your GitHub account, now link a GitHub repo to the build pipeline.

Select the empty GitHub repo you have created for this Project.
1.Empty source GitHub repo
2. Allow the project to read, write and check source code from the repository you selected earlier. Then confirm this process by clicking Approve & Install. Confirm Azure DevOps Github integration
3. By clicking confirm, you will see an error generated as shown below. This error will happen because the GitHub Repo selected is empty.
Error when GitHub repo is empty
At this point, your GitHub repo will be linked to your Azure DevOps build pipeline.

You don’t have to create an entirely new pipeline every time you want to link a GitHub repo to an Azure DevOps build pipeline.

Populating Sample GitHub Repository Code

The Azure DevOps build pipeline won’t run without some code in the GitHub repo. The code doesn’t necessarily matter at this point. To get some code in the repo, we’ll use an existing repo to clone some code from. In this repo, I have all the source code available to publish an e-commerce sample application in Azure WebApps.

The sample code for this repo will contain a sample e-commerce application called SimplCommerce. This application is an open-source dotnetcore application that’s more realistic than the typical “hello world” project.

Navigate to this sample GitHub repo and click on Import code as shown below.
2. For the old repository’s clone URL use and click on Begin import.

When the repo import step is completed successfully, refresh the Azure DevOps Pipeline window, which should allow you to continue now.

Inspecting and Viewing the Build Pipeline in YAML
At this point, you will be at the Review phase of the pipeline creation process. You will now be presented with a representation of the build pipeline in YAML. This YAML file is automatically built based on the detection of the source code language which is dotnetcore in this Project.

One of the real benefits of Azure DevOps is the pipeline YAML. With many other DevOps tools, you have to build a pipeline file manually.
Manually Running the Azure Build Pipeline
In a continuous integration (CI) pipeline, the build is typically triggered by a commit to source control. But, you can also manually trigger the build pipeline to run. Let’s kick off a build pipeline manually to see what happens.

If you’ve been following along, at this point, you should be to the point where you can click on Run to kick off the build pipeline. This will start the pipeline build process as you can see from the below screenshot.

After a few seconds, you can see that the process will be running for macOS as shown below. The build pipeline will return information in real-time as each job and task in the pipeline runs.

The build pipeline will then repeat the same process for the other operating systems (in this example) as shown below. The steps taken will vary greatly depending on how the YAML file was built.

Once the build has completed, you are greeted with green checkmarks as you can see below. This screenshot indicates each platform build passed successfully.

You’ve now created an Azure DevOps build pipeline!

Building an Azure DevOps Release Pipeline
The build pipeline is created which is some great progress. If you were to stop at this point, you’d be well on your way for a complete, automated pipeline. But we’re not done! The ultimate goal of software needs to be deployed so customers can use it. It’s time to automate a release too with a release pipeline!

A release pipeline takes a build artifact, a result of the build process and deploys that to one or more environments. In this Project, you’re going to use a release pipeline to publish code in the GitHub repo to an Azure Web App.

From Azure DevOps, click Pipelines and then Releases.
2. Next, select New and then New Release Pipeline. This launches the New release pipeline wizard.

  1. From the template list on the right, select Azure App Service Deployment. You’ll see many different kinds of templates available to save time creating future release pipelines.

  2. Provide a description for the Stage Name. The stage will contain release tasks. For this project, use the name Deploy_to_webapp.

The release pipeline should now look like the below screenshot.

  1. In the Stages field, select 1 job, 1 task. This field is where you will eventually provide the settings of the Azure Web App environment you will use for the actual deployment.

  2. Part of the Azure App Service deployment template comes a few parameters you’ll need to define in this screen.

Stage Name - Deploy-to-webapps2 in this example.
Azure Subscription - select your subscription and confirm with Authorize (needs credentials)
App type - Web App on Linux
App Service Name - the web app you created earlier
7. When done, click Pipeline in the top menu of your Azure Pipeline project as shown below. This will return you to the main screen and allow you to complete the next step which is specifying the artifacts.

Adding Artifacts to the Azure DevOps Release Pipeline

Within a release pipeline, there are many different items that need to be deployed. These items are called artifacts. Artifacts put simply, are deployable components of your application. Azure Pipelines can deploy artifacts that are produced by a wide range of artifact sources.

In this section, let’s cover how to add artifacts to the release pipeline.
While on the Pipeline screen, click on Add an Artifact.
The Source Type is already set to Build which is what you want since you’ll be deploying the output of the build pipeline created earlier. Choose the build pipeline created earlier in the Source (build pipeline) dropdown.
When finished, click on Add to save the configuration.
4. Finally, click on the Save button in the upper right corner of the screen to save the release pipeline.

Creating the Azure DevOps Release
Once the release pipeline is created, you’ll then begin creating releases. A release is a particular run of the pipeline. Think of the release pipeline as the template and specific instances of that release pipeline as releases. Once you have a release, that release can then be deployed.

To create a release:

Click the Create Release button in the upper right corner of the window as shown below.
2. You’re not going to do anything fancy so at the Create a new release screen, accept the defaults and click on Create. You only have a single stage at this time and have a single version of the build artifact to deploy now.

  1. After a few seconds, you will receive a notification (the green bar) that the release has been created as shown below. Now click on Release-1 which will redirect you to the details of the release process.

You have now created a release and it’s ready to be deployed!

Manually Deploying a Release
A release is a set of instructions to perform on how to deploy a build. You’ve done all of that. Now it’s time to actually run those instructions and deploy code to an environment.

**While on the Release-1 release you created earlier:

Select the Deploy_to_webapp stage and confirm Deploy. When you do so, the status will change to In progress as shown below. At this point, the release is grabbing the source code from the build pipeline executed earlier and pushing it to the Azure WebApp instance.

This step will initiate a job running in the back-end on an agent copying source code and performing the actual release cycle.
2. Click on the stage while it’s in progress. You will see any logging information shown in the console output area.

When complete, the stage should show Succeeded as seen below.

If all went well, the release should have finished and you should now have a web app published to an Azure Web App!

**Inspecting the Deployed Azure WebApp

Now that the entire process is complete, be sure to check out the fruits of your labor.

Log into the Azure Portal, navigate to the Azure WebApp you selected in the release pipeline as target and copy the URL as shown below.
2. Now paste that URL into a browser by providing the URL of the Azure Web App. Your browser should now load the e-commerce sample application, similar to below home page

You’ve now done it all! You now have an Azure web app deployed from a GitHub repo able to be automated to the fullest!

Cleaning Up
If you followed along throughout this Project, you now have a new Azure DevOps organization, project, build pipeline, release pipeline and release. If you’re done testing things out, be sure to remove the original organization created so you don’t risk being charged for any Azure resources.

To do so:

Go back to your main Azure DevOps workspace.
Click on Organization settings in the lower left corner.
On the Overview screen, click on the Delete button at the bottom of the page under Delete organization.
Type the organization name and then click on Delete. This should remove all of the work you just performed.
Your Takeaways
If you followed the instructions in this Project, you’ve created an entire CI/CD pipeline from scratch in Azure DevOps. You should now have a good idea of what this process entails.
Azure Pipelines can go much deeper than what you did in this Project but you should now have some foundational knowledge of the entire process.
know more information from real time experts at DevOps Online Training India

What is GEEK

Buddha Community

How to Build an Azure Pipeline (Build/Release) from Scratch
Franz  Becker

Franz Becker


How to Create Custom Azure Pipelines Build Agent Fully Automated

Currently, most software teams that are using Azure DevOps to build their software choose to use Azure-hosted agents as their build server. This comes with a lot of benefits. Many libraries are already installed on these agents so it’s quite compatible with most projects’ specifications. For every build, you get a new agent that is provisioned and after the build will be destroyed. However, this has its own limitations. Azure Hosted Agents are using a specific type of VM so it is not possible to increase hardware specifications. If your projects also have libraries that are not supported by Azure Hosted agents, you need to install them on the agents. This might not be an issue for projects that have started in recent years, but if you have a legacy code that takes an hour to build then you might start to think about how you can speed up the build process. For such a situation, there are a lot of solutions like reconsidering the architecture of software and so on which is not the scope of this article. In this article, I am going to focus on how you can create a custom self-hosted agent.

#azure #azure devops #cicd pipeline #build pipeline

Eric  Bukenya

Eric Bukenya


Learn NoSQL in Azure: Diving Deeper into Azure Cosmos DB

This article is a part of the series – Learn NoSQL in Azure where we explore Azure Cosmos DB as a part of the non-relational database system used widely for a variety of applications. Azure Cosmos DB is a part of Microsoft’s serverless databases on Azure which is highly scalable and distributed across all locations that run on Azure. It is offered as a platform as a service (PAAS) from Azure and you can develop databases that have a very high throughput and very low latency. Using Azure Cosmos DB, customers can replicate their data across multiple locations across the globe and also across multiple locations within the same region. This makes Cosmos DB a highly available database service with almost 99.999% availability for reads and writes for multi-region modes and almost 99.99% availability for single-region modes.

In this article, we will focus more on how Azure Cosmos DB works behind the scenes and how can you get started with it using the Azure Portal. We will also explore how Cosmos DB is priced and understand the pricing model in detail.

How Azure Cosmos DB works

As already mentioned, Azure Cosmos DB is a multi-modal NoSQL database service that is geographically distributed across multiple Azure locations. This helps customers to deploy the databases across multiple locations around the globe. This is beneficial as it helps to reduce the read latency when the users use the application.

As you can see in the figure above, Azure Cosmos DB is distributed across the globe. Let’s suppose you have a web application that is hosted in India. In that case, the NoSQL database in India will be considered as the master database for writes and all the other databases can be considered as a read replicas. Whenever new data is generated, it is written to the database in India first and then it is synchronized with the other databases.

Consistency Levels

While maintaining data over multiple regions, the most common challenge is the latency as when the data is made available to the other databases. For example, when data is written to the database in India, users from India will be able to see that data sooner than users from the US. This is due to the latency in synchronization between the two regions. In order to overcome this, there are a few modes that customers can choose from and define how often or how soon they want their data to be made available in the other regions. Azure Cosmos DB offers five levels of consistency which are as follows:

  • Strong
  • Bounded staleness
  • Session
  • Consistent prefix
  • Eventual

In most common NoSQL databases, there are only two levels – Strong and EventualStrong being the most consistent level while Eventual is the least. However, as we move from Strong to Eventual, consistency decreases but availability and throughput increase. This is a trade-off that customers need to decide based on the criticality of their applications. If you want to read in more detail about the consistency levels, the official guide from Microsoft is the easiest to understand. You can refer to it here.

Azure Cosmos DB Pricing Model

Now that we have some idea about working with the NoSQL database – Azure Cosmos DB on Azure, let us try to understand how the database is priced. In order to work with any cloud-based services, it is essential that you have a sound knowledge of how the services are charged, otherwise, you might end up paying something much higher than your expectations.

If you browse to the pricing page of Azure Cosmos DB, you can see that there are two modes in which the database services are billed.

  • Database Operations – Whenever you execute or run queries against your NoSQL database, there are some resources being used. Azure terms these usages in terms of Request Units or RU. The amount of RU consumed per second is aggregated and billed
  • Consumed Storage – As you start storing data in your database, it will take up some space in order to store that data. This storage is billed per the standard SSD-based storage across any Azure locations globally

Let’s learn about this in more detail.

#azure #azure cosmos db #nosql #azure #nosql in azure #azure cosmos db

Noah  Rowe

Noah Rowe


Azure DevOps Pipelines: Multi-Stage Pipelines

The last couple of posts have been dealing with Release managed from the Releases area under Azure Pipelines. This week we are going to take what we were doing in that separate area of Azure DevOps and instead make it part of the YAML that currently builds our application. If you need some background on how the project got to this point check out the following posts.

Getting Started with Azure DevOps

Pipeline Creation in Azure DevOps

Azure DevOps Publish Artifacts for ASP.NET Core

Azure DevOps Pipelines: Multiple Jobs in YAML

Azure DevOps Pipelines: Reusable YAML

Azure DevOps Pipelines: Use YAML Across Repos

Azure DevOps Pipelines: Conditionals in YAML

Azure DevOps Pipelines: Naming and Tagging

Azure DevOps Pipelines: Manual Tagging

Azure DevOps Pipelines: Depends On with Conditionals in YAML

Azure DevOps Pipelines: PowerShell Task

Azure DevOps Releases: Auto Create New Release After Pipeline Build

Azure DevOps Releases: Auto Create Release with Pull Requests

Image for post


The current setup we have uses a YAML based Azure Pipeline to build a couple of ASP.NET Core web applications. Then on the Release side, we have basically a dummy release that doesn’t actually do anything but served as a demo of how to configure a continuous deployment type release. The following is the current YAML for our Pipeline for reference.

name: $(SourceBranchName)_$(date:yyyyMMdd)$(rev:.r)

  - repository: Shared
    name: Playground/Shared
    type: git 
    ref: master #branch name

trigger: none

  buildConfiguration: 'Release'

- job: WebApp1
  displayName: 'Build WebApp1'
    vmImage: 'ubuntu-latest'

  - task: PowerShell@2
      targetType: 'inline'
      script: 'Get-ChildItem -Path Env:\'

  - template: buildCoreWebProject.yml@Shared
      buildConFiguration: $(buildConfiguration)
      project: WebApp1.csproj
      artifactName: WebApp1

- job: WebApp2
  displayName: 'Build WebApp2'
  condition: and(succeeded(), eq(variables['BuildWebApp2'], 'true'))
    vmImage: 'ubuntu-latest'

  - template: build.yml
      buildConFiguration: $(buildConfiguration)
      project: WebApp2.csproj
      artifactName: WebApp2

- job: DependentJob
  displayName: 'Build Dependent Job'
    vmImage: 'ubuntu-latest'

  - WebApp1
  - WebApp2

  - template: buildCoreWebProject.yml@Shared
      buildConFiguration: $(buildConfiguration)
      project: WebApp1.csproj
      artifactName: WebApp1Again

- job: TagSources
  displayName: 'Tag Sources'
    vmImage: 'ubuntu-latest'

  - WebApp1
  - WebApp2
  - DependentJob
  condition: |
      eq(dependencies.WebApp1.result, 'Succeeded'),
      in(dependencies.WebApp2.result, 'Succeeded', 'Skipped'),
      in(dependencies.DependentJob.result, 'Succeeded', 'Skipped')

  - checkout: self
    persistCredentials: true
    clean: true
    fetchDepth: 1

  - task: PowerShell@2
      targetType: 'inline'
      script: |
        $env:GIT_REDIRECT_STDERR` = '2>&1'
        $tag = "manual_$(Build.BuildNumber)".replace(' ', '_')
        git tag $tag
        Write-Host "Successfully created tag $tag" 

        git push --tags
         Write-Host "Successfully pushed tag $tag"     

      failOnStderr: false

#azure-pipelines #azure #azure-devops #devops

Ruthie  Bugala

Ruthie Bugala


How to set up Azure Data Sync between Azure SQL databases and on-premises SQL Server

In this article, you learn how to set up Azure Data Sync services. In addition, you will also learn how to create and set up a data sync group between Azure SQL database and on-premises SQL Server.

In this article, you will see:

  • Overview of Azure SQL Data Sync feature
  • Discuss key components
  • Comparison between Azure SQL Data sync with the other Azure Data option
  • Setup Azure SQL Data Sync
  • More…

Azure Data Sync

Azure Data Sync —a synchronization service set up on an Azure SQL Database. This service synchronizes the data across multiple SQL databases. You can set up bi-directional data synchronization where data ingest and egest process happens between the SQL databases—It can be between Azure SQL database and on-premises and/or within the cloud Azure SQL database. At this moment, the only limitation is that it will not support Azure SQL Managed Instance.

#azure #sql azure #azure sql #azure data sync #azure sql #sql server

Aisu  Joesph

Aisu Joesph


Azure SDK Release (June 2021)

Release Highlights

Welcome to the June release of the Azure SDK. We have updated the following libraries:

Stable Releases

  • Azure Cognitive Search for .NET, Java (version 11.4), and JavaScript and Python (version 11.2)
  • Adds stable features and bug fixes from the beta releases. See the Cognitive Search changelog for more details.
  • Preview service features not generally available yet, like Semantic Search and Normalizers, are not included in this release.
  • Support for geospatial types in core for .NET and Java.
  • Support for knowledge store.
  • Azure Data Tables version 12.0
  • Read more here: Announcing the new Azure Data Table Libraries.
  • Azure SDK for Python (Conda) packages are now generally available in the Microsoft channel.
  • Read more here: Introducing the Azure SDK for Python (Conda).
  • See also:
  • Event Grid for Java (version 4.4), JavaScript and Python (version 4.3)
  • Adds new system events definition for Storage Blob and Azure Communication Service.
  • Form Recognizer version 3.1
  • This release marks the stability of the changes introduced in package versions 3.1.0-beta.1 through 3.1.0-beta.3.
  • Core, Identity, and Azure Storage for C++ version 1.0
  • This release marks the general availability for Core, Identity, and Azure Storage.
  • To get started and view samples, view the README on the Azure SDK for C++ repo.
  • Quickstarts and documentation are being updated at Microsoft Docs.
  • Key Vault Administration, Certificates, Keys and Secrets.
  • Key Vault Administration is a new library that allows for role-based access control (RBAC), and backup and restore operations for Managed HSM.
  • Key Vault Keys added functionality:
  • Support for Managed HSM.
  • Cryptography clients now support executing all operations locally if given a JsonWebKey.
  • Support for creating and importing symmetric keys for Managed HSM.
  • RSA keys now support providing a public exponent.

#azure sdk #azure #azure-sdk #javascript #python #release #sdk