Migrating Data From CosmosDB to Azure Storage using Azure Data Factory

Migrating Data From CosmosDB to Azure Storage using Azure Data Factory

Here's what the process looks like for setting up Data Factory: In the portal, navigate to Data Factories. A short guide for migrating between Azure CosmosDB and Storage accounts using Data Factory.

Recently I had the pleasure of being tasked with migrating an old CosmosDB Core(SQL) account to Azure Storage. This account had been around since the early days of the project, and it contained nearly ~100GB across two collections, each containing PID (Personally Identifiable Data) that had to be kept for compliance purposes. Needless to say, I was more than apprehensive about touching this. The backstory behind this is that a few months ago we changed our app to use the MongoDB API for CosmosDB in a separate account. We already used MongoDB for storing other data, so we decided to consolidate all of these records in one account and access it with one client. were previously kept in the SQL account. The tech leads decided to consolidate everything in the MongoDB account and to bin off the SQL one when we knew how we would go about doing so. I started digging around to find if the Data Migration Tool would fit our purposes, but I’m on a Mac, and the tool only runs on Windows 🤦‍♂. I debated spinning up a VM as a temporary measure and installing it there, but felt that this might cause us problems later on if the VM was restarted at any point during the migration. Creating a Data Factory Full disclosure, I used the GUI for all of this, mainly because this is a one-off export that I don’t plan to do again. I would usually use Terraform, however it seemed like overkill to figure out all of the resources and arguments for something that would be used once and only once. Here’s what the process looks like for setting up Data Factory: In the portal, navigate to Data Factories Image for post 2. Click Add, then complete the creation wizard. Set the Version to V2, and tick the box for Configure Git later if you don’t want to do this step now. Once you’re done, click Review + Create to create the Data Factory. 3. Wait for it to finish creating.

devops azure migration data storage

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

Create, Build, Deploy and Configure an Azure Function with Azure DevOps and Azure CLI

How to create, build, deploy and configure an Azure Function using Azure DevOps, Azure CLI and Powershell.

Updated: Announcing Azure DevOps Server 2020 RC1 | Azure DevOps Blog

Update: We received feedback in this blog and the Developer Community about an issue after upgrading from Azure DevOps Server 2019 Update 1.1 to Azure DevOps Server 2020 RC1.

Migrating Your Data with Azure Migrate

In this video, learn about the process of migrating your database to Azure. We'll cover what tools and services are available to help you through this journey, highlighting Azure Migrate and Database Migration Service as two helpful assets to help you navigate this change

How to Extend your DevOps Strategy For Success in the Cloud?

DevOps and Cloud computing are joined at the hip, now that fact is well appreciated by the organizations that engaged in SaaS cloud and developed applications in the Cloud. During the COVID crisis period, most of the organizations have started using cloud computing services and implementing a cloud-first strategy to establish their remote operations. Similarly, the extended DevOps strategy will make the development process more agile with automated test cases.

Populate Azure SQL Database from Azure Blob Storage using Azure Data Factory

Populate Azure SQL Database from Azure Blob Storage using Azure Data Factory. In this article, we will learn how to use Azure Data Factory to populate data from Azure Blob Storage to Azure SQL Database.