How to automate data operations leveraging Python with the power of Azure Data Factory and Batch Services

Before I took on my new role, the work I was doing mainly focused on transforming data, analyzing it and building insightful dashboards and visualizations from it. I was a consumer of the data, and the data I was using was being provided to me in the form of well structured data warehouses as well as flat files and Excel workbooks. While a certain portion of my work still involves analytics and visualization, the focus now is also on supporting these operations by building robust data pipelines that transform data from various sources into the right form, which can then be used by data analysts and BI analysts for generating insights.

Most of the data transformations I perform leverage Python and it’s libraries, mainly Pandas, to efficiently transform data. One of the advantages of using Python is the ability to connect to a multitude of data sources and sinks, which makes establishing workflows and pipelines quite convenient. However, most of these workflows reside in Python scripts that need to be executed, either manually or scheduled to run periodically. In an attempt to further streamline these pipelines, I started looking at automation solutions that would completely eliminate the need for manual intervention as well as use resources in the most optimal way.

I had previously read about Azure Data Factory and it’s ability to perform a multitude of ETL operations on data on the cloud as well as its support for a wide variety of data sources and sinks. I was also aware of it’s Python integration capabilities using Function Apps, Custom Activities or** Azure Databricks**. Upon further researching these three potential solutions, I decided to use Custom Activities to build a proof of concept for an automated data pipeline that leverages Python.

Azure Data Factory Interface

Azure Data Factory is a cloud based service that lets users perform a variety of data integration tasks using convenient visual elements rather than code, although it does support the latter as well. Besides support for various conventional ETL operations, it also supports custom data movement or transformation logic with the help of Custom Activities. The custom activity runs your customized code logic on an Azure Batch pool of virtual machines.

Azure Blob Storage will be our data repository since it supports easy file upload/download operations through Python and supports integration with Microsoft Event Grid, which will be crucial to establishing an event based trigger to run our pipeline based on file uploads to the Blob Storage.

Before we go into any further details about the functional understanding of our proof of concept, let’s talk about the elephant in the room, that is, Azure Batch Services.

#data-factory #azure #data-science #data-engineering #python