Python data automation in Azure ecosystem by connecting JupyterHub, Blob Storage...

Python data automation in Azure ecosystem by connecting JupyterHub, Blob Storage...

This Tutorial, Automating Python Data in Azure Ecosystem by Connecting JupyterHub, Blob Storage and PowerBI…

For any Data professional, automating data is key. Once we set up the process of data capture, data analysis, data storage and data visualisation, we want to be able to schedule periodic refreshes of the resulting dashboard to prevent the need for any manual intervention.

This tutorial was created based on a real world need to architect an automated data flow. Essentially, external data needs to be captured, analyzed and published to a PowerBI dashboard on a daily basis. It can be described as follows:

  1. *Access external data (public API’s or websites) with Python script running on Azure VM. *Virtual Machine needs to run completely in the cloud and have the ability to be scheduled via cronjobs.
  2. Clean, analyze, run ML algorithms and output data in a simple format for consumption into PowerBI. VM that is running a python script / JupyterHub notebook needs to ingest external data on a regular basis. Machine learning and analysis is then executed on this data set. The output of this step is a cleaned data set with business insights that need to flow into PowerBI. Yes, PowerBI has the capability to do data cleaning, processing and wrangling — yet for many analysts, it is inferior to the power of Python. Adding this layer of Python analysis gives the analyst much more power and flexibility for data analysis/machine learning.
  3. *Store data as an intermediary and connect to PowerBI. *The output of the above step is then uploaded to a Blob storage container in Azure, which is then connected to PowerBI for use in a dashboard.
  4. Publish PowerBI dashboard and set up automatic refresh schedule. The dashboard is published to a public report which is consumed within Teams by business stakeholders.

The following steps outline the process of building this data flow.

azure jupyter-notebook power-bi python

What is Geek Coin

What is GeekCash, Geek Token

Best Visual Studio Code Themes of 2021

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Microsoft Power BI Consulting | Power BI Solutions in India

Perform Data Analysis & Report Creation with our Microsoft Power BI Consulting Services. Our Power BI consultants help in BI solutions and execute business requirements gathering, architecture design, implementation, user training, cloud management, report enhancements projects, etc

Hire Power BI Developer | Microsoft Power BI consultants in India

Hire our expert Power BI consultants to make the most out of your business data. Our power bi developers have deep knowledge in Power BI data modeling, structuring, and analysis. 16+ Yrs exp | 2500+ Clients| 450+ Team

power bi course | power bi training | OnlineITGuru

Our Power BI Training will provide you to learn the Power BI tools and making reports easily with realty. Our Power BI Course also includes live sessions, live Pro

Power BI In Brief – 2020

Every month, we bring you news, tips, and expert opinions on Power BI? Do you want to tap into the power of Power BI? Ask the Power BI experts at ArcherPoint. ![This is image...

Comparing Power BI with other tools

Our Power BI Training will provide you to learn the Power BI tools and making reports easily with realty. Our Power BI Course also includes live sessions, live Pro