Use Case

• Load data from data source in this case sample dataset
• Process data (ETL) using Pyspark
• ETL work is done in combination with pyspark dataframe and Spark SQL
• Save processed data into Synapse dedicated SQLPools
• Resume and Pause dedicated SQL pools as we run ETL
• Save a copy in default storage for serverless activities
• Build and Train Machine learning model
• We are using same notebook as pyspark but building with scala code
• Build different Pipeline for Resuming and Pausing dedicated SQL Pools

#data-science #azure-synapse-analytics