Ability to run notebook code as Pipeline

Prerequisite

• Azure Account
• Azure Machine learning
• Create a compute instance
• Create a compute cluster as cpu-cluster
• Select Standard D series version
• Create Train file to train the model
• Create a pipeline file to run the as pipeline

Steps

Create Train file as train.py

• Create a directory ./train_src
• Create a train.py
• Should be a python file not notebook

Create Pipeline code

• Load the workspace config

• Get the default store information

• Create compute cluster

• Load the package dependencies

• Load the data set

• set the dataset as input

• Setup output optional

• I am only creating single step

• setup the pipeline config and assign

• Validate the pipeline

• Now time to submit the pipeline

• Wait for pipeline to finish

• Now lets publish the pipeline

• Every publish will create a REST endpoint

• I logged into the Azure ML Studio

• Go to Pipeline on the left menu

• Click on pipeline endpoint

• should see a pipeline — Published_Titanic_Pipeline_Notebook

• Click submit and see if the pipeline line runs

• Now go to ADF or Synapse Integrate

• Create a New pipeline

• Name is AzureMLPipelinetest

• Drag and drop Azure Machine learning services (only to run published pipeline)

• Create a New Source for Azure Machine learning using service principal account

#data-factory #machine-learning #azure-ai #azure-machine-learning #azure data factory