Create Secure Batch ML Pipelines

Prerequisites

• Install Azure CLI

Create Virtual Network

• az group create -l westus -n ASecureMLPlatform

• az network vnet create -n mlvnet -g ASecureMLPlatform -l westus --address-prefix 10.1.0.0/16 --subnet-name mldefault --subnet-prefix 10.1.0.0/24

Create Storage Account

• az storage account create -n jomitsecuremldata -g ASecureMLPlatform -l westus --sku Standard_LRS --https-only true

• az network vnet subnet update -g ASecureMLPlatform --vnet-name mlvnet --name mldefault --service-endpoints "Microsoft.Storage"

• $subnetid=(az network vnet subnet show -g ASecureMLPlatform --vnet-name mlvnet --name mldefault --query id --output tsv)

• az storage account network-rule add -g ASecureMLPlatform --account-name jomitsecuremldata --subnet $subnetid

Create ML Workspace

• az extension add -n azure-cli-ml

• $storageid=(az storage account show -n jomitsecuremldata --query id --output tsv)

• az ml workspace create -w securemlws -g ASecureMLPlatform -l westus --storage-account $storageid

• az ml datastore list -w securemlws -g ASecureMLPlatform

• az ml datastore set-default -n workspaceblobstore -w securemlws -g ASecureMLPlatform

• az storage account update -g ASecureMLPlatform -n jomitsecuremldata --default-action Deny

• (Optional) Add Client IP in storage account firewall for access issues outside the network.

• (Optional) Entire ML Workspace behind VNET in on roadmap.

Create Data Science Virtual Machine

• az group deployment create -g ASecureMLPlatform --template-file dsvm-template.json --parameters dsvm-parameters.json --parameters adminPassword=<Your Password>

• (Optional) To enable MSI Authentication for notebooks, create Identity for VM and provide Contributor access to AML Workspace

• (Optional) Also try Notebook VM in VNET once GA

Attach AML Remote Compute

• az network public-ip update -g ASecureMLPlatform -n jomitsecuremlvm-ip --dns-name jomitsecuremlvm --allocation-method Dynamic

• az ml computetarget attach remote -a jomitsecuremlvm.westus.cloudapp.azure.com -n default -u jomit -p <Your Password> -g ASecureMLPlatform -w securemlws --ssh-port 22

Upload Input Datasets

• (Optional) You can also create a seperate storage account dedicated for input datasets but for now we will use the storage account associated with the workspace)

• az ml datastore upload -n workspaceblobstore -w securemlws -g ASecureMLPlatform -p ./data -u /inputdata

Create AML Pipeline

• az ml run submit-script -e buildpipeline -w securemlws -g ASecureMLPlatform -d ./aml_pipeline/dependencies.yaml ./aml_pipeline/build_pipeline.py --target default

• az ml dataset list -w securemlws -g ASecureMLPlatform

Run Published Pipeline

• az ml run submit-script -e buildpipeline -w securemlws -g ASecureMLPlatform -d ./aml_pipeline/dependencies.yaml ./tests/run_pipeline.py --target default

Additional Resources

• AML Pipeline Getting Started

• Secure Azure ML experimentation and inference jobs within an Azure Virtual Network

• MLOps using Azure ML Services and Azure DevOps