Training Jobs, Sweeps, Distributed Training and Pipelines

What it is

A command job runs your training script on compute with a defined environment, code and inputs. A sweep job runs many trials of that command with different hyperparameters. Distributed training spreads one job across several nodes or GPUs. A pipeline chains components (prepare, train, evaluate) into a repeatable workflow.

How it works

Command jobs

$schema: https://azuremlschemas.azureedge.net/latest/commandJob.schema.json
code: ./src
command: python train.py --data ${{inputs.training_data}} --lr ${{inputs.lr}}
inputs:
  training_data:
    type: uri_folder
    path: azureml:churn-raw:3
  lr: 0.01
environment: azureml:sklearn-train:2
compute: azureml:cpu-cluster
experiment_name: churn-training

az ml job create --file job.yml --stream

The folders ./outputs and ./logs are uploaded automatically. If you omit compute and add a resources block (instance_type, instance_count), the job uses serverless compute. In Python SDK v2 the same job is command(code=..., command=..., inputs=..., environment=..., compute=...) submitted with ml_client.jobs.create_or_update.

Hyperparameter tuning with sweeps

$schema: https://azuremlschemas.azureedge.net/latest/sweepJob.schema.json
type: sweep
trial:
  code: ./src
  command: python train.py --lr ${{search_space.lr}} --depth ${{search_space.depth}}
  environment: azureml:sklearn-train:2
compute: azureml:cpu-cluster
sampling_algorithm: random
search_space:
  lr:
    type: choice
    values: [0.001, 0.01, 0.1]
  depth:
    type: choice
    values: [4, 6, 8, 10]
objective:
  goal: maximize
  primary_metric: auc
limits:
  max_total_trials: 20
  max_concurrent_trials: 4
  timeout: 7200
early_termination:
  type: bandit
  slack_factor: 0.1
  evaluation_interval: 1
  delay_evaluation: 5

Early termination policies stop poor trials: bandit (stop trials outside a slack of the best), median stopping and truncation selection. The script must log the primary metric with exactly the name in primary_metric, using mlflow.log_metric.

Distributed training

Add a distribution block and set the node count in resources.

command: python train.py
environment: azureml:pytorch-gpu:1
compute: azureml:gpu-cluster
resources:
  instance_count: 2
distribution:
  type: pytorch
  process_count_per_instance: 4

Use GPU VM sizes, usually one process per GPU. In data parallel training each GPU holds a copy of the model and processes a different slice of data; for models too large for one GPU you need model or sharded parallelism, for example with DeepSpeed. Write checkpoints to outputs so a preempted low priority node can resume.

Training pipelines

$schema: https://azuremlschemas.azureedge.net/latest/pipelineJob.schema.json
type: pipeline
display_name: churn-pipeline
settings:
  default_compute: azureml:cpu-cluster
inputs:
  raw_data:
    type: uri_folder
    path: azureml:churn-raw:3
jobs:
  prep:
    type: command
    component: azureml:prep_data:1
    inputs:
      raw: ${{parent.inputs.raw_data}}
  train:
    type: command
    component: azureml:train_model:1
    inputs:
      training_data: ${{parent.jobs.prep.outputs.clean}}

The step order comes from the data wiring: train waits for prep.outputs.clean. Unchanged steps are reused from cache. Run a pipeline on a timer with a schedule:

$schema: https://azuremlschemas.azureedge.net/latest/schedule.schema.json
name: weekly-retrain
trigger:
  type: recurrence
  frequency: week
  interval: 1
create_job: ./pipeline.yml

az ml schedule create --file schedule.yml activates it. Python users build the same graph with the @dsl.pipeline decorator.

A worked example

A team sweeps lr and depth with random sampling and a bandit policy, then promotes the best settings into the train_model component. A weekly schedule runs prep, train and evaluate as one pipeline on a GPU cluster that scales to zero between runs.

How the exam asks about this

• "Stop low performing trials early." An early termination policy such as bandit.
• "Search a continuous range cheaply." Random sampling (grid needs discrete choices).
• "Train a PyTorch model across four nodes." distribution: pytorch with instance_count: 4.