MLflow Tracking, AutoML, Notebooks and Comparing Jobs

What it is

Every training attempt is a job (an MLflow run) inside an experiment. The Azure Machine Learning workspace is an MLflow tracking server, so parameters, metrics and artifacts logged with MLflow are stored in the workspace and shown in the studio. Automated machine learning (AutoML) tries algorithms and settings for you. Notebooks are where you explore before turning code into jobs.

How it works

MLflow tracking

Inside a job on Azure Machine Learning compute the tracking URI is already set. From your own machine or a notebook you point MLflow at the workspace:

import mlflow
from azure.ai.ml import MLClient
from azure.identity import DefaultAzureCredential

ml_client = MLClient(DefaultAzureCredential(), sub_id, rg, ws_name)
tracking_uri = ml_client.workspaces.get(ws_name).mlflow_tracking_uri
mlflow.set_tracking_uri(tracking_uri)
mlflow.set_experiment("churn-training")

with mlflow.start_run():
    mlflow.log_param("max_depth", 6)
    mlflow.log_metric("auc", 0.91)
    mlflow.log_artifact("confusion_matrix.png")

The packages mlflow and azureml-mlflow must be installed. mlflow.autolog() (or a framework call such as mlflow.sklearn.autolog()) logs parameters, metrics and the model automatically for supported libraries.

Logged values appear on the job's Metrics, Outputs + logs and Overview tabs in the studio. A metric logged repeatedly with a step becomes a chart, which is how you watch loss fall during training. Artifacts are stored in the workspace's default storage, and the job also keeps the code snapshot, the environment and the input assets, so any run can be traced and reproduced. Add tags (for example the Git commit or data version) to make runs easy to filter.

Automated machine learning

AutoML trains many models with different algorithms and featurization, then ranks them on a primary metric. Supported tasks include classification, regression, time series forecasting, and vision and natural language tasks.

from azure.ai.ml import automl, Input

job = automl.classification(
    compute="cpu-cluster",
    experiment_name="churn-automl",
    training_data=Input(type="mltable", path="azureml:churn-train:1"),
    target_column_name="churned",
    primary_metric="AUC_weighted",
    n_cross_validations=5,
    enable_model_explainability=True,
)
job.set_limits(timeout_minutes=60, trial_timeout_minutes=15, max_trials=20,
               enable_early_termination=True)
returned = ml_client.jobs.create_or_update(job)

Key points: training data must be an mltable; set limits to control cost; use n_cross_validations or validation data; block algorithms with set_training(blocked_training_algorithms=[...]); the best model can be registered and deployed like any other. AutoML produces a parent job and one child run per trial, and can build ensemble models.

Notebooks

Azure Machine Learning studio has built in notebooks and can open VS Code (web or desktop) attached to a compute instance. The compute instance supplies the kernel and the files live in your user folder. Use notebooks to explore data and prototype, then move stable code into scripts, components and jobs so it can be automated and reviewed.

Comparing jobs

In the studio, select several runs and choose Compare to see parameters and metric charts side by side. In code, query runs with MLflow:

runs = mlflow.search_runs(
    experiment_names=["churn-training"],
    filter_string="metrics.auc > 0.9",
    order_by=["metrics.auc DESC"],
)
best = runs.iloc[0]
print(best.run_id, best["metrics.auc"], best["params.max_depth"])

For a sweep or AutoML, compare child runs under the parent. Compare only runs that used the same data and the same metric definition.

A worked example

A data scientist prototypes in a notebook, then runs AutoML with a 60 minute limit on cpu-cluster. The best trial scores 0.91 AUC. A hand-built XGBoost job logs 0.93. She uses mlflow.search_runs ordered by AUC to select the winner and registers that model.

How the exam asks about this

• "Log metrics from a job without extra setup." MLflow in the job; the tracking URI is preset on Azure Machine Learning compute.
• "AutoML fails with a CSV path as training data." Supply an mltable data asset.
• "Rank all runs in an experiment by accuracy in code." mlflow.search_runs with order_by.