Online and Batch Endpoints, Testing and Safe Rollout

What it is

An endpoint is a stable URL; a deployment is the model, code and compute behind it. One endpoint can hold several deployments and split traffic between them. Managed online endpoints serve low latency, synchronous requests. Batch endpoints score large datasets asynchronously. Both are managed: Azure patches the hosts, scales and recovers nodes for you.

How it works

Online endpoint and deployment

# endpoint.yml
$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineEndpoint.schema.json
name: churn-endpoint
auth_mode: key

# blue.yml
$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineDeployment.schema.json
name: blue
endpoint_name: churn-endpoint
model: azureml:churn-model:4
instance_type: Standard_DS3_v2
instance_count: 2

az ml online-endpoint create --file endpoint.yml
az ml online-deployment create --file blue.yml --all-traffic
az ml online-endpoint invoke --name churn-endpoint --request-file sample.json

For an MLflow model nothing else is needed. For a custom model add code_configuration (a scoring script with init() and run()) and an environment. Online endpoints use key or aml_token authentication. Scale out with more instances or Azure Monitor autoscale rules; tune request_settings (timeouts, concurrency) and liveness_probe / readiness_probe for slow starting models.

Batch endpoint

$schema: https://azuremlschemas.azureedge.net/latest/modelBatchDeployment.schema.json
name: nightly
endpoint_name: churn-batch
type: model
model: azureml:churn-model:4
compute: azureml:cpu-cluster
resources:
  instance_count: 2
settings:
  max_concurrency_per_instance: 2
  mini_batch_size: 10
  output_action: append_row
  output_file_name: predictions.csv
  retry_settings:
    max_retries: 3
    timeout: 300

az ml batch-endpoint invoke --name churn-batch --input azureml:churn-new:1

For batch, a mini batch is the unit of work, and output_action: append_row merges results into one file.

Test and troubleshoot

• Test one deployment before it gets traffic: az ml online-endpoint invoke --deployment-name green ....
• Debug locally with the --local flag on create commands before paying for cloud compute.
• Read container logs: az ml online-deployment get-logs --name blue --endpoint-name churn-endpoint --lines 100. Add --container storage-initializer to see model download problems.
• Common causes: a failing init() in the scoring script, a missing package in the environment, not enough quota for the VM size, or the managed identity lacking access to the model or registry.
• Status codes: 401 authentication or expired token, 404 no valid deployment with positive traffic, 408 request timeout, 424 error returned by the model container, 429 too many pending requests or rate limited (add instances or retry with backoff), 503 a spike in requests that autoscale cannot follow.

Use Application Insights and endpoint metrics (request latency, requests per minute, CPU and memory) to see the problem pattern.

Progressive rollout and rollback

1. Create a new deployment (green) without --all-traffic; it receives 0 percent.
2. Test it by deployment name.
3. Optionally mirror a share of live traffic to it: az ml online-endpoint update --name churn-endpoint --mirror-traffic "green=10". Mirrored requests are copied to green but only blue's response reaches the caller. Mirroring is limited to 50 percent.
4. Shift live traffic gradually: --traffic "blue=90 green=10", then 50/50, then 0/100.
5. Roll back instantly with --traffic "blue=100 green=0". Keep blue until green is proven, then delete it.

A worked example

Version 5 is deployed as green with 0 percent. After a clean mirror at 10 percent, traffic goes 10, 50 and 100 percent while latency and error alerts are watched. At 50 percent the error rate rises, so traffic returns to blue in one command and green is debugged offline.

How the exam asks about this

• "Test new model on real traffic without affecting users." Mirror traffic to the new deployment.
• "Gradually move users to a new version and revert fast." Blue/green deployments with traffic percentages.
• "Score 20 million rows overnight at lowest cost." A batch endpoint on a cluster that scales to zero.