Automating and orchestrating ML pipelines · Topic 2 of 2
Retraining Policies and CI/CD/CT Pipelines
Choose a retraining policy based on schedule, new data, drift or performance, and deploy models through CI/CD/CT pipelines built with Cloud Build.
- 6 min
- Includes a diagram
- Retraining policy depends on data change rate, label delay and retraining cost.
- Performance-based triggers need labels; drift-based triggers work without them.
- Pipeline schedules, Cloud Scheduler, events and monitoring alerts can all start runs.
What it is
Models go stale as the world changes. A retraining policy decides when a model is retrained: on a schedule, when enough new data arrives, or when monitoring shows drift or falling performance.
CI/CD/CT brings software delivery practice to ML. Continuous integration tests and builds code, continuous delivery deploys the pipeline and models safely, and continuous training reruns the training pipeline automatically to produce new models.
How it works
Choosing a retraining policy
| Policy | Trigger | Fits | Trade-off |
|---|---|---|---|
| Scheduled | Fixed time, for example weekly | Stable, predictable change; labels arrive regularly | Simple, but may retrain for nothing or too late |
| New data volume | Enough new labeled data has arrived | Data arrives in batches | Needs tracking of new data |
| Performance based | A quality metric falls below a threshold | Labels (ground truth) arrive soon after predictions | Best signal, but needs labels |
| Drift based | Model Monitoring detects feature or prediction drift | Labels are delayed or rare | Drift does not always mean worse accuracy |
| On demand | A person or upstream event starts it | Known changes such as a new product line | Manual decision |
Choose by how fast the data changes, how quickly labels arrive, and what a retrain costs. A fraud model facing new attack patterns may retrain daily; a model for a stable physical process may need it rarely.
"Labels arrive months later, but input distributions change" points to drift-based triggers. "Labels arrive daily" allows a performance-based policy, which is the most direct signal.
Triggering retraining
| Trigger | Mechanism |
|---|---|
| Schedule | A pipeline schedule on Agent Platform Pipelines, or Cloud Scheduler |
| New data | An event when files land in Cloud Storage or a table is loaded, routed through Eventarc or Pub/Sub to a function that starts a pipeline run |
| Drift or performance alert | Model Monitoring or a Cloud Monitoring alert sends a notification (for example to Pub/Sub) that starts a pipeline run |
| Larger workflow | A Managed Service for Apache Airflow DAG starts the pipeline after upstream jobs finish |
from google.cloud import aiplatform
job = aiplatform.PipelineJob(
display_name="churn-retrain",
template_path="churn_pipeline.yaml",
pipeline_root="gs://my-bucket/pipeline-root",
parameter_values={"model_uri": "gs://my-models/churn/candidate"},
)
job.create_schedule(display_name="weekly-churn-retrain", cron="0 3 * * 1")The three parts of CI/CD/CT
| Part | What it does for ML |
|---|---|
| Continuous integration | On each commit, Cloud Build runs unit tests on components and data code, builds container images, compiles the pipeline |
| Continuous delivery | Pushes images and pipeline templates to Artifact Registry and deploys the pipeline to staging, then production |
| Continuous training | The deployed pipeline retrains on new data by schedule or trigger and registers candidate models |
| Model deployment | A gate checks the candidate; passing models go to an endpoint, often as a canary, with approval for high-risk models |
A minimal cloudbuild.yaml:
steps:
- name: python:3.11
entrypoint: bash
args: ["-c", "pip install -r requirements.txt && pytest tests/"]
- name: gcr.io/cloud-builders/docker
args: ["build", "-t", "us-central1-docker.pkg.dev/$PROJECT_ID/ml/trainer:$SHORT_SHA", "trainer/"]
- name: gcr.io/cloud-builders/docker
args: ["push", "us-central1-docker.pkg.dev/$PROJECT_ID/ml/trainer:$SHORT_SHA"]
- name: python:3.11
entrypoint: bash
args: ["-c", "pip install -r requirements.txt && python compile_and_submit.py --image-tag=$SHORT_SHA"]A Cloud Build trigger runs this on each push to the main branch. Separate projects for development, staging and production keep tests away from live models.
Environments and approvals
- Promote the same pipeline template and image from staging to production; do not rebuild for production.
- Store everything in Git: pipeline code, component code, infrastructure definitions.
- Require a manual approval before production deployment when the model is high risk or regulated.
- Record which commit, data and run produced each model; pipelines and Model Registry give this lineage.
A worked example
A lender's credit model is retrained by hand whenever someone remembers. Labels (loan defaults) take months to arrive.
The engineer puts the pipeline code in Git. Cloud Build tests and compiles it on each merge and pushes the template to Artifact Registry. Because labels are slow, retraining runs monthly on a schedule and also when Model Monitoring reports feature drift. Each run registers a challenger; it is deployed only if it beats the champion on the latest labeled data and a risk officer approves.
How the exam asks about this
- "Automatically retrain when the monitoring service detects feature drift." Route the monitoring alert through Pub/Sub to start an Agent Platform Pipelines run.
- "Each commit to the training code should be tested and the pipeline rebuilt automatically." A Cloud Build trigger that runs tests, builds images and compiles the pipeline.
- "Fresh labels arrive daily and accuracy drops quickly when patterns change." Use a performance-based policy: evaluate daily and retrain when the metric falls below a threshold.
What to remember for the exam
- Retraining policy depends on data change rate, label delay and retraining cost.
- Performance-based triggers need labels; drift-based triggers work without them.
- Pipeline schedules, Cloud Scheduler, events and monitoring alerts can all start runs.
- CI tests and builds; CD promotes templates and models; CT retrains automatically.
- Gate every retrained model; add approvals for high-risk models.
Retraining on a schedule and deploying automatically without evaluation can push a worse model to production. Continuous training always needs a validation gate.
Common mistakes
Retraining too often. Daily retraining of a stable model wastes compute and adds release risk for no gain.
Treating drift as proof of lower accuracy. Drift is a warning; confirm with evaluation before replacing a model.
Rebuilding artifacts for each environment. Promote the tested image and template so production runs exactly what staging tested.
Check yourself
When is a drift-based retraining trigger better than a performance-based one?
When ground-truth labels arrive late or rarely, so accuracy cannot be measured quickly but input or prediction drift can.
What does continuous training add to CI/CD?
Automatic reruns of the training pipeline on new data, by schedule or trigger, producing candidate models that are validated before deployment.
What does Cloud Build do in an ML CI pipeline?
It runs tests, builds and pushes container images, and compiles and publishes the pipeline when code is committed.
Topic 2 of 2 in Automating and orchestrating ML pipelines
Test yourself on Automating and orchestrating ML pipelines
The full Google Cloud Professional Machine Learning Engineer question bank has 749 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Google Cloud. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.