Building, Validating and Orchestrating ML Pipelines

What it is

An ML pipeline is a graph of steps (components) such as ingest, validate data, transform, train, evaluate and deploy, run automatically with each step's inputs and outputs recorded. Pipelines make training reproducible and let a team rerun everything with one command or trigger.

Good pipelines also validate what flows through them: data is checked before training, models are checked before deployment, and preprocessing is shared so serving sees data the same way training did.

How it works

Choosing the orchestrator

Templates or custom pipelines

Start from templates and prebuilt components; write custom components only for logic they lack.

A small KFP pipeline with a model gate

from kfp import dsl, compiler

@dsl.component(base_image="python:3.11")
def evaluate(model_uri: str) -> float:
    # load the candidate model and return its AUC on the test split
    return 0.91

@dsl.component(base_image="python:3.11")
def deploy(model_uri: str):
    print(f"deploying {model_uri}")

@dsl.pipeline(name="churn-train")
def churn_pipeline(model_uri: str):
    auc = evaluate(model_uri=model_uri)
    with dsl.If(auc.output >= 0.85):
        deploy(model_uri=model_uri)

compiler.Compiler().compile(churn_pipeline, "churn_pipeline.yaml")

from google.cloud import aiplatform

aiplatform.PipelineJob(
    display_name="churn-train",
    template_path="churn_pipeline.yaml",
    pipeline_root="gs://my-bucket/pipeline-root",
    parameter_values={"model_uri": "gs://my-models/churn/candidate"},
).submit()

Steps whose inputs have not changed can be skipped through execution caching, which saves time and cost on reruns.

Validating data

A failed check should stop the pipeline before training wastes compute or produces a bad model.

Validating models

TFX's Evaluator marks a passing model as "blessed"; in KFP, a conditional step like dsl.If deploys only models that pass.

Consistent preprocessing between training and serving

Training-serving skew appears when serving transforms data differently from training. Prevent it by having one implementation:

A worked example

A team retrains a demand model by running six notebooks in order. Last quarter an upstream table lost a column and a broken model reached production.

The engineer rebuilds the process as an Agent Platform Pipelines run: data validation stops on schema or distribution anomalies; Google Cloud Pipeline Components handle training and batch inference; and a gate deploys only if the candidate beats the champion. Preprocessing moves into a shared component also used by the serving container.

How the exam asks about this

• "Run ML steps reproducibly with lineage and no infrastructure to manage." Agent Platform Pipelines with Kubeflow Pipelines or TFX.
• "Upstream schema changes sometimes produce bad models." Add a data validation step that stops the pipeline on schema anomalies.
• "The company already orchestrates hundreds of ETL jobs in Airflow and wants model training in the same schedule." Use Managed Service for Apache Airflow to trigger Agent Platform Pipelines runs.