Automating and orchestrating ML pipelines

What this chapter covers

Automating and orchestrating ML pipelines is about 18% of the Google Cloud Professional Machine Learning Engineer exam. It covers replacing manual notebook steps with pipelines that prepare data, train, validate and deploy models the same way every time, and that retrain models automatically when needed. The managed pipeline service runs on Gemini Enterprise Agent Platform (the platform formerly called Vertex AI).

It covers developing end-to-end pipelines (validating data and models; building and orchestrating pipelines with managed or unmanaged services, from templates or custom code, such as Agent Platform Pipelines, Managed Service for Apache Airflow and Ray on Gemini Enterprise Agent Platform; keeping preprocessing consistent between training and serving) and automating retraining (choosing a retraining policy; deploying models through continuous integration, continuous delivery and continuous training pipelines, for example with Cloud Build).

The exam has 50 to 60 multiple choice and multiple select questions in 2 hours. Questions here usually describe a manual or fragile process and ask "What should you do?"; the answer is the managed, automated and repeatable option that fits the team's tools.

How the topics fit together

CI tests and packages the pipeline, the pipeline trains and validates models (continuous training), and only models that pass the gate are deployed. Monitoring closes the loop by triggering the next run.

How to use these pages

Each topic page has the same shape, so you can skim for what you need.

• What it is: the idea in plain words.
• How it works: the mechanism, with tables and code examples.
• What to remember for the exam: the facts to memorise and how questions are phrased.
• Common mistakes: the classic traps.
• Check yourself: three questions to test your understanding.