Serving and scaling models · Topic 1 of 3
Deploying Models for Batch and Online Inference
Serve models in batch or online on Agent Platform, Model Garden, Cloud Run or GKE, package them in prebuilt or custom containers, add preprocessing.
- 5 min
- Batch inference: no endpoint, reads and writes BigQuery or Cloud Storage, pays only while running.
- Online endpoints: real-time, autoscaling, traffic splitting, monitoring.
- Prebuilt containers for TensorFlow, scikit-learn, XGBoost and PyTorch; custom containers for the rest.
What it is
Serving makes a trained model answer requests. Batch inference scores a whole dataset at once and writes the results to storage. Online inference answers individual requests through an endpoint in real time.
On Gemini Enterprise Agent Platform, models are packaged in a serving container: a prebuilt one for common frameworks, or a custom one you build. Many models also need preprocessing of the request and postprocessing of the output.
How it works
Choosing where to serve
| Service | Choose it when |
|---|---|
| Agent Platform Inference, online endpoint | Managed real-time serving with autoscaling, traffic splitting, monitoring and explanations |
| Agent Platform Inference, batch inference | Score large datasets from BigQuery or Cloud Storage; no endpoint, pay only while the job runs |
| Model Garden deployment | Deploy an open or partner model to an endpoint with a ready-made serving container |
| Cloud Run | Serverless container serving that scales to zero; light models or apps that already run on Cloud Run (GPUs available) |
| GKE | Full control of the serving stack, existing Kubernetes platform, or high-throughput LLM serving with your own tools |
BigQuery ML ML.PREDICT | Batch predictions where the data and consumers are already in BigQuery |
from google.cloud import aiplatform
PREBUILT_XGBOOST_SERVING_IMAGE = "..." # a prebuilt XGBoost serving container URI
model = aiplatform.Model.upload(
display_name="churn-xgb",
artifact_uri="gs://my-models/churn/v3/",
serving_container_image_uri=PREBUILT_XGBOOST_SERVING_IMAGE,
)
# Online: deploy to an endpoint
endpoint = model.deploy(machine_type="n1-standard-4",
min_replica_count=1, max_replica_count=5)
print(endpoint.predict(instances=[[34, 0, 12.5, 3]]).predictions)
# Batch: no endpoint needed
model.batch_predict(
job_display_name="churn-nightly",
bigquery_source="bq://my-project.retail.customers_today",
bigquery_destination_prefix="bq://my-project.retail_scores",
machine_type="n1-standard-4",
)Prebuilt and custom containers
| Framework | Prebuilt serving container expects |
|---|---|
| TensorFlow | A SavedModel directory |
| scikit-learn | model.joblib |
| XGBoost | model.bst or model.joblib |
| PyTorch | A model archive named model.mar, served by TorchServe |
Use a custom container when the framework, runtime or dependencies are not covered, or when the server must do extra work. A custom container must run an HTTP server that answers health checks and prediction requests on the port and routes the platform provides (environment variables such as AIP_HTTP_PORT, AIP_HEALTH_ROUTE and AIP_PREDICT_ROUTE). The image is stored in Artifact Registry.
"The scikit-learn model needs a custom feature transformation at prediction time" does not require writing a full web server. A custom prediction routine adds your own preprocessing to the standard serving flow.
Inference preprocessing and postprocessing
| Option | How it works | Best for |
|---|---|---|
| Inside the model | Preprocessing layers or a saved scikit-learn Pipeline are part of the model artifact | Same logic in training and serving with no extra code |
| BigQuery ML TRANSFORM | Preprocessing saved with the BigQuery ML model | SQL models |
| Custom prediction routine | A Python Predictor class with preprocess, predict and postprocess methods, built into a container by the SDK | Custom logic without writing a web server |
| Custom container | Your server does everything | Full control |
| Feature Store lookup | The app or server fetches stored features by entity ID before calling the model | Features computed from history, not in the request |
| Application code | The caller transforms data | Last resort; risks drift from training logic |
Postprocessing turns raw output into something the app can use: map class indexes to labels, apply a business threshold, or format the response.
A worked example
A team has a PyTorch text classifier and an XGBoost pricing model. The text model must tokenize text exactly as in training and return label names, not indexes.
The engineer serves the XGBoost model with the prebuilt XGBoost container. For the PyTorch model, a custom prediction routine loads the same tokenizer used in training in preprocess and maps indexes to names in postprocess. Nightly price recalculation for the whole catalog runs as a batch inference job from BigQuery.
How the exam asks about this
- "Score 50 million rows in BigQuery every night at the lowest cost." A batch inference job reading from and writing to BigQuery.
- "Serve a model built with a framework that has no prebuilt container." Build a custom container with an HTTP server for health and prediction routes, push it to Artifact Registry and deploy it.
- "Requests need the same normalization as training without changing client apps." Put preprocessing in the model or a custom prediction routine.
What to remember for the exam
- Batch inference: no endpoint, reads and writes BigQuery or Cloud Storage, pays only while running.
- Online endpoints: real-time, autoscaling, traffic splitting, monitoring.
- Prebuilt containers for TensorFlow, scikit-learn, XGBoost and PyTorch; custom containers for the rest.
- Custom containers must answer health and prediction routes on the provided port.
- Custom prediction routines add preprocessing and postprocessing without a hand-written server.
Do not deploy to an always-on endpoint just to score a table once a day. Batch inference avoids paying for idle serving nodes.
Common mistakes
Duplicating preprocessing in every client app. Clients drift from training logic; put it in the model or the serving routine.
Building a custom container when a prebuilt one works. Prebuilt containers are maintained and patched for you.
Forgetting health checks in custom containers. The platform marks the model unhealthy and it never receives traffic.
Check yourself
What does batch inference need that online inference does not, and vice versa?
Batch inference needs input and output locations such as BigQuery or Cloud Storage but no endpoint; online inference needs a deployed endpoint that answers single requests.
What must a custom serving container provide?
An HTTP server that answers the health route and the prediction route on the port the platform passes in environment variables.
How can you add preprocessing to a scikit-learn model without writing a web server?
Use a custom prediction routine with a Predictor class that implements preprocess, predict and postprocess, or save a scikit-learn Pipeline that includes the preprocessing.
Topic 1 of 3 in Serving and scaling models
Test yourself on Serving and scaling models
The full Google Cloud Professional Machine Learning Engineer question bank has 749 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Google Cloud. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.