Serving and scaling models · Topic 3 of 3
Scaling Online Serving with Endpoints and Features
Serve features from Feature Store, choose public or private endpoints and serving hardware, autoscale on throughput and tune models for production serving.
- 5 min
- Feature views sync BigQuery features into an online store for low-latency reads.
- Private Service Connect endpoints keep traffic off the public internet.
- CPUs for small models, GPUs and TPUs for large ones, edge for offline and on-device.
What it is
An online model must answer every request within its latency target, whether traffic is ten requests a minute or thousands a second. Scaling online serving means fetching features fast, putting the endpoint on the right network, choosing hardware, letting replicas autoscale with throughput, and making the model itself cheaper to run.
How it works
Serving features with Feature Store
Many online models need history the request does not contain, such as "orders in the last 7 days". Agent Platform Feature Store serves these from an online store: a feature view syncs selected BigQuery features into it, and the app or serving code reads them by entity ID in milliseconds.
| Online serving type | Fits |
|---|---|
| Bigtable online serving | The recommended type; large data volumes, scheduled or continuous sync |
| Optimized online serving | Embedding management; deprecated, retiring in February 2027 |
Set the feature view's sync schedule to match how fresh the features must be. Training reads the same features from BigQuery, so both paths share one definition.
Public and private endpoints
| Endpoint | Reached through | Choose when |
|---|---|---|
| Public endpoint | The internet, authenticated with IAM | Clients outside your network; simplest setup |
| Dedicated public endpoint | Its own public DNS name, isolated from other users' traffic | Production traffic that should be isolated from shared public serving |
| Private endpoint with Private Service Connect | An internal IP in your VPC | Traffic must stay off the public internet; access from several VPCs or on-premises |
| Private endpoint with VPC Network Peering | A peered VPC network | Existing private setups built on peering |
"Prediction traffic must never cross the public internet" points to a private endpoint, usually through Private Service Connect. Add VPC Service Controls when data exfiltration is also a concern.
Choosing serving hardware
| Hardware | Serve |
|---|---|
| CPU | Tree models, linear models, small neural networks; low cost per request |
| GPU | Large deep learning models, image and video models, LLMs; best with batching |
| TPU | Large models at high throughput, including LLM serving |
| Edge | Models exported to run on devices (for example AutoML image models exported for mobile) for offline use or very low latency |
Measure latency and cost per 1,000 predictions on real traffic. A GPU that sits mostly idle costs more than several CPU replicas.
Scaling the serving backend
Agent Platform Inference scales each deployed model between a minimum and maximum replica count.
| Setting | Effect |
|---|---|
min_replica_count | Replicas always running: protects latency at sudden spikes, but costs money when idle |
max_replica_count | Upper bound: caps cost and protects quota |
| CPU utilization target | Adds replicas when average CPU use passes the target |
| Accelerator duty cycle target | Scales GPU-served models on accelerator use |
endpoint = model.deploy(
machine_type="n1-standard-8",
min_replica_count=2,
max_replica_count=20,
autoscaling_target_cpu_utilization=60,
)New replicas take time to start, so autoscaling does not absorb an instant spike. Raise the minimum before a known peak.
| Containerized serving option | Scaling behaviour |
|---|---|
| GKE | Horizontal Pod Autoscaler and cluster autoscaler; GPU and TPU node pools; full control of LLM servers |
| Cloud Run | Scales on concurrent requests, down to zero when idle; good for spiky, light traffic |
| Gemini and models as a service | Scaling handled by Google; Provisioned Throughput reserves capacity |
Tuning models for production serving
| Technique | Gain | Cost |
|---|---|---|
| Quantization (for example 8-bit) | Smaller, faster model | Small accuracy loss to measure |
| Distillation | A small student model learns from a large teacher | Extra training |
| Pruning or a smaller architecture | Lower latency and memory | Possible accuracy loss |
| Request batching | Higher throughput per GPU | Slightly higher latency per request |
| Compiled or optimized runtimes | Faster inference on the same hardware | Compatibility testing |
| Caching frequent predictions | Skips the model for repeat inputs | Stale results if inputs change meaning |
Training choices matter too: a model trained with a simpler architecture or fewer features may meet accuracy targets and serve at a fraction of the cost.
A worked example
A payments company scores each card transaction in under 100 ms. Peaks are ten times the night-time load, and traffic must stay inside its network.
The engineer serves the model from a private endpoint through Private Service Connect. Customer history features come from a Bigtable online store. The deployed model runs on CPUs with a minimum of 4 replicas, a maximum of 40, and a CPU target, and the minimum is raised before the holiday peak. Quantization cut model latency by a third with no measurable accuracy loss.
How the exam asks about this
- "Latency spikes when traffic suddenly rises at 9 am each day." Raise the minimum replica count before the known peak, as autoscaling takes time to add replicas.
- "Online predictions need customer features computed from history with millisecond reads." Serve them from a Feature Store online store through a feature view.
- "A GPU-served model is too slow and expensive; a small accuracy drop is acceptable." Quantize or distill the model.
What to remember for the exam
- Feature views sync BigQuery features into an online store for low-latency reads.
- Private Service Connect endpoints keep traffic off the public internet.
- CPUs for small models, GPUs and TPUs for large ones, edge for offline and on-device.
- Autoscaling runs between min and max replicas; set the minimum for known peaks.
- Quantization, distillation and batching cut serving cost and latency.
Setting the minimum replica count to the peak level fixes latency but pays for peak capacity all day. Use autoscaling and raise the minimum only around predictable peaks.
Common mistakes
Putting GPUs behind a small tabular model. CPUs are usually cheaper per prediction for tree and linear models.
Using a public endpoint for traffic that must stay private. IAM authentication does not keep packets off the internet; use a private endpoint.
Computing history features inside the request. Querying BigQuery per request is too slow; serve precomputed features from an online store.
Check yourself
How does Feature Store deliver features to an online model?
A feature view syncs selected BigQuery features into an online store, and the app or model server reads them by entity ID at low latency.
Why can latency rise during a sudden spike even with autoscaling?
New replicas need time to start, so requests queue until they are ready; a higher minimum replica count covers known peaks.
Which endpoint type keeps prediction traffic on internal IPs?
A private endpoint, typically through Private Service Connect.
Topic 3 of 3 in Serving and scaling models
Test yourself on Serving and scaling models
The full Google Cloud Professional Machine Learning Engineer question bank has 749 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Google Cloud. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.