Implementation and Integration · Topic 4 of 8
FM Deployment Options and LLM Hosting Challenges
Choose Bedrock on demand, Provisioned Throughput or SageMaker AI endpoints for an FM, and handle LLM needs for GPU memory, tokens and model loading.
- 5 min
- Bedrock on demand is pay per token with no infrastructure; Provisioned Throughput gives dedicated capacity for steady volume.
- Lambda is the usual on demand invocation layer in front of Bedrock.
- SageMaker AI endpoints host open weight and custom models with full control; hybrid designs mix Bedrock and SageMaker AI.
What it is
Deploying an FM means deciding where inference runs and how capacity is paid for. On AWS the choice is mainly between Amazon Bedrock, which serves models through an API, and Amazon SageMaker AI, where you host a model on instances you choose.
LLMs also break assumptions from classic ML hosting. They are large, they need GPU memory, and their cost and latency depend on tokens, not on request count.
How it works
Choosing the deployment option
| Option | Capacity model | Pick it when |
|---|---|---|
| Bedrock on demand | Pay per token, shared capacity, quotas per account | Variable or unpredictable traffic, fastest start, no infrastructure |
| Bedrock cross Region inference | On demand, spread across Regions through an inference profile | On demand traffic is throttled at peaks |
| Bedrock Provisioned Throughput | Dedicated model units billed hourly, with no commitment or a 1 or 6 month term | Steady high volume needing guaranteed throughput; common for serving custom models |
| SageMaker AI real time endpoint | Instances you choose, always on | Open weight or custom models, full control of container, instance and scaling |
| SageMaker AI asynchronous inference | Queued requests, can scale to zero | Long running or large payload requests that do not need an instant answer |
| SageMaker AI batch transform | Job over a dataset | Offline scoring of many records |
Lambda for on demand invocation is the usual pattern in front of Bedrock. The function receives a request, calls Bedrock, and returns the result; there is no idle compute when traffic is zero.
A hybrid solution mixes the two. Bedrock serves general tasks, while a SageMaker AI endpoint hosts a specialized or fine tuned open weight model; the application or a router calls whichever fits the task.
"No infrastructure to manage" and "unpredictable traffic" point to Bedrock on demand. "Guaranteed throughput for a steady workload" points to Provisioned Throughput. "Our own fine tuned open source model with a custom container" points to SageMaker AI.
Why LLM deployment is different
| Challenge | Classic ML model | LLM |
|---|---|---|
| Size | Megabytes, fits on one CPU instance | Billions of parameters, may need several GPUs |
| Memory | Weights only | Weights plus a key value cache that grows with context length and concurrent requests |
| Latency | One fixed computation per request | Time to first token plus time per output token |
| Throughput | Requests per second | Tokens per second; batching many sequences matters |
| Startup | Seconds | Loading large weights can take minutes |
Container based deployment patterns
On SageMaker AI, LLMs usually run in Large Model Inference (LMI) containers, which package inference engines such as vLLM and TensorRT-LLM. The configuration controls how the model uses GPU memory and how it batches tokens.
- Tensor parallelism splits a model too large for one GPU across several GPUs on the instance.
- Continuous batching adds new requests to the running batch as others finish, which keeps the GPU busy and raises tokens per second.
- Quantization stores weights in lower precision, which cuts memory so a model fits on a smaller or cheaper instance, with some possible quality loss.
- Maximum tokens and concurrency settings cap how much key value cache memory requests can use, so the container does not run out of GPU memory under load.
Inference components let one SageMaker AI endpoint host several models, each with its own share of accelerators and its own scaling.
Model loading strategies
Large weights make cold starts slow. Specialized loading strategies reduce that time.
- Store weights in Amazon S3 close to the endpoint and load them in parallel or streamed, rather than baking huge files into the container image.
- Keep a minimum instance count so scale out does not start from zero during traffic.
- SageMaker AI offers Fast Model Loader, which streams weights from Amazon S3 straight to the accelerators to cut load time for large models.
- Scale on a metric that reflects load for LLMs, such as concurrent requests per model copy, not CPU.
CPU utilization is a poor scaling signal for GPU bound LLM endpoints. A scenario where scaling "does not react while latency rises" usually needs a concurrency or GPU based metric.
How the exam asks about this
- "A startup needs a chatbot quickly with unpredictable traffic and no servers" -> Lambda calling Bedrock on demand.
- "A steady, high volume workload hits on demand throttling and needs predictable throughput" -> Bedrock Provisioned Throughput (or cross Region inference if a commitment is not wanted).
- "A 70 billion parameter open weight model runs out of GPU memory on one GPU" -> an LMI container with tensor parallelism across GPUs, or quantization.
Common mistakes
Buying Provisioned Throughput for spiky traffic. Dedicated units are paid whether used or not. On demand fits uneven load.
Treating an LLM like a small model. Sizing by request count ignores tokens, context length and the key value cache, so endpoints run out of memory under real prompts.
Scaling from zero on a large model. Loading weights takes time. Keep warm capacity for latency sensitive traffic.
Using SageMaker AI serverless inference for LLMs. Serverless inference does not provide GPUs, so it does not fit large generative models.
What to remember for the exam
- Bedrock on demand is pay per token with no infrastructure; Provisioned Throughput gives dedicated capacity for steady volume.
- Lambda is the usual on demand invocation layer in front of Bedrock.
- SageMaker AI endpoints host open weight and custom models with full control; hybrid designs mix Bedrock and SageMaker AI.
- LLM hosting is limited by GPU memory, key value cache and tokens per second.
- Tensor parallelism, quantization and continuous batching make large models fit and perform.
- Fast weight loading and warm capacity reduce cold start time.
Check yourself
Which Bedrock option fits a steady, high volume workload that needs guaranteed throughput?
Provisioned Throughput, which reserves dedicated model units for that model.
What lets a model too large for one GPU run on a multi GPU instance?
Tensor parallelism in an LMI container, which splits the model across the GPUs.
Why is CPU a poor autoscaling metric for an LLM endpoint?
The work is GPU and token bound, so CPU stays low while latency rises; use concurrency or GPU based metrics.
Topic 4 of 8 in Implementation and Integration
Test yourself on Implementation and Integration
The full AWS AIP-C01 question bank has 827 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.