Implementation and Integration · Topic 4 of 8

FM Deployment Options and LLM Hosting Challenges

Choose Bedrock on demand, Provisioned Throughput or SageMaker AI endpoints for an FM, and handle LLM needs for GPU memory, tokens and model loading.

  • 5 min
In a nutshell
  • Bedrock on demand is pay per token with no infrastructure; Provisioned Throughput gives dedicated capacity for steady volume.
  • Lambda is the usual on demand invocation layer in front of Bedrock.
  • SageMaker AI endpoints host open weight and custom models with full control; hybrid designs mix Bedrock and SageMaker AI.

What it is

Deploying an FM means deciding where inference runs and how capacity is paid for. On AWS the choice is mainly between Amazon Bedrock, which serves models through an API, and Amazon SageMaker AI, where you host a model on instances you choose.

LLMs also break assumptions from classic ML hosting. They are large, they need GPU memory, and their cost and latency depend on tokens, not on request count.

ApplicationAPI callAmazon Bedrockon demand or Provisioned ThroughputApplicationHTTPSSageMaker AI endpointyour model on GPU instances

How it works

Choosing the deployment option

OptionCapacity modelPick it when
Bedrock on demandPay per token, shared capacity, quotas per accountVariable or unpredictable traffic, fastest start, no infrastructure
Bedrock cross Region inferenceOn demand, spread across Regions through an inference profileOn demand traffic is throttled at peaks
Bedrock Provisioned ThroughputDedicated model units billed hourly, with no commitment or a 1 or 6 month termSteady high volume needing guaranteed throughput; common for serving custom models
SageMaker AI real time endpointInstances you choose, always onOpen weight or custom models, full control of container, instance and scaling
SageMaker AI asynchronous inferenceQueued requests, can scale to zeroLong running or large payload requests that do not need an instant answer
SageMaker AI batch transformJob over a datasetOffline scoring of many records

Lambda for on demand invocation is the usual pattern in front of Bedrock. The function receives a request, calls Bedrock, and returns the result; there is no idle compute when traffic is zero.

Event or API callinvokeLambdaon demandInvokeModel or ConverseAmazon BedrockresponseLambdareturnCaller

A hybrid solution mixes the two. Bedrock serves general tasks, while a SageMaker AI endpoint hosts a specialized or fine tuned open weight model; the application or a router calls whichever fits the task.

Exam tip

"No infrastructure to manage" and "unpredictable traffic" point to Bedrock on demand. "Guaranteed throughput for a steady workload" points to Provisioned Throughput. "Our own fine tuned open source model with a custom container" points to SageMaker AI.

Why LLM deployment is different

ChallengeClassic ML modelLLM
SizeMegabytes, fits on one CPU instanceBillions of parameters, may need several GPUs
MemoryWeights onlyWeights plus a key value cache that grows with context length and concurrent requests
LatencyOne fixed computation per requestTime to first token plus time per output token
ThroughputRequests per secondTokens per second; batching many sequences matters
StartupSecondsLoading large weights can take minutes

Container based deployment patterns

On SageMaker AI, LLMs usually run in Large Model Inference (LMI) containers, which package inference engines such as vLLM and TensorRT-LLM. The configuration controls how the model uses GPU memory and how it batches tokens.

  • Tensor parallelism splits a model too large for one GPU across several GPUs on the instance.
  • Continuous batching adds new requests to the running batch as others finish, which keeps the GPU busy and raises tokens per second.
  • Quantization stores weights in lower precision, which cuts memory so a model fits on a smaller or cheaper instance, with some possible quality loss.
  • Maximum tokens and concurrency settings cap how much key value cache memory requests can use, so the container does not run out of GPU memory under load.

Inference components let one SageMaker AI endpoint host several models, each with its own share of accelerators and its own scaling.

Model loading strategies

Large weights make cold starts slow. Specialized loading strategies reduce that time.

Model weightsAmazon S3download and loadGPU memoryinstancewarmEndpoint readyserving tokens
  • Store weights in Amazon S3 close to the endpoint and load them in parallel or streamed, rather than baking huge files into the container image.
  • Keep a minimum instance count so scale out does not start from zero during traffic.
  • SageMaker AI offers Fast Model Loader, which streams weights from Amazon S3 straight to the accelerators to cut load time for large models.
  • Scale on a metric that reflects load for LLMs, such as concurrent requests per model copy, not CPU.
Watch out

CPU utilization is a poor scaling signal for GPU bound LLM endpoints. A scenario where scaling "does not react while latency rises" usually needs a concurrency or GPU based metric.

How the exam asks about this

  • "A startup needs a chatbot quickly with unpredictable traffic and no servers" -> Lambda calling Bedrock on demand.
  • "A steady, high volume workload hits on demand throttling and needs predictable throughput" -> Bedrock Provisioned Throughput (or cross Region inference if a commitment is not wanted).
  • "A 70 billion parameter open weight model runs out of GPU memory on one GPU" -> an LMI container with tensor parallelism across GPUs, or quantization.

Common mistakes

Buying Provisioned Throughput for spiky traffic. Dedicated units are paid whether used or not. On demand fits uneven load.

Treating an LLM like a small model. Sizing by request count ignores tokens, context length and the key value cache, so endpoints run out of memory under real prompts.

Scaling from zero on a large model. Loading weights takes time. Keep warm capacity for latency sensitive traffic.

Using SageMaker AI serverless inference for LLMs. Serverless inference does not provide GPUs, so it does not fit large generative models.

What to remember for the exam

  • Bedrock on demand is pay per token with no infrastructure; Provisioned Throughput gives dedicated capacity for steady volume.
  • Lambda is the usual on demand invocation layer in front of Bedrock.
  • SageMaker AI endpoints host open weight and custom models with full control; hybrid designs mix Bedrock and SageMaker AI.
  • LLM hosting is limited by GPU memory, key value cache and tokens per second.
  • Tensor parallelism, quantization and continuous batching make large models fit and perform.
  • Fast weight loading and warm capacity reduce cold start time.

Check yourself

  • Which Bedrock option fits a steady, high volume workload that needs guaranteed throughput?

    Provisioned Throughput, which reserves dedicated model units for that model.

  • What lets a model too large for one GPU run on a multi GPU instance?

    Tensor parallelism in an LMI container, which splits the model across the GPUs.

  • Why is CPU a poor autoscaling metric for an LLM endpoint?

    The work is GPU and token bound, so CPU stays low while latency rises; use concurrency or GPU based metrics.

Topic 4 of 8 in Implementation and Integration

Test yourself on Implementation and Integration

The full AWS AIP-C01 question bank has 827 questions, with timed practice exams and an explanation for every answer.

This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.