Cost-Optimized Architecture Design · Topic 3 of 8
AI and ML Cost Optimization
Cut Amazon Bedrock and SageMaker AI costs with service tiers, batch inference, prompt caching, right endpoint types and cross Region inference routing.
- 6 min
- Bedrock tiers are Reserved, Priority, Standard and Flex, chosen per request with
service_tier. - Batch inference reads from and writes to S3 and is priced below on-demand.
- Prompt caching bills cache reads at a lower rate, and the prefix must stay identical.
What it is
Generative AI and machine learning bills grow with tokens, endpoint hours and accelerator time. The same model call can cost very different amounts depending on the tier it runs in, whether it is batched, and whether part of the prompt is served from a cache.
The exam expects you to match a latency requirement to the cheapest way of meeting it.
How it works
Amazon Bedrock service tiers
Bedrock offers four service tiers for on-demand style inference. You choose the tier per request with the optional service_tier parameter.
| Tier | What it gives | Use when |
|---|---|---|
| Reserved | Reserved tokens per minute capacity for 1 or 3 months, fixed monthly price | Mission critical, steady traffic that cannot tolerate throttling |
| Priority | Fastest responses at a premium price, no reservation | Customer facing flows that need speed but not 24x7 capacity |
| Standard | Default tier with consistent performance | Everyday tasks |
| Flex | Discounted price, longer processing time | Evaluations, summarization, agent steps that can wait |
Reserved tier traffic above the reservation overflows to the Standard tier. The Priority, Standard and Flex tiers share the account's on-demand quota.
Batching strategies
Bedrock batch inference takes many prompts in files in Amazon S3, runs them asynchronously and writes the results back to S3. It is priced below on-demand for supported models, so it fits nightly classification, enrichment and summarization jobs where nobody waits for the answer. Batch inference is not supported for provisioned models, and prompt caching does not apply to the batch API.
On SageMaker AI, the batch equivalent is Batch Transform. It starts instances for the job, scores a whole dataset from S3 and then stops, so you pay nothing between jobs.
Caching patterns
Prompt caching stores a repeated prompt prefix, such as a long system prompt, tool definitions or an uploaded document. Tokens read from the cache are billed at a lower cache read rate and reduce latency. Writing to the cache can be billed above the normal input rate for some models, so caching pays off only when the prefix is reused.
Put static content first and variable content last, because any change to the prefix causes a cache miss. The cache has a time to live that resets on each hit, often five minutes, with a longer option on some models.
A second pattern is a response cache in front of the model. Store answers to repeated or semantically similar questions, for example in a vector capable store, and return them without calling the model at all.
SageMaker AI inference and training choices
| Option | Billing idea | Fits |
|---|---|---|
| Real time endpoint | Instance hours while running | Steady, low latency traffic |
| Serverless Inference | Per use, scales with traffic | Intermittent traffic that tolerates cold starts |
| Asynchronous Inference | Instance hours, can scale to zero | Large payloads, long processing, bursty queues |
| Batch Transform | Instance hours for the job only | Scoring whole datasets offline |
| Multi model endpoints | One endpoint hosts many models | Many small, rarely used models |
For training, managed Spot training runs jobs on spare capacity at a discount and uses checkpoints to resume after interruption. Savings Plans for SageMaker AI give a commitment discount on steady instance usage, and AWS Trainium and AWS Inferentia instances target lower cost training and inference.
Cross Region inference and data transfer
Cross Region inference sends Bedrock requests through an inference profile that can route them to other Regions. Geographic profiles keep processing inside a geography such as US, EU or APAC. Global profiles can use any supported commercial Region, and AWS documents approximately 10% savings for global over geographic profiles.
AWS states there is no additional routing cost for cross Region inference. The price is based on the Region where you call the profile, and data stays on the AWS network, encrypted in transit.
Data residency decides the profile. Choose geographic when regulations limit where prompts are processed, and global when cost and throughput matter more. The SCP must allow the destination Regions for geographic profiles, or the unspecified requested Region value for global ones.
Other data transfer habits matter too. Keep training data, SageMaker AI jobs and Bedrock knowledge base sources in the same Region. Reach Bedrock and SageMaker AI from private subnets through interface VPC endpoints instead of a NAT gateway.
How the exam asks about this
- "Thousands of support tickets must be summarized each night and results are needed by morning." -> Bedrock batch inference from S3, or the Flex tier.
- "A chatbot sends the same long policy document with every question." -> Prompt caching with the document placed in the static prefix.
- "A company has no data residency limits and wants lower Bedrock cost and more throughput." -> A global cross Region inference profile.
What to remember for the exam
- Bedrock tiers are Reserved, Priority, Standard and Flex, chosen per request with
service_tier. - Batch inference reads from and writes to S3 and is priced below on-demand.
- Prompt caching bills cache reads at a lower rate, and the prefix must stay identical.
- SageMaker AI Serverless Inference and Asynchronous Inference avoid paying for idle endpoints.
- Cross Region inference adds no routing cost, and pricing follows the source Region.
- Global profiles save about 10% but process data in any supported commercial Region.
Do not pick a global inference profile when the scenario states data residency or sovereignty rules. A geographic profile, or a single Region, is required.
Common mistakes
Keeping a real time endpoint up for occasional traffic. An always on instance bills every hour. Serverless or asynchronous inference fits sporadic load far better.
Expecting prompt caching to help batch jobs. Prompt caching is for on-demand inference endpoints. For batch work, the batch price itself is the saving.
Building custom cross Region failover for model capacity. Cross Region inference already routes across Regions with no extra routing charge. Custom routing adds cost and effort.
Check yourself
Which Bedrock tier suits latency tolerant model evaluations at the lowest price?
The Flex tier, which trades longer processing time for a pricing discount.
How is a cross Region inference request priced when it is processed in another Region?
At the price of the source Region where the profile was called, with no additional routing cost.
Which SageMaker AI option scores a full dataset in S3 and stops paying when the job ends?
Batch Transform, which runs instances only for the duration of the job.
Topic 3 of 8 in Cost-Optimized Architecture Design
Test yourself on Cost-Optimized Architecture Design
The full AWS SAP-C03 question bank has 850 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.