Foundation Model Integration, Data Management, and Compliance · Topic 2 of 9
Foundation Model Selection, Customization and Lifecycle
Choose FMs with benchmarks and Bedrock model evaluation, then fine tune, register, deploy, roll back and retire customized models on SageMaker AI.
- 6 min
- Select FMs on capability, context window, latency, cost, Regional availability and results on your own data.
- Bedrock model evaluation compares models with automatic, LLM as a judge or human evaluation.
- LoRA trains small adapters on a frozen base model; adapters are cheap to train, store and swap.
What it is
Choosing an FM means matching a model's capabilities, limits and cost to a specific use case, on evidence rather than reputation. A model that tops a public leaderboard may still be too slow, too expensive or weak on your data.
When prompting and RAG are not enough, you customize the model. Then the model becomes a software artifact: it needs versions, approval, safe deployment, rollback and eventually retirement.
How it works
Assessing and choosing an FM
Compare candidates on the dimensions the scenario cares about.
| Dimension | What to check |
|---|---|
| Capability | Text, image, embeddings, tool use, supported languages, reasoning strength |
| Context window | Whether the longest prompt plus retrieved context fits |
| Performance | Latency, throughput, quality on your own test set |
| Cost | Price per input and output token, or hosting cost for self hosted models |
| Limitations | Regional availability, modalities, maximum output length, customization support |
| Governance | License terms, data handling, responsible AI behavior |
Performance benchmarks such as public leaderboards give a first shortlist, but they measure generic tasks. Capability analysis checks that the model supports the features you need, such as tool use or image input. Limitation evaluation rules out models that are unavailable in your Region, cannot be customized or cannot handle your context length.
Final selection should come from Amazon Bedrock model evaluation on your own prompts. It supports automatic metrics, LLM as a judge scoring and human review, so you can compare candidates side by side.
The smallest model that meets the quality bar is usually the right answer. It is cheaper and faster, and the exam often rewards that trade.
Customization options
| Option | What changes | Use when |
|---|---|---|
| Prompt engineering | Nothing in the model | Always try first |
| RAG | Context at query time | Model needs current or private facts |
| Fine tuning | Model weights, with labeled examples | Model must follow a domain style, format or task |
| Continued pre-training | Model weights, with unlabeled domain text | Model must learn a domain's language broadly |
| Distillation | A smaller student learns from a larger teacher | Need near teacher quality at lower cost and latency |
Parameter efficient fine tuning (PEFT) trains only a small number of extra parameters and leaves the base weights frozen. Low-rank adaptation (LoRA) is the common form: it adds small low rank matrices, called adapters, to selected layers.
Adapters are small and quick to swap, so one base model can serve several domain adapters instead of one full model copy per domain.
Deploying customized models
On Amazon SageMaker AI, you deploy a fine tuned model to a real time endpoint. With inference components, several models or LoRA adapters can share the same endpoint and its accelerators, which improves utilization.
On Amazon Bedrock, fine tuned models are served with Provisioned Throughput, or with on demand inference where the model supports it. Bedrock Custom Model Import brings weights you customized elsewhere, such as on SageMaker AI, into Bedrock for supported model architectures, so the app uses the same Bedrock API.
Versioning, pipelines and rollback
SageMaker Model Registry stores models as versions inside a model package group. Each version carries metrics and an approval status: PendingManualApproval, Approved or Rejected.
An automated pipeline, built with SageMaker Pipelines or AWS CodePipeline, trains, evaluates and registers each new model. An approval status change can trigger deployment through an Amazon EventBridge rule.
For rollout, SageMaker AI deployment guardrails use blue/green deployments with all at once, canary or linear traffic shifting. You attach Amazon CloudWatch alarms, and if an alarm fires during the baking period, the endpoint rolls back automatically to the previous fleet.
| Strategy | Behavior | Choose for |
|---|---|---|
| Canary | Small share of traffic first, then the rest | Catch problems with minimal user impact |
| Linear | Traffic shifts in equal steps | Gradual confidence building |
| All at once | Full switch after the new fleet is ready | Fastest, still with alarm based rollback |
| Shadow test | New model receives a copy of traffic, users see the old one | Compare a model on live traffic with zero user risk |
Retiring and replacing models
Models age out. Amazon Bedrock marks model versions as Active, Legacy or End-of-Life (EOL), and calls to a model fail after its EOL date. Track the dates, evaluate the replacement on your test set and migrate before the deadline.
For your own models, retire old registry versions, delete unused endpoints to stop cost, and keep the approved previous version available as a rollback target.
How the exam asks about this
- "Five business units need domain behavior from one base model at low hosting cost" -> LoRA adapters served from one SageMaker AI endpoint with inference components.
- "A new model version must roll back automatically if latency degrades" -> SageMaker AI deployment guardrails with a CloudWatch alarm.
Common mistakes
Choosing by leaderboard rank only. Public benchmarks do not measure your data, cost or latency. Validate on your own dataset.
Fine tuning when RAG is needed. Weights do not stay current with changing facts. Fine tune for behavior, retrieve for knowledge.
Deploying without an approval gate. Pushing every trained model straight to production skips evaluation. Use Model Registry approval status as the gate.
A model past its EOL date stops answering. Ignoring the Legacy status notice is an outage waiting to happen.
What to remember for the exam
- Select FMs on capability, context window, latency, cost, Regional availability and results on your own data.
- Bedrock model evaluation compares models with automatic, LLM as a judge or human evaluation.
- LoRA trains small adapters on a frozen base model; adapters are cheap to train, store and swap.
- SageMaker inference components let several models or adapters share one endpoint.
- Custom Model Import brings externally customized weights of supported architectures into Bedrock.
- Model Registry: model package groups, versions, approval status as the deployment gate.
- Deployment guardrails: canary or linear shifting with automatic rollback on CloudWatch alarms.
- Bedrock model lifecycle: Active, Legacy, EOL.
Check yourself
Why is LoRA cheaper to deploy for many domains than full fine tuning?
Each domain needs only a small adapter on a shared frozen base model, so you do not host a full model copy per domain.
What controls which model version a pipeline may deploy?
The approval status in SageMaker Model Registry, where only Approved versions should trigger deployment.
How do you make a new endpoint version revert on high error rates?
Use deployment guardrails with canary or linear traffic shifting and a CloudWatch alarm that triggers automatic rollback.
Topic 2 of 9 in Foundation Model Integration, Data Management, and Compliance
Test yourself on Foundation Model Integration, Data Management, and Compliance
The full AWS AIP-C01 question bank has 827 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.