Applications of Foundation Models · Topic 1 of 7
FM Selection and Inference Settings
Pick a foundation model for an application using cost, modality, latency and customisation needs, and tune output with temperature, top p and top k.
- 4 min
- Choose a model by cost, modality, latency, language support, size, complexity, customisation and context length.
- Inference parameters change the response without changing the model.
- Temperature controls randomness: low for steady answers, high for creative ones.
What it is
When you build an application on a foundation model (FM), two design choices matter early. You choose which model to use, and you choose the inference settings that shape how it answers.
The model decides what is possible and what it costs. The settings decide how creative, long and predictable each response is.
How it works
Selection criteria
Use these criteria to narrow the choice. They build on the model selection factors from the GenAI chapter.
| Criterion | What to check |
|---|---|
| Cost | Price per request or per token, and total expected volume |
| Modality | Text, image, audio or a mix of inputs and outputs |
| Latency | How fast the answer must arrive for the user |
| Multi lingual support | The languages the application must handle |
| Model size | Larger models are often stronger but slower and costlier |
| Model complexity | Whether the task needs deep reasoning or a simple answer |
| Customisation | Whether you can fine tune or ground the model with your data |
| Input and output length | The context window against your document size |
| Provider and licence | Terms of use, and where the model is offered |
There is no single best model. A fast, cheap model can be right for short classification, while a larger model fits long document analysis.
Inference parameters
Inference parameters are settings you pass with each request. They change how the model picks the next token, without changing the model itself.
| Parameter | Effect |
|---|---|
| Temperature | Higher values make output more random and creative, and lower values make it steadier |
| Top p | Limits choices to the smallest group of tokens whose probabilities add up to p |
| Top k | Limits choices to the k most likely tokens |
| Maximum tokens (response length) | Caps how long the answer can be |
| Stop sequences | Text that tells the model to stop generating |
For factual or structured tasks, use a lower temperature. For brainstorming or creative writing, a higher temperature gives more variety.
How temperature, top p and top k relate
All three reduce or widen the pool of tokens the model may choose from. A low top k or low top p narrows the pool and gives safer answers. A high value widens it and gives more varied answers.
Cost and length
The response length setting also affects cost and delay. A cap on maximum tokens keeps answers short and controls spend, but a cap that is too low can cut an answer off.
Testing choices
Try several models and settings on the same set of real prompts, then compare quality, delay and cost. Amazon Bedrock playgrounds and model evaluation help with this comparison.
Pre trained model sources
On AWS, pre trained models are available through Amazon Bedrock as a managed API. Amazon SageMaker JumpStart provides models you can deploy to your own endpoints. Choose Bedrock for convenience and JumpStart for more control over the hosting.
What to remember for the exam
- Choose a model by cost, modality, latency, language support, size, complexity, customisation and context length.
- Inference parameters change the response without changing the model.
- Temperature controls randomness: low for steady answers, high for creative ones.
- Top k and top p limit the pool of tokens the model can choose from.
- Maximum tokens caps response length and affects cost.
- Stop sequences tell the model where to stop.
- A larger model is not always better; match the model to the task.
- Compare candidate models on your own prompts before deciding.
If the question wants more consistent, predictable output, lower the temperature. If it wants more varied or creative output, raise it.
Common mistakes
Believing temperature changes what the model knows. It only changes how random the choice of words is.
Picking the most powerful model by default. It may be too slow or costly for a simple task.
Setting a response cap too low. The answer may stop in the middle of a sentence.
Confusing top k and top p. Top k counts a fixed number of likely tokens. Top p uses a probability total.
Ignoring context length. A long document will not work in a model with a small window.
Check yourself
Which setting makes output more random and creative?
A higher temperature.
What does the maximum tokens parameter control?
The longest response the model may produce.
Name two selection criteria for a foundation model.
Examples are cost, latency, modality, language support, model size and customisation options.
Topic 1 of 7 in Applications of Foundation Models
Test yourself on Applications of Foundation Models
The full AWS AIF-C01 question bank has 849 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.