Techniques to improve gen AI model output · Topic 5 of 5
Sampling Parameters and Settings
How token count, temperature, top-p (nucleus sampling), safety settings and output length control gen AI model behavior, with settings for common business tasks.
- 3 min
- Models generate one token at a time from probabilities.
- Temperature: randomness. Low = predictable; high = creative.
- Top-p: sample from the smallest set of tokens whose probabilities add up to p.
What it is
Besides the prompt, a gen AI model's behavior can be controlled with parameters set on each request. The exam names token count, temperature, top-p (nucleus sampling), safety settings and output length. These decide how predictable or creative the output is, how long it is, and what content is blocked.
How it works
How models pick words
A language model generates text one token at a time. A token is a piece of text, often a word or part of a word. For each step, the model calculates the probability of every possible next token. Sampling parameters decide how the next token is chosen from those probabilities.
The parameters
| Parameter | What it controls | Low value | High value |
|---|---|---|---|
| Temperature | How random the choice is | Focused, predictable, repeatable answers | More varied, creative, surprising answers |
| Top-p (nucleus sampling) | Chooses only from the smallest set of tokens whose probabilities add up to p | Only the most likely tokens are considered | A wider range of tokens is considered |
| Token count | How many tokens the request and response use; models have limits, and cost is often based on tokens | Fewer tokens, lower cost | More context and output, higher cost |
| Output length (maximum output tokens) | The longest response allowed | Short, concise answers; may cut off | Longer answers allowed |
| Safety settings | Thresholds for blocking content in categories such as harassment, hate speech, sexually explicit and dangerous content | Block more (stricter) | Block less (more permissive) |
Some platforms also offer top-k, which limits the choice to the k most likely tokens.
Factual, consistent tasks want low temperature. Brainstorming and creative writing want higher temperature.
Settings for common tasks
| Task | Suggested direction |
|---|---|
| Extracting fields from invoices | Low temperature, short output length |
| Answering policy questions from grounded sources | Low temperature, moderate length |
| Writing marketing slogans | Higher temperature, possibly higher top-p |
| Summarizing for executives | Low to moderate temperature, strict output length |
| A public-facing chatbot | Strict safety settings |
These are directions, not exact numbers. Teams test settings against their evaluation set.
Tokens and cost
Because usage is commonly billed by tokens, long prompts (for example, many grounding passages) and long outputs both raise cost. Controlling output length and keeping prompts focused are simple ways to manage spending at scale.
A worked scenario
A bank runs two gen AI features. Its compliance answer tool uses low temperature so the same question gets the same, precise answer, a maximum output length to keep answers short, and strict safety settings. Its marketing team's slogan generator uses higher temperature and top-p to produce varied ideas, with humans choosing the final slogan.
How the exam asks about this
- "Make outputs more consistent and less random." Lower the temperature.
- "Generate more creative and varied ideas." Raise the temperature (or top-p).
- "Responses are too long and costly." Reduce the maximum output length.
- "Block harmful content in a public chatbot." Safety settings.
- "Sample only from tokens that together reach a probability threshold." Top-p.
What to remember for the exam
- Models generate one token at a time from probabilities.
- Temperature: randomness. Low = predictable; high = creative.
- Top-p: sample from the smallest set of tokens whose probabilities add up to p.
- Token count and output length affect completeness and cost.
- Safety settings set thresholds for blocking harmful content categories.
Raising temperature does not make a model more accurate. For factual tasks, lower temperature plus grounding is the safer combination.
Common mistakes
Using high temperature for factual answers. It increases variation and the chance of errors.
Confusing top-p with top-k. Top-p uses a cumulative probability threshold; top-k uses a fixed number of tokens.
Ignoring output length for cost. Long outputs cost more at scale.
Check yourself
What does lowering the temperature do?
It makes outputs more focused, predictable and consistent.
What is top-p sampling?
Choosing the next token only from the smallest set of tokens whose probabilities add up to the value p.
Which setting blocks categories such as hate speech or dangerous content?
Safety settings.
Topic 5 of 5 in Techniques to improve gen AI model output
Test yourself on Techniques to improve gen AI model output
The full Google Cloud Generative AI Leader question bank has 512 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Google Cloud. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.