Implement Generative AI and Agentic Solutions · Topic 3 of 3
Optimizing and Operationalizing Generative AI
AI-103 tuning and operations: prompt engineering, model parameters, reasoning models, self-critique, tracing and token analytics, and orchestrating models.
- 4 min
- Prompt techniques: clear instructions, few-shot examples, grounding, output format, task breakdown.
- Lower temperature for consistency; cap output with max tokens.
- Reasoning models: max_completion_tokens and reasoning_effort, usually no temperature; unsupported parameters are rejected.
What it is
After an app works, it must work well and cheaply in production. This skill group covers tuning the model's behavior with prompts and parameters, improving answers with reflection and self-critique, observing systems with tracing and token analytics, and combining several models or rules.
How it works
Prompt engineering
| Technique | Purpose |
|---|---|
| Clear system instructions | Set role, task, rules and output format |
| Few-shot examples | Show the model examples of the expected input and output |
| Grounding content | Supply sources and tell the model to use only them |
| Output format | Ask for a structure, or use structured outputs with a JSON schema |
| Break down tasks | Ask for steps, or split work into several calls |
Model parameters
| Parameter | Effect |
|---|---|
| temperature | Randomness; lower is more focused and repeatable, higher is more creative |
| top_p | Limits choices to the most likely tokens; usually change this or temperature, not both |
| max_tokens / max_completion_tokens | Caps the length of the output |
| stop | Sequences that end the output |
| frequency and presence penalties | Reduce repetition |
Reasoning models work differently. On the Chat Completions API they use max_completion_tokens instead of max_tokens (the Responses API uses max_output_tokens), and the limit also covers their internal reasoning tokens. Most reasoning models do not support temperature, top_p, the penalty parameters or max_tokens, and they accept a reasoning_effort setting whose values, such as low, medium or high, vary by model. Requests that send unsupported parameters are rejected.
Reflection and self-critique
- Reflection: the model reviews its own draft against the requirements and revises it.
- Self-critique loops: a second call, or a second model, scores the answer and asks for fixes.
- Chain-of-thought evaluation: check the reasoning steps, not only the final answer, to find where logic fails.
These raise quality but add latency and cost, so use them for high-value tasks.
Observability
| Signal | Why it matters |
|---|---|
| Traces | Follow one request across model calls, tools and retrieval |
| Token analytics | See prompt and completion tokens per call to control cost |
| Latency breakdown | Find which step is slow: retrieval, model or tool |
| Safety signals | Track filtered prompts and outputs |
Tracing in Foundry is based on OpenTelemetry and is commonly sent to Application Insights.
Orchestrating several models
- Route easy requests to a small, cheap model and hard ones to a larger or reasoning model.
- Deploy model router, a single deployment that chooses an underlying model for each request in Balanced, Quality or Cost mode.
- Combine an LLM with a rules engine for decisions that must be exact, such as price calculations or compliance checks.
A worked example: a support bot is slow and expensive. Tracing shows most cost comes from long prompts to a large model. The team trims the prompt, routes simple questions to a small model, and keeps the large model for complex cases.
If a question says requests to a newly selected reasoning model fail with an error about max_tokens and temperature, replace max_tokens with max_completion_tokens and remove temperature.
How the exam asks about this
- "Answers must be consistent between runs." Lower the temperature.
- "Find which step adds most latency." Use tracing with a latency breakdown.
- "Cut cost while keeping quality for hard questions." Route simple requests to a smaller model.
What to remember for the exam
- Prompt techniques: clear instructions, few-shot examples, grounding, output format, task breakdown.
- Lower temperature for consistency; cap output with max tokens.
- Reasoning models: max_completion_tokens and reasoning_effort, usually no temperature; unsupported parameters are rejected.
- Reflection and self-critique improve quality at extra cost.
- Tracing (OpenTelemetry, Application Insights), token analytics and latency breakdowns guide tuning.
Raising temperature does not make answers more accurate. It makes them more varied.
Common mistakes
Changing temperature and top_p together. Adjust one at a time.
Sending chat-model parameters to a reasoning model. Check which parameters it supports.
Optimizing without measurements. Use traces and token analytics first.
Check yourself
Which parameter should you lower for more repeatable answers?
Temperature.
Which parameter replaces max_tokens for reasoning models?
max_completion_tokens.
What does a trace show for a generative AI request?
Each step of the request, such as model calls, retrieval and tool calls, with timings and tokens.
Topic 3 of 3 in Implement Generative AI and Agentic Solutions
Test yourself on Implement Generative AI and Agentic Solutions
The full Microsoft AI-103 question bank has 648 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Microsoft. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.