System Prompt and Instruction Design · Topic 2 of 2
Few-Shot Examples and System Prompt Tuning
Databricks Context Engineer: choose few-shot examples by marginal value, fix a miscalibrated system prompt cheaply, and judge prompt runs in MLflow.
- 5 min
- Include examples by marginal contribution: untested tool paths, output structure, ambiguous inputs.
- Exclude near duplicates and cases the agent already handles well.
- Fix a miscalibrated prompt by editing the instruction that causes the failure, not by a rewrite.
What it is
The system prompt is the fixed part of an agent's context: role, rules, tool guidance, output format and any few-shot examples. It is paid for on every call, so each element must earn its tokens.
The exam tests three skills here. You choose which few-shot examples fit a token budget, you fix a miscalibrated prompt with the smallest change that resolves the failure, and you read MLflow experiment results to decide whether a longer prompt is worth its cost.
How it works
Choosing few-shot examples by marginal value
An example is worth including when it teaches something the rest of the prompt does not. Judge each candidate by its marginal contribution: what the agent would get wrong without it.
| Candidate example | Marginal value | Decision |
|---|---|---|
| Uses a tool path no other example or instruction covers | High: the agent otherwise skips or misuses that tool | Include |
| Shows the exact output structure downstream code parses | High: fixes format errors | Include one |
| Shows how to handle an ambiguous request, for example by asking a clarifying question | High: prevents guessing | Include |
| Near duplicate of an example already included | Near zero | Exclude |
| Common case the agent already handles well in evaluation | Low | Exclude |
| Very long example of a rare edge case | Low per token | Exclude or shorten |
Worked example: the budget for examples is 1,500 tokens. Candidates are A (new tool path, 450 tokens), B (output format, 300), C (ambiguous input, 400), D (near duplicate of A, 450) and E (common case, 350). A, B and C cover three distinct gaps in 1,150 tokens; adding D would exceed the budget, and E would use the last 350 tokens while teaching nothing new.
Targeted revisions for a miscalibrated prompt
Miscalibration means the prompt pushes behavior too far one way. The cheapest fix changes the specific instruction that causes the failure, rather than rewriting the prompt or adding a long list of examples.
| Observed failure pattern | Targeted revision |
|---|---|
| Refuses valid requests because a restriction is too broad | Narrow the rule's scope to the cases it was meant for |
| Calls a tool on every turn, even when it already has the answer | State when to call the tool and when not to |
| Output breaks a parser | Specify the exact structure, or add one format example |
| Follows the wrong rule when two instructions conflict | Remove or reconcile the conflict, and state which rule wins |
| Long answers where short ones are wanted | Add a clear length or format rule |
| Ignores one rule buried in a long prompt | Move it near the start or end and state it once |
Rewriting the whole prompt adds risk: behaviors that worked can break, and the change is hard to review.
Judging a higher-token configuration
Compare configurations as MLflow runs on the same evaluation set with the same scorers, for example with mlflow.genai.evaluate(). Look at all metrics and at repeated runs, not at one headline number.
| Configuration | Prompt tokens | Correctness | Groundedness | Format pass rate |
|---|---|---|---|---|
| A: base prompt | 1,200 | 0.81 | 0.88 | 0.72 |
| B: base plus examples plus policy text | 3,000 | 0.84 | 0.88 | 0.95 |
B costs 1,800 more tokens per call, an extra 18 million input tokens across 10,000 calls. Correctness rose only 0.03, which can be run to run noise, but the format pass rate rose 0.23.
To find the element driving the trade-off, remove one element at a time and rerun. If A plus only the format example (300 tokens) reaches a 0.94 format pass rate, the example drives the gain and the other 1,500 added tokens are not justified.
The higher-token configuration is justified only when the gain is real across repeated runs, matters for the stated goal, and cannot be reached with a smaller subset of the added text.
How the exam asks about this
- "Two candidate examples show the same tool call with different wording; the budget fits only one more." Exclude the near duplicate and keep an example that covers an untested path.
- "The agent refuses questions about refunds because the prompt says never to discuss payments." Narrow the restriction to the case it was meant for.
- "The 3,000-token prompt beats the 1,200-token prompt by 0.02 on correctness only." Not justified on that evidence; rerun several times and test which element, if any, causes the gain.
What to remember for the exam
- Include examples by marginal contribution: untested tool paths, output structure, ambiguous inputs.
- Exclude near duplicates and cases the agent already handles well.
- Fix a miscalibrated prompt by editing the instruction that causes the failure, not by a rewrite.
- Compare prompt configurations on the same data and scorers, on all metrics, across repeated runs; remove one element at a time to find the driver.
- A longer prompt is justified only by a real, goal relevant gain that a smaller change cannot reach.
A small gain on one metric from a single run is not evidence. Run variance can be larger than the difference, so repeat the runs before paying for extra tokens on every call.
Common mistakes
Adding more examples to fix every failure. Examples cost tokens on every call and can over fit the agent to their wording. When one rule is wrong, edit the rule.
Keeping examples that look good. An example that shows a case the agent already handles adds cost without changing behavior.
Crediting the whole prompt. When a longer configuration wins, usually one element causes the gain. Ablation finds it so you keep that element and drop the rest.
Check yourself
Which few-shot example adds the most value under a tight budget?
One that covers a gap nothing else in the prompt covers, such as an untested tool path, the required output structure or an ambiguous input.
An agent over refuses because of one broad rule. What is the least costly fix?
Narrow the scope of that rule to the cases it was meant for, then rerun the same evaluation set to confirm.
How do you identify the prompt element behind a cost and quality trade-off?
Remove one added element at a time, rerun the evaluation in MLflow, and see which removal loses the gain.
Topic 2 of 2 in System Prompt and Instruction Design
Test yourself on System Prompt and Instruction Design
The full Databricks Context Engineer Associate question bank has 600 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Databricks. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.