Prompts, Variants and Git Version Control

What it is

A prompt is the instruction set that shapes a model's behaviour. In production it is a versioned asset, like code: it has an owner, a history, tests and a release process. Variants are alternative versions of a prompt that you compare on the same data. Git stores them so a change is reviewed and a bad one can be reverted.

How it works

Design a prompt

Lower temperature gives more repeatable output, which suits extraction and classification. Where the output must be machine readable, use structured outputs (a JSON schema) rather than asking politely. Keep user input in a separate message from your instructions to reduce prompt injection risk, and tell the model what to do when the context lacks the answer.

Store the prompt as a file

Keep the prompt text and its settings in a file, not buried in application code. Many teams use one file per prompt, for example a .prompty file (YAML front matter for model settings, then the template with placeholders), or plain text plus a small JSON or YAML config.

prompts/
  support-answer/
    prompt.md          template with {question} and {context}
    config.yaml        model deployment, temperature, max tokens
    eval-data.jsonl    test cases
    CHANGELOG.md

The application loads the prompt by name and version at startup or build time, so changing wording is a reviewed commit, not a hot edit in a portal.

Create variants and compare them

A variant changes one thing: the wording, the examples, the output format or the model parameters. Compare variants by running them over the same test dataset and scoring the answers with the same evaluators, then comparing the results.

from azure.ai.evaluation import evaluate, GroundednessEvaluator, RelevanceEvaluator

variants = {"v1": "answers_v1.jsonl", "v2": "answers_v2.jsonl"}
for name, path in variants.items():
    evaluate(
        data=path,
        evaluators={"groundedness": GroundednessEvaluator(model_config),
                    "relevance": RelevanceEvaluator(model_config)},
        evaluation_name=f"support-answer-{name}",
        output_path=f"results_{name}.json",
    )

Each output file holds per row scores and aggregate metrics, and when you also log to a Foundry project, the runs can be viewed and compared in the portal. Compare more than average quality: look at the worst cases, token count (cost), latency and safety results. Choose the winner on evidence, then promote it.

Version control with Git

# .github/workflows/prompt-eval.yml (fragment)
on:
  pull_request:
    paths: ['prompts/**']
jobs:
  evaluate:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: actions/checkout@v4
      - uses: azure/login@v2
        with:
          client-id: ${{ vars.AZURE_CLIENT_ID }}
          tenant-id: ${{ vars.AZURE_TENANT_ID }}
          subscription-id: ${{ vars.AZURE_SUBSCRIPTION_ID }}
      - run: pip install azure-ai-evaluation
      - run: python eval/run_eval.py --prompt prompts/support-answer

The script exits with a non zero code when a metric is below its threshold, so the check blocks the merge.

A worked example

The team tests a v2 prompt that adds two worked examples and a stricter "cite the source" rule. On 200 test questions v2 raises groundedness but also adds 120 tokens per answer. They accept it after checking cost, merge the pull request, and tag support-answer-1.4.0.

How the exam asks about this

• "Know which prompt version produced yesterday's bad answers." Tag or commit pinned in the deployed app.
• "Compare two prompt rewrites fairly." Same dataset, same evaluators, one change at a time.
• "Prevent a weaker prompt reaching production." Pull request evaluation check with thresholds.