Test Datasets, Data Mapping and Quality Metrics

What it is

Evaluation of a generative AI app means running it over a test dataset and scoring the answers with evaluators. The azure-ai-evaluation Python package provides built in evaluators; data mapping tells each evaluator which dataset column to read. Quality evaluators such as groundedness, relevance, coherence and fluency use a judge model (an LLM) to score answers, usually on a 1 to 5 scale.

How it works

Build a good test dataset

A dataset is a JSONL file with one test case per line.

{"query": "How do I reset my VPN token?", "context": "To reset a VPN token, open the Security portal and choose Reset token.", "response": "Open the Security portal and choose Reset token.", "ground_truth": "Use Reset token in the Security portal."}

Good datasets are representative of real traffic, cover edge cases (ambiguous, out of scope, multi language), include adversarial inputs, are big enough for stable averages, and are versioned in Git or as a data asset so results stay comparable. If real data is scarce, generate queries with the simulator in the evaluation package, then have a person review them. Keep a separate golden set that you do not tune against.

If you do not have responses yet, supply a target: a function or app that the evaluation calls for each query to generate response (and context) at run time.

Data mapping

When dataset columns have different names from the evaluator inputs, map them with column_mapping. ${data.<column>} reads from the dataset, and ${target.<field>} reads from the target's output.

from azure.ai.evaluation import (evaluate, GroundednessEvaluator, RelevanceEvaluator,
                                 CoherenceEvaluator, FluencyEvaluator)

model_config = {
    "azure_endpoint": "https://<resource>.openai.azure.com",
    "azure_deployment": "<judge-deployment>",
    "api_version": "<api-version>",
}

result = evaluate(
    data="eval-data.jsonl",
    evaluators={
        "groundedness": GroundednessEvaluator(model_config),
        "relevance": RelevanceEvaluator(model_config),
        "coherence": CoherenceEvaluator(model_config),
        "fluency": FluencyEvaluator(model_config),
    },
    evaluator_config={
        "groundedness": {"column_mapping": {
            "query": "${data.question}",
            "context": "${data.retrieved}",
            "response": "${data.answer}",
        }},
        "relevance": {"column_mapping": {
            "query": "${data.question}",
            "response": "${data.answer}",
        }},
    },
    output_path="./eval-out.json",
)

Evaluators without an entry in evaluator_config use columns with matching names. The judge model is configured with model_config, so use a capable deployment as the judge and keep it the same between runs.

Foundry cloud evaluations use a different mapping syntax. Each testing criterion has type: azure_ai_evaluator, an evaluator_name such as builtin.coherence or builtin.fluency, and a data_mapping where {{item.query}} reads a dataset field and {{sample.output_text}} reads text generated during the run. Judge based evaluators also take a deployment_name parameter.

The four quality metrics

Other useful built in evaluators: retrieval (quality of the retrieved context for the query), similarity and the text overlap metrics (F1, BLEU, ROUGE, METEOR) which need ground_truth, and agent evaluators for tool use and task adherence. Results contain, per metric, a score, a pass or fail result against a threshold (default 3 on the 1 to 5 scale), and a reason from the judge. Read the reasons for low scoring rows: they show whether the retrieval or the generation is at fault.

Interpreting results

A worked example

A team has 150 test questions from support tickets. Groundedness averages 3.4. Row level reasons show the answers cite a policy not present in the retrieved context, so retrieval is changed to return more chunks. A rerun lifts groundedness to 4.3 while relevance stays the same.

How the exam asks about this

• "Detect that the bot states facts not found in the retrieved documents." Groundedness evaluator.
• "Dataset columns are named question and answer." Use column_mapping with ${data.question} and ${data.answer}.
• "Score answers against known correct text." Similarity or F1 with a ground_truth column.