Retrieval Failure Modes and Authoritative Source Governance

What it is

A retrieval failure is a wrong answer caused by what reached the context, not by the model's reasoning. MLflow 3 traces record which documents and tables each step retrieved, and evaluation scorers say whether that context was relevant and whether the answer was grounded in it.

Unity Catalog metadata then tells you where the retrieved data came from: its owner, comments, tags and lineage. Governance actions in Unity Catalog, such as grants, index scope and tags, fix the failure at its source rather than in the prompt.

How it works

Reading the evaluation logs

Run mlflow.genai.evaluate() over an evaluation set with retrieval scorers, then open the traces of the failing rows. The pattern of scores separates the failure modes.

import mlflow
from mlflow.genai.scorers import (
    Correctness, RetrievalRelevance, RetrievalGroundedness)
results = mlflow.genai.evaluate(
    data=eval_df, predict_fn=agent_answer,
    scorers=[RetrievalRelevance(), RetrievalGroundedness(), Correctness()])

The second row is the one this objective targets. The answer faithfully repeats the context, and the context itself is the problem.

Using UC metadata to name the cause

Once a trace names the table or index that supplied a bad fact, Unity Catalog shows why it was wrong.

Governance actions that fix it

The fix should act where the agent's access is defined. Telling the model "prefer official tables" in a prompt is weaker, because the derived table is still reachable and will still be retrieved.

Designing the retrieval space before deployment

1. Identify authoritative assets. Use lineage, owners and comments to find the governed gold tables and approved document volumes, and the derived copies built from them.
2. Mark them. Apply a governed tag to authoritative assets and describe each in its comment.
3. Grant by least privilege. The agent runs as an identity, often a service principal. Grant it USE CATALOG, USE SCHEMA and SELECT only on the authoritative schema, tables and indexes.
4. Scope every retrieval channel. Build AI Search indexes only from authoritative source tables, and add only authoritative tables and views to Genie spaces.
5. Verify. Run an MLflow evaluation and inspect traces to confirm no derived source appears in retrieved context.

GRANT USE CATALOG ON CATALOG finance TO `<agent-sp-application-id>`;
GRANT USE SCHEMA ON SCHEMA finance.gold TO `<agent-sp-application-id>`;
GRANT SELECT ON SCHEMA finance.gold TO `<agent-sp-application-id>`;

A service principal is named by its application ID in GRANT statements. Putting authoritative assets in their own schema makes the grant simple and keeps new derived tables out by default.

How the exam asks about this

• "Retrieval relevance is high but answers quote last quarter's prices; the trace shows a table whose lineage points to the gold price table." The source is a derived copy; revoke the agent's access to it and keep the gold table.
• "Before launch, the catalog holds gold tables, analyst extracts and sandbox copies." Put authoritative assets in a dedicated schema, grant the agent identity only that, and index only from it.
• "Answers are relevant and grounded in the context but still wrong." The context source is the problem, not the model; check the source's UC metadata.