Intent Resolution and Comparing Configurations in MLflow 3

What it is

An answer can be factually correct and still wrong for the user: true for another product, region, plan or version. That is a contextually mismatched response, and it almost always means the retriever fetched context for the literal words instead of what the user meant.

The fix is an intent resolution stage before retrieval. The second half of this page is how you prove a context configuration is better: compare MLflow 3 runs on every metric and across repeated runs.

How it works

Where intent is resolved

Each stage of an agent pipeline has a job. Intent has to be settled before retrieval, because retrieval is where the wrong context gets chosen.

Worked case: a user discussing their EU account asks, "What is the refund window?" Without intent resolution the retriever finds the US policy, which is correct and well grounded, but wrong for this user. An intent stage turns the query into "refund window, EU consumer accounts" using session history and profile, and retrieval filters on region.

Prompting the generator to "consider the user's region" does not fix this. The model can only use the passages it was given, and none of them were EU passages.

Typical intent work:

• Resolve references to earlier turns ("the second option", "that table").
• Pull scope from session or long-term memory (region, product, plan, role).
• Classify the task (lookup, comparison, action) to choose the retriever or tool.
• Ask a clarifying question when the scope cannot be determined.

Comparing context configurations in MLflow 3

To choose between context setups (chunk size, number of documents, memory loading, a summary in the system prompt), run each one against the same evaluation set with mlflow.genai.evaluate(), one MLflow run per configuration, in one experiment.

import mlflow
from mlflow.genai.scorers import Correctness, RetrievalGroundedness
for name, app in configs.items():
    with mlflow.start_run(run_name=name):
        mlflow.genai.evaluate(data=eval_set, predict_fn=app,
                              scorers=[Correctness(), RetrievalGroundedness()])

Then compare the runs in the MLflow UI. The parallel coordinates chart shows each run as a line across every metric axis, so a configuration that stays high on all axes stands out, and one with a single low metric shows a dip.

What "most reliable" means

Reliable means two things: good on every metric the task cares about, and stable across repeated runs. A single best number, or the best average, can hide a weak metric or a lucky run.

Worked numbers, three repeats each, scores as fractions:

A has the highest correctness but weak groundedness, so it is answering from model knowledge, not the context. C has the highest single values but swings widely between runs. B is the most reliable: high on all three metrics with low variance.

LLM judges and agents are not deterministic, so run each configuration more than once before drawing a conclusion from a small gap.

How the exam asks about this

• "Answers are accurate and grounded but often describe the wrong product line the user is asking about." Add intent resolution before retrieval, then filter retrieval by the resolved scope.
• "Pick the configuration that scores well on correctness, faithfulness and recall with none lagging." Compare runs on all metrics, for example with the parallel coordinates chart, not by mean or top value.
• "One configuration won a single run by two points." Repeat the runs and check variance before choosing it.