Evaluation and Monitoring

What this chapter covers

Evaluation and Monitoring is 12% of the Databricks Generative AI Engineer Associate exam. It covers how you prove an agent is good before release and how you watch its quality, usage and cost once it is live.

The chapter has two parts: offline evaluation with MLflow scorers, tracing, LLM judges, custom scorers and expert feedback, including choosing a model by its metrics; and production monitoring with inference tables, Agent Monitoring, AI Gateway usage tables and rate limits, plus the levers that control LLM costs.

Questions are mostly short scenarios in which the stated requirement (latency, cost, freshness, who may access what, quality signal) decides the answer. They are single-answer and some multiple-answer multiple choice, and short Python or SQL is shown as text. There are no exhibits.

How the topics fit together

Evaluation scores the agent against a dataset before release, with expert labels supplying the ground truth. After deployment, inference tables, Agent Monitoring and AI Gateway show quality, usage and cost. Weak production answers flow back into the evaluation dataset.

• Judge questions ask which judge fits and whether it needs ground truth.
• Scorer questions ask when to write a custom code scorer.
• Feedback questions ask how experts review answers and how their labels are used.
• Monitoring questions ask which table or feature tracks a live endpoint.
• Cost questions ask which feature caps or reduces LLM spending.

How to use these pages

Each topic page has the same shape, so you can skim for what you need.

• What it is: the idea in plain words.
• How it works: the mechanism, with tables and examples.
• What to remember for the exam: the facts to memorize and how questions are phrased.
• Common mistakes: the classic traps.
• Check yourself: three questions to test your understanding.