Snowflake Gen AI Governance · Topic 14 of 15
Snowflake AI Observability - Features, Evaluation Metrics and Comparisons
Evaluate GenAI apps with Snowflake AI Observability: features, evaluation metrics, run comparisons in Snowsight and the privileges needed to run them.
- 4 min
- Features: evaluations, comparisons, tracing, logging, cost metadata.
- Metrics: answer relevance, context relevance, groundedness, correctness, coherence.
- Comparison is in Snowsight under AI & ML, Evaluations, on runs sharing a dataset.
What it is
AI Observability is a set of features in Snowflake Cortex that helps you answer questions about AI workloads: what did the app do, how good were its answers, how much did it cost, and which version is better.
It covers production request traces, batch evaluations, and cost and request metadata. Native Cortex features (Cortex Agents, Snowflake Intelligence, Cortex Analyst, Cortex Search) show monitoring in Snowsight. Custom applications, for example a RAG pipeline or an agent workflow, stream telemetry through TruLens, whether they run on Snowflake or elsewhere.
How it works
Features at a glance
| Feature | What it gives you |
|---|---|
| Evaluations | Batch runs of an app over a dataset, scored with metrics |
| Comparisons | Side-by-side view of several runs of the same dataset |
| Tracing | Step-by-step record of one request, with inputs, outputs and latency per span |
| Logging | Warnings and errors from evaluation runs, plus stored events |
| Cost and request metadata | Credits, tokens, models and request IDs from Account Usage views |
Built-in AI functions and the Cortex REST API report usage only through Account Usage views, not through traces. Base spend reporting on those views, not on the event table, because trace delivery is best effort and can under-report consumption.
Evaluation metrics
A metric is a score computed for each record in a run. The server-side metrics are:
| Metric | Question it answers |
|---|---|
| Answer relevance | Does the answer address the question? |
| Context relevance | Was the retrieved context relevant to the question? |
| Groundedness | Is the answer supported by the retrieved context (low value suggests hallucination)? |
| Correctness | Does the answer match the ground truth answer in the dataset? |
| Coherence | Is the answer well formed and consistent? |
Usage cost and latency are also tracked. The default LLM judge is llama3.1-70b, and the user needs access to the judge model. You can add a custom client-side metric with a TruLens Metric object and selectors such as the record output. Metrics that need intermediate data depend on the span attributes you map, for example RETRIEVAL.QUERY_TEXT and RETRIEVAL.RETRIEVED_CONTEXTS.
LLM-judge metrics use Cortex models, so computing them needs the USE AI FUNCTIONS privilege (or USE AI FUNCTION AI_COMPLETE). They also cost tokens.
Map the symptom to the metric. Wrong or invented facts point to groundedness. Off-topic answers point to answer relevance. Good model but poor search points to context relevance.
The evaluation workflow
- Instrument the app with TruLens.
- Register it with
TruAppor a framework wrapper. Snowflake creates an External Agent object that stores only metadata. - Create a run with a dataset and a metric list (
RunConfig,add_run). - Invoke the run (
run.start()) to produce outputs and traces. - Compute metrics (
run.compute_metrics([...])) and review them. Computing is asynchronous, and a metric cannot be recomputed for the same run.
Comparing runs
Comparisons answer "did my change help?". You change a prompt, a model, a retrieval setting or a chunk size, create a new run on the same dataset, and compare.
- In Snowsight go to AI & ML, then Evaluations.
- Select the application (the External Agent).
- Select several runs that share a dataset.
- Choose Compare.
Only runs on a shared dataset can be compared meaningfully. Version names for the app and run help you tell them apart.
Privileges
| Task | Needs |
|---|---|
| Use AI Observability | SNOWFLAKE.CORTEX_USER database role |
| Register an application | CREATE EXTERNAL AGENT on the schema |
| Create and run evaluations | USAGE on the External Agent, CREATE TASK on the schema, EXECUTE TASK on the account |
| Compute metrics | USE AI FUNCTIONS, or USE AI FUNCTION AI_COMPLETE |
| Modify or drop the app | OWNERSHIP on the External Agent |
GRANT DATABASE ROLE SNOWFLAKE.CORTEX_USER TO ROLE observability_user_role;
GRANT CREATE EXTERNAL AGENT ON SCHEMA app_schema TO ROLE observability_user_role;
GRANT CREATE TASK ON SCHEMA app_schema TO ROLE observability_user_role;
GRANT EXECUTE TASK ON ACCOUNT TO ROLE observability_user_role;
GRANT USE AI FUNCTIONS ON ACCOUNT TO ROLE observability_user_role;
External Agent objects share a namespace with model objects, so an application and a model cannot have the same name in one schema.
Common mistakes
Using the event table for billing. Use Account Usage views; traces can under-report.
Comparing runs from different datasets. Comparison needs a shared dataset.
Recomputing a metric on the same run. It is not allowed; create a new run.
Forgetting CREATE TASK and EXECUTE TASK. Running evaluations needs both.
Reading groundedness as correctness. Groundedness checks support from context; correctness checks against ground truth.
What to remember for the exam
- Features: evaluations, comparisons, tracing, logging, cost metadata.
- Metrics: answer relevance, context relevance, groundedness, correctness, coherence.
- Comparison is in Snowsight under AI & ML, Evaluations, on runs sharing a dataset.
- Apps register as External Agent objects; needs CORTEX_USER and CREATE EXTERNAL AGENT.
Check yourself
Which metric flags answers not supported by the retrieved passages?
Groundedness.
What must runs share to be compared?
The same dataset.
What does registering a TruLens app create in Snowflake?
An External Agent object that stores only metadata, while traces and scores live in the event table.
Topic 14 of 15 in Snowflake Gen AI Governance
Test yourself on Snowflake Gen AI Governance
The full Snowflake GES-C02 question bank has 585 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Snowflake. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.