Snowflake Gen AI Governance · Topic 14 of 15

Snowflake AI Observability - Features, Evaluation Metrics and Comparisons

Evaluate GenAI apps with Snowflake AI Observability: features, evaluation metrics, run comparisons in Snowsight and the privileges needed to run them.

  • 4 min
In a nutshell
  • Features: evaluations, comparisons, tracing, logging, cost metadata.
  • Metrics: answer relevance, context relevance, groundedness, correctness, coherence.
  • Comparison is in Snowsight under AI & ML, Evaluations, on runs sharing a dataset.

What it is

AI Observability is a set of features in Snowflake Cortex that helps you answer questions about AI workloads: what did the app do, how good were its answers, how much did it cost, and which version is better.

It covers production request traces, batch evaluations, and cost and request metadata. Native Cortex features (Cortex Agents, Snowflake Intelligence, Cortex Analyst, Cortex Search) show monitoring in Snowsight. Custom applications, for example a RAG pipeline or an agent workflow, stream telemetry through TruLens, whether they run on Snowflake or elsewhere.

GenAI appcustom or nativeinstrumentedTracesevent tablescored byEvaluation runmetricsviewed inSnowsightAI and ML, Evaluations

How it works

Features at a glance

FeatureWhat it gives you
EvaluationsBatch runs of an app over a dataset, scored with metrics
ComparisonsSide-by-side view of several runs of the same dataset
TracingStep-by-step record of one request, with inputs, outputs and latency per span
LoggingWarnings and errors from evaluation runs, plus stored events
Cost and request metadataCredits, tokens, models and request IDs from Account Usage views

Built-in AI functions and the Cortex REST API report usage only through Account Usage views, not through traces. Base spend reporting on those views, not on the event table, because trace delivery is best effort and can under-report consumption.

Evaluation metrics

A metric is a score computed for each record in a run. The server-side metrics are:

MetricQuestion it answers
Answer relevanceDoes the answer address the question?
Context relevanceWas the retrieved context relevant to the question?
GroundednessIs the answer supported by the retrieved context (low value suggests hallucination)?
CorrectnessDoes the answer match the ground truth answer in the dataset?
CoherenceIs the answer well formed and consistent?

Usage cost and latency are also tracked. The default LLM judge is llama3.1-70b, and the user needs access to the judge model. You can add a custom client-side metric with a TruLens Metric object and selectors such as the record output. Metrics that need intermediate data depend on the span attributes you map, for example RETRIEVAL.QUERY_TEXT and RETRIEVAL.RETRIEVED_CONTEXTS.

LLM-judge metrics use Cortex models, so computing them needs the USE AI FUNCTIONS privilege (or USE AI FUNCTION AI_COMPLETE). They also cost tokens.

Exam tip

Map the symptom to the metric. Wrong or invented facts point to groundedness. Off-topic answers point to answer relevance. Good model but poor search points to context relevance.

The evaluation workflow

Instrument appTruLensregisterTruAppExternal AgentcreateRundataset and metricsinvokeTracesgeneratedcomputeMetrics
  1. Instrument the app with TruLens.
  2. Register it with TruApp or a framework wrapper. Snowflake creates an External Agent object that stores only metadata.
  3. Create a run with a dataset and a metric list (RunConfig, add_run).
  4. Invoke the run (run.start()) to produce outputs and traces.
  5. Compute metrics (run.compute_metrics([...])) and review them. Computing is asynchronous, and a metric cannot be recomputed for the same run.

Comparing runs

Comparisons answer "did my change help?". You change a prompt, a model, a retrieval setting or a chunk size, create a new run on the same dataset, and compare.

  1. In Snowsight go to AI & ML, then Evaluations.
  2. Select the application (the External Agent).
  3. Select several runs that share a dataset.
  4. Choose Compare.

Only runs on a shared dataset can be compared meaningfully. Version names for the app and run help you tell them apart.

Privileges

TaskNeeds
Use AI ObservabilitySNOWFLAKE.CORTEX_USER database role
Register an applicationCREATE EXTERNAL AGENT on the schema
Create and run evaluationsUSAGE on the External Agent, CREATE TASK on the schema, EXECUTE TASK on the account
Compute metricsUSE AI FUNCTIONS, or USE AI FUNCTION AI_COMPLETE
Modify or drop the appOWNERSHIP on the External Agent
GRANT DATABASE ROLE SNOWFLAKE.CORTEX_USER TO ROLE observability_user_role;
GRANT CREATE EXTERNAL AGENT ON SCHEMA app_schema TO ROLE observability_user_role;
GRANT CREATE TASK ON SCHEMA app_schema TO ROLE observability_user_role;
GRANT EXECUTE TASK ON ACCOUNT TO ROLE observability_user_role;
GRANT USE AI FUNCTIONS ON ACCOUNT TO ROLE observability_user_role;

External Agent objects share a namespace with model objects, so an application and a model cannot have the same name in one schema.

Common mistakes

Using the event table for billing. Use Account Usage views; traces can under-report.

Comparing runs from different datasets. Comparison needs a shared dataset.

Recomputing a metric on the same run. It is not allowed; create a new run.

Forgetting CREATE TASK and EXECUTE TASK. Running evaluations needs both.

Reading groundedness as correctness. Groundedness checks support from context; correctness checks against ground truth.

What to remember for the exam

  • Features: evaluations, comparisons, tracing, logging, cost metadata.
  • Metrics: answer relevance, context relevance, groundedness, correctness, coherence.
  • Comparison is in Snowsight under AI & ML, Evaluations, on runs sharing a dataset.
  • Apps register as External Agent objects; needs CORTEX_USER and CREATE EXTERNAL AGENT.

Check yourself

  • Which metric flags answers not supported by the retrieved passages?

    Groundedness.

  • What must runs share to be compared?

    The same dataset.

  • What does registering a TruLens app create in Snowflake?

    An External Agent object that stores only metadata, while traces and scores live in the event table.

Topic 14 of 15 in Snowflake Gen AI Governance

Test yourself on Snowflake Gen AI Governance

The full Snowflake GES-C02 question bank has 585 questions, with timed practice exams and an explanation for every answer.

This study guide is independent and is not affiliated with, endorsed by, or sponsored by Snowflake. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.