Monitoring, Tracing, Performance and Cost

What it is

Observability is the ability to explain what a production AI app did and why. In Foundry it rests on Application Insights connected to the project, OpenTelemetry tracing from your code, Azure Monitor metrics and logs from the model resource, and a monitoring view in the Foundry portal that brings token use, latency, errors and evaluation results together.

How it works

Continuous monitoring in Foundry

Connect an Application Insights resource to the Foundry project. For agents, the Monitor tab then shows operational data over time: token usage, latency, run success rate, plus evaluation scores and red teaming results when those are turned on. The recurring evaluations, red team scans and alerts on that tab are in preview.

Continuous evaluation is a rule that runs the evaluators you choose on live agent responses as they complete. It is capped by max_hourly_runs (default 100), and the project managed identity needs the Foundry User role. A quality drop then appears as a trend instead of a complaint. Scheduled evaluation runs on a fixed recurrence instead. For agents hosted in Foundry, server side tracing starts once Application Insights is connected, with no code change.

Tracing with OpenTelemetry

A trace follows one request; each step (retrieval, model call, tool call) is a span. Foundry uses OpenTelemetry with the generative AI semantic conventions, so spans carry attributes such as gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens.

import os
os.environ["AZURE_EXPERIMENTAL_ENABLE_GENAI_TRACING"] = "true"

from azure.ai.projects import AIProjectClient
from azure.ai.projects.telemetry import AIProjectInstrumentor
from azure.identity import DefaultAzureCredential
from azure.monitor.opentelemetry import configure_azure_monitor

client = AIProjectClient(endpoint=project_endpoint, credential=DefaultAzureCredential())
conn = client.telemetry.get_application_insights_connection_string()
configure_azure_monitor(connection_string=conn)
AIProjectInstrumentor().instrument()

GenAI tracing in the SDK is an opt in preview feature, so set AZURE_EXPERIMENTAL_ENABLE_GENAI_TRACING to true before calling instrument(). Add your own spans around retrieval and business steps. Message content is not recorded by default because prompts and answers can hold personal data. Turn it on with the OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT environment variable (or enable_content_recording=True in instrument()), only where policy allows, and use sampling and short retention in production.

from opentelemetry import trace
tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("retrieve") as span:
    span.set_attribute("prompt.version", "support-answer-1.4.0")
    docs = search(query)

Tagging spans with the prompt version and deployment name lets you tie a bad answer to the exact release.

Performance metrics

The Foundry resource publishes Azure Monitor metrics such as request counts, processed prompt and generated tokens, response time and provisioned utilization, split by deployment name. Chart them, set alerts, and compare to your latency target. In traces, long spans point to the slow step (often retrieval or a tool call, not the model).

Cost metrics

Cost for pay per token models is input tokens plus output tokens at each model's price; provisioned deployments are billed per PTU hour. To track and reduce it:

• Log token counts per request and aggregate by app, user or feature.
• Use Microsoft Cost Management with tags and budgets and alerts.
• Use a smaller model where quality allows, or a model router.
• Shorten prompts, limit retrieved chunks, trim conversation history and cap max_tokens.
• Reuse a stable prompt prefix so prompt caching applies to repeated input.
• Send offline work to batch deployments for lower unit cost.
• Choose provisioned throughput only when steady volume makes it cheaper.

dependencies
| where timestamp > ago(1d)
| extend inTok = toint(customDimensions["gen_ai.usage.input_tokens"]),
         outTok = toint(customDimensions["gen_ai.usage.output_tokens"])
| where isnotnull(inTok)
| summarize requests = count(), inputTokens = sum(inTok), outputTokens = sum(outTok),
            p95ms = percentile(duration, 95) by bin(timestamp, 1h)
| order by timestamp asc

Logging and debugging

Enable diagnostic settings on the Foundry resource and send logs (such as audit and request logs) and metrics to a Log Analytics workspace. Combine them with application traces in KQL. For troubleshooting, capture a correlation or request ID from responses and log it with the trace, record prompt version, model deployment and token counts, and keep error details. Do not log secrets or raw personal data; use sampling and retention limits.

A worked example

Users report slow answers. The monitoring view shows p95 latency rising, and traces show that the retrieval span grew from 200 ms to 2 s. The Azure AI Search tier was throttling. The team fixes the search capacity, and a KQL alert on p95 duration is added.

How the exam asks about this

• "See each step of a RAG request and which one is slow." OpenTelemetry tracing to Application Insights.
• "Find which feature consumes the most tokens." Aggregate token attributes from traces, or tagged cost data.
• "Capture prompts for debugging without exposing them by default." Content recording is off by default; enable deliberately.