Distributed Tracing with OpenTelemetry and Cloud Trace

What it is

A user request often passes through a load balancer, several services and a database. A distributed trace records that whole journey as a tree of timed spans, so you can see where the time went and where an error started.

Cloud Trace stores and displays traces on Google Cloud. OpenTelemetry is the recommended way to produce them.

How it works

Traces, spans and context

If one service does not propagate context, the trace breaks into separate pieces. Every hop must pass the header on.

Instrumenting with OpenTelemetry

OpenTelemetry provides SDKs for common languages and automatic instrumentation for popular frameworks and HTTP and gRPC clients. Export spans directly or through an OpenTelemetry collector to Cloud Trace.

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

def checkout(cart):
    with tracer.start_as_current_span("charge-card") as span:
        span.set_attribute("cart.items", len(cart.items))
        return payments.charge(cart)

Automatic instrumentation creates spans for incoming and outgoing calls; add manual spans for important internal steps such as a pricing calculation. Use the same instrumentation on every platform, which is why OpenTelemetry suits hybrid estates.

Analyzing trace waterfalls and spans

The trace explorer in Cloud Trace shows a scatter of requests by latency, and opening one shows its waterfall: each span as a bar on a timeline, nested under its parent.

Compare slow and normal traces of the same endpoint. The span that grew is the place to look.

Correlating trace IDs with structured logs

When a log entry carries the trace and span IDs, Cloud Logging links it to the trace, and the trace view shows its logs. Write them in structured logs with these special fields:

{
  "severity": "ERROR",
  "message": "payment gateway timeout",
  "logging.googleapis.com/trace": "projects/shop-prod/traces/4bf92f3577b34da6a3ce929d0e0e4736",
  "logging.googleapis.com/spanId": "00f067aa0ba902b7"
}

Then a query on trace="projects/shop-prod/traces/4bf92f3577b34da6a3ce929d0e0e4736" returns every log line for that request across all services. Many logging libraries and OpenTelemetry integrations add these fields automatically.

Gemini Cloud Assist for trace analysis

Gemini Cloud Assist can help summarize a trace, point out the spans that contribute most latency and suggest likely causes, together with related logs. As with any AI help, confirm the conclusion against the spans themselves.

Where traces come from automatically

Some Google Cloud services, such as Cloud Run and App Engine, send trace data for incoming requests without code changes. These show the edge of the request, but end-to-end visibility inside and between your services needs instrumentation and propagation in the code.

A worked example

Users report a slow checkout at the 99th percentile only. Cloud Monitoring shows latency rising, but logs show no errors.

In Cloud Trace, the engineer compares slow and fast checkout traces. The slow ones show the inventory service calling the database 40 times in sequence; the log lines linked through the trace ID show a cache miss. The team batches the queries and fixes the cache.

How the exam asks about this

• "Find which of six microservices adds most latency to a request." Instrument with OpenTelemetry and analyze the trace waterfall in Cloud Trace.
• "Engineers need to jump from a trace to the matching log entries." Write the trace and span IDs into structured logs using the special logging.googleapis.com/trace fields.
• "Traces end at the second service and restart as new traces." Fix context propagation so the traceparent header is passed on.