Collecting Metrics, Managed Prometheus, Synthetic Monitors and Custom Metrics

What it is

Cloud Monitoring stores time series metrics: numbers measured over time, with labels. Google Cloud services publish many metrics automatically, but application metrics, VM memory and disk, Prometheus metrics and business counters need extra setup.

Metrics only show what happens when users or traffic arrive. Synthetic monitors and uptime checks probe endpoints on a schedule, so you find failures before users do.

How it works

Sources of metrics

Google Cloud Managed Service for Prometheus

Managed Service for Prometheus collects Prometheus metrics and stores them in Google's global, managed backend. You keep Prometheus exporters, PromQL queries and Grafana, without running Prometheus servers or worrying about storage scale.

apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
  name: web
  namespace: shop
spec:
  selector:
    matchLabels:
      app: web
  endpoints:
  - port: metrics
    interval: 30s

Queries use PromQL in Cloud Monitoring, in Grafana or in alerting policies. Cost depends on the number of samples ingested, so a longer scrape interval costs less.

Synthetic monitoring and uptime checks

Each probe produces metrics, and you alert on failures. Synthetic monitors catch problems such as a broken login flow that a simple health endpoint would still report as up.

Custom metrics

When no existing metric measures what you need, such as orders processed or queue depth, write a custom metric with the Cloud Monitoring API or through OpenTelemetry. API-written custom metrics use the custom.googleapis.com/ prefix.

Keep label cardinality low. A label such as user ID creates a separate time series per user, which increases cost and slows queries.

Log-based metrics

A log-based metric turns matching log entries into a metric, with no code change.

gcloud logging metrics create checkout_errors \
  --description="Checkout errors" \
  --log-filter='resource.type="k8s_container" AND severity>=ERROR AND jsonPayload.route="/checkout"'

Label extractors add labels from log fields, for example the error code. A log-based metric counts entries only from when it is created; it does not backfill older logs.

A worked example

An online shop runs on GKE, with a legacy order service on VMs. The SRE team installs the Ops Agent on the VMs for memory and disk metrics, and enables Managed Service for Prometheus managed collection with a PodMonitoring for each service's metrics port.

A synthetic monitor runs the login and checkout journey every five minutes. A log-based counter tracks payment declines from structured logs, and alerts fire on synthetic failures and on spikes in declines.

How the exam asks about this

• "VM memory utilization is missing from Cloud Monitoring." Install the Ops Agent on the VMs.
• "The team uses Prometheus exporters and PromQL and wants a managed backend for many GKE clusters." Use Managed Service for Prometheus with managed collection.
• "Alert when a multi-step checkout flow breaks, even at night with no users." Create a synthetic monitor for the journey and alert on its failures.