Metrics Explorer, Dashboards, Alerting Policies and Notifications

What it is

Metrics become useful when people can explore them, watch them on dashboards and be told when something needs action. Cloud Monitoring provides the Metrics Explorer, custom dashboards and alerting policies, with notification channels to email, chat and on-call tools.

Good alerting is about what users feel and what a person must do, not about every metric that moves.

How it works

Metrics Explorer

The Metrics Explorer charts any metric without building a dashboard first. You pick a metric and resource type, then shape the data.

Switch to the PromQL editor to write queries directly. PromQL is the recommended query language; MQL is deprecated, and new MQL charts and alerting policies can no longer be created in the console.

sum by (code) (rate(http_requests_total{namespace="shop"}[5m]))

Managing dashboards

gcloud monitoring dashboards create --config-from-file=checkout-dashboard.json

Keep dashboards as code in Git so they are reviewed and identical across environments. Google Cloud also provides predefined dashboards for many services.

Alerting policies

An alerting policy has one or more conditions, notification channels and documentation sent with the notification.

displayName: Checkout 5xx ratio high
combiner: OR
conditions:
- displayName: 5xx above 5 percent for 5 minutes
  conditionPrometheusQueryLanguage:
    query: |
      sum(rate(http_requests_total{service="checkout",code=~"5.."}[5m]))
        / sum(rate(http_requests_total{service="checkout"}[5m])) > 0.05
    duration: 300s
    evaluationInterval: 60s
documentation:
  content: "Runbook: https://wiki.example.com/runbooks/checkout-5xx"
  mimeType: text/markdown
notificationChannels:
- projects/shop-prod/notificationChannels/1234567890

Prefer alerts on SLIs and SLO burn rate for paging; send cause-based alerts such as disk filling up as tickets. Use the duration setting to ignore brief spikes, and snooze alerts during planned work.

Alerting for cost control

Cost alerts come from Cloud Billing budgets, not Cloud Monitoring. A budget sends email at threshold percentages of actual or forecast spend and can publish to Pub/Sub, where a function can act, for example by stopping non-production resources.

Budgets alert; they do not cap spending by themselves.

Third-party alerting tools

Gemini Cloud Assist for metrics

Gemini Cloud Assist can help interpret a chart or an alert in the console: summarizing what changed, suggesting related metrics and possible causes, and helping write queries. Confirm what it suggests against the data.

A worked example

On-call engineers get 200 pages a week, mostly CPU threshold alerts that need no action. The team replaces them with SLO burn rate alerts for availability and latency on the checkout and search services, routed to PagerDuty.

CPU and disk alerts become tickets through a webhook to the ticketing system. Every paging policy now has documentation linking to a playbook, and pages drop to about ten a week.

How the exam asks about this

• "On-call engineers are paged by alerts that need no action." Page on SLO burn rate and user-facing symptoms; send cause-based alerts as tickets.
• "Alerts must open incidents in PagerDuty." Add a PagerDuty notification channel to the alerting policy.
• "Notify the team when monthly spend is forecast to exceed the plan." Create a Cloud Billing budget with forecast thresholds.