Implementing observability practices and troubleshooting issues · Topic 6 of 6
Troubleshooting Infrastructure, Pipeline, App and Latency Issues
Troubleshoot Google Cloud infrastructure, CI/CD pipeline, application, observability and latency issues with the right tool for each kind of symptom.
- 5 min
- Connectivity Tests for network paths; Policy Troubleshooter for IAM denials.
- Build and rollout logs explain most pipeline failures; check service account roles.
- Error Reporting groups exceptions;
OOMKilledmeans memory limits.
What it is
Troubleshooting is finding the cause of a symptom quickly. The skill the exam tests is choosing the right tool for the symptom: a network path test for a connection problem, Error Reporting for a new exception, Cloud Profiler for high CPU, the audit log for a permission denial.
Work from the user impact inward, change one thing at a time, and mitigate before deep diagnosis when users are affected.
How it works
Infrastructure issues
| Symptom | Tool or check |
|---|---|
| VM will not boot or SSH fails | Serial port output, gcloud compute instances get-serial-port-output; OS Login and firewall for port 22 |
| VMs cannot reach each other or the internet | Connectivity Tests in Network Intelligence Center; firewall rules logging; routes; Cloud NAT |
| Permission denied calling an API | Policy Troubleshooter for IAM; audit logs for the denied call; organization policy and VPC Service Controls (Policy Denied logs) |
| Resource creation fails | Quota errors, organization policy constraints, zone resource availability |
| GKE pods pending | kubectl describe pod events: insufficient CPU, node selector or taint mismatch, autoscaler limits |
gcloud compute instances get-serial-port-output web-1 --zone=europe-west1-b
kubectl describe pod web-7d9c6 -n shopCI/CD pipeline issues
| Symptom | Likely cause |
|---|---|
| Trigger does not start | Branch or file filter does not match; repository connection broken |
| Build step permission denied | Build service account lacks a role on the target resource |
| Build cannot reach a private host | Not running in a private pool with the right network route |
| Cloud Deploy render fails | Skaffold, Kustomize or Helm config error; read the render logs |
| Rollout fails or times out | Image pull error, failing readiness probes, Binary Authorization denial, verification failure |
| Terraform apply fails midway | State lock held, drift, API not enabled, missing permission |
Read the build or rollout logs first; nearly every pipeline error is stated there.
Application issues
| Symptom | Tool |
|---|---|
| New exceptions after a release | Error Reporting groups exceptions, shows first seen, count and stack trace |
| Container keeps restarting | Container logs; OOMKilled in pod status means memory limit too low or a leak |
| Errors from one dependency | Traces with error status; dependency metrics |
| Wrong results, no errors | Logs with request context; compare with the previous version |
Error Reporting picks up stack traces from logs automatically on many services, so new crash types appear without extra setup.
Observability issues
| Symptom | Likely cause and fix |
|---|---|
| VM logs or metrics missing | Ops Agent not installed or not running; service account lacks Logs Writer or Monitoring Metric Writer |
| Logs missing in a project | Exclusion filter on _Default; a sink routing elsewhere; retention ended |
| Logs missing in a central bucket | Sink filter wrong; writer identity has no access to the destination |
| Alert never fires | Wrong filter or aligner, duration too long, metric absent so threshold never crossed |
| Alert fires constantly | Threshold or duration too tight; alert on symptoms with burn rate |
| Trace broken into pieces | Context not propagated between services |
| Metrics from another project not visible | Project not in the metrics scope |
Performance and latency issues
| Question | Tool |
|---|---|
| Is latency rising, and for which endpoint or region? | Cloud Monitoring latency metrics, load balancer metrics |
| Which service or call is slow? | Cloud Trace waterfalls |
| Which code uses the CPU or memory? | Cloud Profiler flame graphs, in production with low overhead |
| Is the network path slow? | Network Intelligence Center Performance Dashboard |
| Is the database the bottleneck? | Database insights such as Query Insights for Cloud SQL |
| Is it a capacity limit? | Utilization, autoscaler limits, Cloud Run concurrency and max instances |
Cloud Profiler continuously samples CPU and memory use in running code for languages such as Go, Java, Node.js and Python, and shows which functions consume the most.
Gemini Cloud Assist investigations
Gemini Cloud Assist investigations can collect related logs, metrics, configuration changes and errors for an issue and suggest possible causes. Use them to shorten the search, then verify.
A worked example
After a release, a Cloud Run service shows higher p95 latency and higher CPU, with no new errors. Error Reporting confirms no new exceptions.
Cloud Trace shows the time is spent inside the service, not in downstream calls. Cloud Profiler's flame graph shows a new JSON validation library using most of the CPU. The team reverts that change and latency returns to normal.
How the exam asks about this
- "Two VMs in different subnets cannot connect, and the firewall rules look correct." Run a Connectivity Test to analyze the path.
- "CPU usage of a service doubled after a release and you need the responsible function." Use Cloud Profiler.
- "Application logs stopped appearing for one VM." Check the Ops Agent is running and the service account has the Logs Writer role.
What to remember for the exam
- Connectivity Tests for network paths; Policy Troubleshooter for IAM denials.
- Build and rollout logs explain most pipeline failures; check service account roles.
- Error Reporting groups exceptions;
OOMKilledmeans memory limits. - Missing telemetry: agent, service account roles, exclusions, sinks, metrics scope.
- Latency: metrics to detect, Trace to locate, Profiler to find the code.
Checking only the most recent deployment can mislead. Configuration, IAM, quota and dependency changes also cause incidents, so review audit logs for all recent changes.
Common mistakes
Guessing at firewall rules. Connectivity Tests analyze the actual path and show which rule or route blocks it.
Using logs to find hot code. Logs rarely show CPU use per function. Use Cloud Profiler.
Assuming missing logs mean no events. Check the agent, roles, exclusions and sinks before concluding nothing happened.
Check yourself
Which tool explains why an IAM permission is denied?
Policy Troubleshooter, together with the audit log entry for the denied call.
What does OOMKilled in a pod status indicate?
The container exceeded its memory limit, from a limit set too low or a memory leak.
How do Cloud Trace and Cloud Profiler differ for latency issues?
Cloud Trace shows which service or call in a request is slow, while Cloud Profiler shows which functions in the code consume CPU or memory.
Topic 6 of 6 in Implementing observability practices and troubleshooting issues
Test yourself on Implementing observability practices and troubleshooting issues
The full Google Cloud Professional Cloud DevOps Engineer question bank has 751 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Google Cloud. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.