Maintaining and automating data workloads · Topic 4 of 5
Monitoring and Troubleshooting Data Processes
Observe pipelines with Cloud Monitoring, Cloud Logging and BigQuery admin charts, track planned usage, and fix common errors, billing surprises and quotas.
- 5 min
- Cloud Monitoring for metrics and alerts; Cloud Logging for job, worker and audit logs.
- Alert on freshness, lag and backlog, not only on job failure.
- Budgets alert but do not cap; billing export and labels attribute cost.
What it is
Monitoring tells you a data process is healthy before users notice a problem. Troubleshooting turns a symptom, such as an error message, a growing backlog or a surprising bill, into a cause and a fix. Both depend on the right metrics, logs and alerts being in place from day one.
How it works
Observability of data processes
| Tool | What it shows |
|---|---|
| Cloud Monitoring | Metrics, dashboards and alerting policies for Dataflow, Pub/Sub, BigQuery, Dataproc and Composer |
| Cloud Logging | Job and worker logs, Airflow task logs, audit logs; log-based metrics and log sinks |
| BigQuery administration pages | Resource charts for slot usage, job concurrency, queued jobs and errors; a jobs explorer |
INFORMATION_SCHEMA views | Job history, bytes billed, slot time, errors and reservation usage, queryable in SQL |
| Dataflow monitoring interface | Job graph, per-step throughput, system lag, data freshness, worker logs |
| Error Reporting | Grouped exceptions from pipeline code |
Key metrics to alert on
| Signal | Meaning when it rises |
|---|---|
| Dataflow system lag | Elements wait longer to be processed: slow step, hot key or too few workers |
| Dataflow data freshness | Output is falling behind real time |
| Pub/Sub oldest unacknowledged message age | Subscribers are not keeping up or are failing to acknowledge |
| Pub/Sub undelivered messages | Backlog size |
| BigQuery slot usage and queued jobs | Capacity is saturated |
| Composer failed task or DAG runs | Orchestrated steps are failing |
Monitoring planned usage
- Budgets and budget alerts on billing accounts or projects. Budgets notify; they do not stop spending by themselves.
- Billing export to BigQuery for cost by service, SKU, project and label.
- Labels on jobs, datasets and clusters to attribute cost to teams.
- Quota dashboards and alerts so you request increases before a peak.
- Query
INFORMATION_SCHEMA.JOBSto find the users and queries that drive cost.
SELECT user_email, job_id,
total_bytes_billed / POW(1024, 4) AS tib_billed,
total_slot_ms
FROM `region-us`.INFORMATION_SCHEMA.JOBS_BY_PROJECT
WHERE creation_time > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 DAY)
AND job_type = 'QUERY'
ORDER BY total_bytes_billed DESC
LIMIT 10;Troubleshooting error messages
| Message or symptom | Likely cause and fix |
|---|---|
| "User does not have bigquery.jobs.create permission in project" | Grant BigQuery Job User on the project that runs the query |
| "Not found: Dataset ... was not found in location" | The job runs in a different location from the data; set the job location |
| "Resources exceeded during query execution" | Too much data on one worker: global ORDER BY without LIMIT, skewed keys; restructure the query |
| "Exceeded rate limits: too many table update operations for this table" | Many tiny loads or DML statements; batch them into fewer, larger operations |
| "Quota exceeded" | A project, user or custom quota is reached; reduce usage or request an increase |
| Dataflow batch "a work item has failed 4 times" | Same element fails each retry; read worker logs, fix code or route bad records to a dead-letter output |
| Dataflow streaming job stuck with rising lag | A failing element is retried indefinitely, a hot key, or the maximum workers is too low |
Troubleshooting billing issues
| Surprise | Where to look |
|---|---|
| BigQuery on-demand bill jumped | INFORMATION_SCHEMA.JOBS for top bytes billed; add partition filters, quotas and maximum bytes billed |
| Dataflow cost grew | Worker count over time; autoscaling limits; hot keys keeping workers busy |
| Dataproc cost grew | Clusters left running; add idle deletion and job-based clusters |
| Network charges | Data moving between regions; co-locate processing with storage |
Managing workloads
Cancel a runaway query with bq cancel JOB_ID. Drain a streaming Dataflow job to stop it safely, and pause a Composer DAG to stop new runs. Adjust reservations or assignments when capacity, not code, is the bottleneck.
Errors first point to permissions or location, then to query shape, and only then to capacity. Read the exact message; the exam's options often differ in exactly this step.
A worked example
At 07:00 a dashboard shows yesterday's data as missing. The Composer DAG succeeded, but the Dataflow streaming job feeding the table shows system lag rising since 02:00.
Worker logs show one malformed message failing again and again, which blocks progress for its key. The engineer adds a dead-letter output, updates the job, and creates an alert on data freshness above 15 minutes.
How the exam asks about this
- "Get notified when a streaming pipeline falls behind." Alert on Dataflow data freshness or system lag, or on the Pub/Sub oldest unacknowledged message age.
- "Find which users caused this month's BigQuery cost increase." Query INFORMATION_SCHEMA.JOBS for bytes billed by user, or analyse the billing export.
- "A query fails: Dataset not found in location US, but the dataset exists." The dataset is in another location; run the job there.
What to remember for the exam
- Cloud Monitoring for metrics and alerts; Cloud Logging for job, worker and audit logs.
- Alert on freshness, lag and backlog, not only on job failure.
- Budgets alert but do not cap; billing export and labels attribute cost.
- Read the error text: permissions, location, query shape, rate limits, then capacity.
- INFORMATION_SCHEMA.JOBS answers who ran what, for how many bytes and slots.
A budget alert does not stop spending. To cap BigQuery costs, use custom quotas or maximum bytes billed.
Common mistakes
Monitoring only job status. A streaming job can be running yet hours behind; watch freshness and lag.
Buying slots for a "Resources exceeded" error. It comes from data concentrated on one worker; change the query.
Running in the wrong location. Queries must run where the data is.
Check yourself
Which Pub/Sub metric shows that subscribers are falling behind?
The age of the oldest unacknowledged message, together with the number of undelivered messages.
Does a billing budget stop resources when it is exceeded?
No. It sends notifications; stopping spend needs quotas or automation triggered by the alert.
What usually causes "Resources exceeded during query execution"?
Too much data on a single worker, for example a global ORDER BY without LIMIT or heavily skewed keys.
Topic 4 of 5 in Maintaining and automating data workloads
Test yourself on Maintaining and automating data workloads
The full Google Cloud Professional Data Engineer question bank has 733 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Google Cloud. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.