Maintaining and automating data workloads · Topic 5 of 5

Fault Tolerance, Recovery and Failover

Design data workloads for restarts, zones and regions, recover from corrupt or missing data, and use replication and failover in Cloud SQL and Memorystore.

6 min

In a nutshell

Idempotent work makes restarts and retries safe.
Dataflow batch fails after 4 failures of one item; streaming retries forever, so dead-letter bad data.
Cloud SQL HA covers zones; cross-region replicas cover regions with manual promotion.

What it is

Failures are normal: a VM dies, a zone goes dark, a bad deployment writes wrong values, a source sends nothing. Mitigating impact means designing jobs that restart safely, placing workloads across zones and regions, keeping a way to repair corrupt or missing data, and using replication with failover for databases and caches.

How it works

Designing for fault tolerance and restarts

Service	Built-in behaviour	Your design
Dataflow batch	Retries a failing work item; the job fails after 4 failures of the same item	Dead-letter bad records so one row cannot fail the job
Dataflow streaming	Retries failing items indefinitely; state is kept	Dead-letter output; alerts on lag; drain or update to deploy
Pub/Sub	Keeps unacknowledged messages until they are acknowledged or expire	Acknowledge only after a successful write; a restarted job resumes from the backlog
Dataproc	Spark retries failed tasks; high availability mode runs three master nodes	Restartable jobs, data in Cloud Storage, idempotent output
Cloud Composer	Task retries with delays	Idempotent tasks so retries cannot duplicate data

A restart is only safe when the work is idempotent. Overwrite a partition, MERGE on a key, or write with unique IDs that a sink can deduplicate.

Running jobs in multiple zones or regions

Layer	Zonal or regional protection	Regional disaster protection
BigQuery	Regional datasets are stored redundantly within the region	Cross-region dataset replication; managed disaster recovery in Enterprise Plus
Cloud Storage	Regional buckets span zones	Dual-region or multi-region buckets
Dataflow	Jobs run in a region; workers can be placed in another zone	Start the pipeline in another region reading the same Pub/Sub subscription
Dataproc	A cluster lives in one zone	Recreate clusters in another region from scripts or templates
Spanner and Bigtable	Spanner replicates across zones; a Bigtable cluster is zonal, so add a cluster in another zone	Multi-region Spanner configurations; Bigtable clusters in several regions with multi-cluster routing

Exam tip

Keep pipelines as code (templates, Terraform, DAGs in Git). Recreating a stateless job in another region is then minutes of work, and only the data needs replication.

Preparing for data corruption and missing data

Problem	Recovery tool
A bad update or delete in BigQuery within the last few days	Time travel: `FOR SYSTEM_TIME AS OF`
Need a known good copy older than time travel	Table snapshots taken on a schedule
Overwritten or deleted objects	Cloud Storage object versioning and soft delete
A bug processed a day of messages wrongly	Pub/Sub seek to a time or snapshot, then reprocess
Source data never arrived	Freshness and row-count checks raise an alert; backfill when it arrives
Derived tables wrong after a code bug	Rebuild from the immutable raw zone with the fixed code

-- restore a table to its state before a bad load
CREATE TABLE sales.order_facts_restored AS
SELECT * FROM sales.order_facts
FOR SYSTEM_TIME AS OF TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 3 HOUR);

# replay messages from 08:00 (needs retained acknowledged messages or a snapshot)
gcloud pubsub subscriptions seek orders-sub --time=2026-10-16T08:00:00Z

Data replication and failover

Service	Option	Behaviour
Cloud SQL	High availability (regional instance)	Synchronous standby in another zone; automatic failover; same connection address
Cloud SQL	Read replicas, including cross-region	Asynchronous copies for reads and disaster recovery; promote a replica manually
Memorystore for Redis	Standard tier	Replica in another zone with automatic failover; Basic tier has no replica
Memorystore for Redis Cluster	Shards with replicas across zones	Automatic failover of a failed primary shard
Spanner	Multi-region configuration	Synchronous replication; up to 99.999% availability
Bigtable	Replication with multi-cluster routing	Requests fail over to another cluster automatically

gcloud sql instances create orders-db --database-version=POSTGRES_16 \
  --region=europe-west1 --availability-type=REGIONAL

gcloud sql instances create orders-dr --master-instance-name=orders-db \
  --region=europe-west4

gcloud sql instances promote-replica orders-dr   # during a regional outage

High availability protects against a zone failure. A cross-region replica protects against a region failure, at the cost of a manual promotion and possible loss of the last few seconds of asynchronous changes.

A worked example

A payment platform uses Cloud SQL for orders, Memorystore for sessions and a Dataflow pipeline from Pub/Sub into BigQuery. The rules: survive a zone failure automatically and recover from a region failure within an hour.

The engineer enables Cloud SQL high availability, adds a cross-region read replica and uses Memorystore Standard tier. The Dataflow job is a Flex Template that can be launched in a second region against the same subscription. BigQuery datasets are replicated to a second region, and table snapshots are taken daily.

How the exam asks about this

"The database must survive a zone outage with automatic failover and no IP change." Cloud SQL high availability (regional) configuration.
"A bug wrote wrong values into Pub/Sub-fed tables for two hours this morning." Fix the code, seek the subscription back to before the bug and reprocess, then correct the affected partitions.
"A Redis cache must keep serving if its zone fails." Memorystore for Redis Standard tier, or Redis Cluster with replicas.

What to remember for the exam

Idempotent work makes restarts and retries safe.
Dataflow batch fails after 4 failures of one item; streaming retries forever, so dead-letter bad data.
Cloud SQL HA covers zones; cross-region replicas cover regions with manual promotion.
Time travel, snapshots, object versioning and Pub/Sub seek repair bad or lost data.
Keep pipelines as code so they can be relaunched in another region.

Watch out

Cloud SQL read replicas do not provide automatic failover. Zone resilience with automatic failover needs the high availability configuration.

Common mistakes

Confusing high availability with disaster recovery. HA protects against a zone failure; a region failure needs cross-region replicas or backups.

Seeking a subscription without retained messages. Seeking back in time only replays acknowledged messages if retention of acknowledged messages or a snapshot was set up first.

Choosing Memorystore Basic tier for a critical cache. Basic has no replica and no failover.

Check yourself

What protects a Cloud SQL instance against a zone failure?
The high availability (regional) configuration, with a synchronous standby in another zone and automatic failover.
How can a team reprocess Pub/Sub messages after a pipeline bug?
Seek the subscription to a time or snapshot before the bug, which requires retained acknowledged messages or a snapshot, and let the fixed pipeline process them again.
Which Memorystore for Redis tier provides automatic failover?
Standard tier, which keeps a replica in another zone; Redis Cluster also fails over shards to replicas.

Topic 5 of 5 in Maintaining and automating data workloads

Test yourself on Maintaining and automating data workloads

The full Google Cloud Professional Data Engineer question bank has 733 questions, with timed practice exams and an explanation for every answer.

Subscribe Try Sample Exam

This study guide is independent and is not affiliated with, endorsed by, or sponsored by Google Cloud. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.