Planning and implementing a cloud solution · Topic 6 of 9

Loading Data and Multi-Region Redundancy

Loading data with gcloud storage, bq load from Cloud Storage and Storage Transfer Service, and keeping multi-region redundancy across Google Cloud data solutions.

4 min

In a nutshell

gcloud storage cp and rsync for command-line uploads.
bq load loads files from Cloud Storage into BigQuery; gcloud sql import loads into Cloud SQL.
Storage Transfer Service for large or recurring transfers from other clouds, URLs, buckets or on premises; Transfer Appliance for offline bulk moves.

What it is

Data has to get into Google Cloud and stay available. The exam covers loading data (command-line uploads, loading from Cloud Storage into databases and BigQuery, Storage Transfer Service) and maintaining multi-region redundancy, so data survives a regional outage.

How it works

Command-line uploads

# copy files to a bucket (parallel by default)
gcloud storage cp ./exports/*.csv gs://shop-data-1234/raw/

# copy a whole folder
gcloud storage cp -r ./images gs://shop-data-1234/images/

# keep a folder in sync with a bucket
gcloud storage rsync -r ./site gs://shop-site-1234

gcloud storage is the current command set for Cloud Storage; the older gsutil tool offers similar commands.

Loading from Cloud Storage

Target	How
BigQuery	`bq load` from files in Cloud Storage (CSV, JSON, Avro, Parquet, ORC), or external tables that query files in place
Cloud SQL	`gcloud sql import sql` or `gcloud sql import csv` from a file in a bucket
Spanner, Bigtable, Firestore	Import jobs (often through Dataflow templates) from Cloud Storage files

bq load --source_format=CSV --autodetect shop.orders gs://shop-data-1234/raw/orders.csv

gcloud sql import sql orders-db gs://shop-data-1234/dumps/orders.sql --database=orders

For a Cloud SQL import, the instance's service account needs read access to the bucket.

Storage Transfer Service

Storage Transfer Service moves large amounts of data into Cloud Storage, managed and scheduled:

Source	Example
Other clouds	Amazon S3, Azure Blob Storage
HTTP and HTTPS URLs	Public datasets
Other Cloud Storage buckets	Copy between projects or regions
On-premises file systems	Using transfer agents installed in the data center

It supports one-time and scheduled transfers, incremental sync and filters. For petabytes over limited bandwidth, Transfer Appliance ships a physical device instead.

Exam tip

"Recurring copy from Amazon S3 to Cloud Storage" is Storage Transfer Service. "A few files from a laptop" is gcloud storage cp. "Hundreds of terabytes with slow internet" is Transfer Appliance.

Multi-region redundancy

Product	How to make it survive a regional outage
Cloud Storage	Use a dual-region or multi-region bucket; optionally turbo replication on dual-region buckets for a faster replication target
BigQuery	A multi-region location (such as US or EU) alone does not give regional redundancy; use cross-region dataset replication or managed disaster recovery
Spanner	Use a multi-region instance configuration
Cloud SQL	Regional HA protects against zone failure; add a cross-region read replica that can be promoted for disaster recovery
Firestore	Choose a multi-region location
Bigtable	Add clusters in other regions with replication
AlloyDB	Use cross-region replication with a secondary cluster

High availability within a region (multiple zones) and disaster recovery across regions are different goals. Multi-region designs cost more, so match them to the recovery objectives.

A worked scenario

A retailer migrates 400 TB from Amazon S3 to Cloud Storage with Storage Transfer Service, scheduled nightly until cutover. Daily CSV extracts are loaded into BigQuery with bq load. For resilience, the product images bucket is multi-region, the orders database on Cloud SQL has a cross-region read replica, and the global inventory database runs on a multi-region Spanner configuration.

How the exam asks about this

"Copy data from Amazon S3 on a schedule." Storage Transfer Service.
"Load a CSV file in Cloud Storage into a BigQuery table." bq load.
"Bucket data must survive a regional outage." Dual-region or multi-region bucket.
"Cloud SQL disaster recovery in another region." A cross-region read replica, promoted when needed.

What to remember for the exam

gcloud storage cp and rsync for command-line uploads.
bq load loads files from Cloud Storage into BigQuery; gcloud sql import loads into Cloud SQL.
Storage Transfer Service for large or recurring transfers from other clouds, URLs, buckets or on premises; Transfer Appliance for offline bulk moves.
Multi-region: dual-region or multi-region buckets, BigQuery cross-region dataset replication, multi-region Spanner configurations, cross-region replicas for Cloud SQL and AlloyDB, Bigtable replication.

Watch out

Cloud SQL's high-availability configuration protects against a zone failure only. Regional disasters need a cross-region replica or backups in another region.

Common mistakes

Scripting large S3 copies by hand. Storage Transfer Service is managed and resumable.

Assuming regional HA equals disaster recovery. It does not cover a full region outage.

Forgetting bucket permissions for imports. The service account must read the file.

Check yourself

Which service performs scheduled transfers from Amazon S3 to Cloud Storage?
Storage Transfer Service.
Which command loads a CSV file from Cloud Storage into a BigQuery table?
bq load with the source format and the gs:// path.
How can a Cloud SQL database be protected against a regional outage?
Create a cross-region read replica that can be promoted, or keep backups in another region.

Topic 6 of 9 in Planning and implementing a cloud solution

Test yourself on Planning and implementing a cloud solution

The full Google Cloud Associate Cloud Engineer question bank has 661 questions, with timed practice exams and an explanation for every answer.

Subscribe Try Sample Exam

This study guide is independent and is not affiliated with, endorsed by, or sponsored by Google Cloud. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.