Designing and planning a cloud solution architecture · Topic 4 of 8
Choosing Storage, Databases and Data Processing
Picking object, file and database storage on Google Cloud: Cloud Storage, Filestore, Cloud SQL, AlloyDB, Spanner, Bigtable, Firestore, BigQuery and data pipelines.
- 3 min
- Object in Cloud Storage, file in Filestore, cache in Memorystore.
- Cloud SQL for compatible relational apps; Spanner for scale with strong consistency.
- Bigtable for high write rates and key lookups; BigQuery for analytics.
What it is
Storage design means choosing where each kind of data lives, based on its shape, how it is read and written, how consistent it must be and how much of it there is. Data processing design means choosing how data moves and is transformed: streaming or batch, SQL or code, managed or open source.
How it works
Storage types
| Type | Product | Use it for |
|---|---|---|
| Object | Cloud Storage | Media, backups, data lake files, static web content |
| File (NFS) | Filestore | Shared file systems for VMs and GKE, lift-and-shift apps that expect a mount |
| Block | Persistent Disk, Hyperdisk | VM and node disks |
| In-memory | Memorystore (Redis, Valkey, Memcached) | Caches, sessions, shopping carts, leaderboards |
Databases
| Product | Model | Choose it when |
|---|---|---|
| Cloud SQL | MySQL, PostgreSQL, SQL Server | Existing relational apps, regional scale, minimal code change |
| AlloyDB for PostgreSQL | PostgreSQL compatible | Demanding PostgreSQL workloads with fast analytics on live data |
| Spanner | Relational, horizontally scaling | Global or very large scale with strong consistency, up to 99.999% availability in multi-region configurations |
| Bigtable | Wide-column NoSQL | Huge write rates and low-latency key lookups: time series, IoT, telemetry |
| Firestore | Document | Mobile and web apps, flexible documents; also offers MongoDB compatibility |
| BigQuery | Analytics warehouse | SQL analytics over large data, BI, ML with BigQuery ML |
Key decision points:
- Strong consistency across many rows at global scale: Spanner, not Bigtable.
- Millions of writes per second, lookups by key and time: Bigtable, with a row key that avoids hotspots.
- Analytics, not transactions: BigQuery, never as the store behind a 50 ms lookup.
- "Cannot change the code" with MySQL or SQL Server: Cloud SQL.
Bigtable row keys that start with a timestamp send every write to one node. Lead with the entity (for example the vehicle ID), then the time.
Data processing
| Need | Product |
|---|---|
| Messaging and event ingestion | Pub/Sub |
| Streaming and batch pipelines in code (Apache Beam) | Dataflow |
| Existing Spark or Hadoop jobs | Managed Service for Apache Spark (formerly Dataproc) |
| Workflow orchestration with Airflow DAGs | Managed Service for Apache Airflow (formerly Cloud Composer) |
| SQL transformations with tests, in Git | Dataform in BigQuery |
| Database change capture | Datastream |
| Low-code data integration | Data Fusion, Application Integration |
Choose by the team's skills and the freshness required. A team with existing Spark code moves faster on managed Spark than rewriting in Beam; a SQL team transforming BigQuery tables uses Dataform.
Consistency, durability and location
- Cloud Storage location types: region, dual-region and multi-region. Dual-region keeps one bucket name across two regions; turbo replication sets a replication target of 15 minutes.
- Spanner configurations can be regional or multi-region; pick one whose regions all sit inside any required geography.
- Residency rules apply to backups, replicas and analytics copies too.
How the exam asks about this
- "Dealer orders, scarce inventory, strongly consistent, 99.999%, EU only." Spanner in an EU multi-region configuration.
- "3 million telemetry writes a second, last 24 hours by vehicle in under 50 ms." Bigtable.
- "Keep MongoDB drivers, Google-operated, serverless." Firestore with MongoDB compatibility.
What to remember for the exam
- Object in Cloud Storage, file in Filestore, cache in Memorystore.
- Cloud SQL for compatible relational apps; Spanner for scale with strong consistency.
- Bigtable for high write rates and key lookups; BigQuery for analytics.
- Dataflow for Beam pipelines, managed Spark for existing Spark, Dataform for SQL in BigQuery.
- Locations and replicas must respect residency rules.
BigQuery is not a serving database. Low-latency per-key lookups belong in Bigtable, Firestore, Spanner or a cache.
Common mistakes
Picking Bigtable for multi-row transactions. It offers single-row atomicity only.
Rewriting working Spark jobs. Managed Spark runs them with little change.
Forgetting hotspots. Sequential keys in Bigtable or Spanner overload one server.
Check yourself
Which database gives strongly consistent transactions with horizontal write scale?
Spanner.
Which product runs existing Spark jobs with little change?
Managed Service for Apache Spark, formerly Dataproc.
Why should a Bigtable row key not start with a timestamp?
Sequential keys send all new writes to one node, creating a hotspot.
Topic 4 of 8 in Designing and planning a cloud solution architecture
Test yourself on Designing and planning a cloud solution architecture
The full Google Cloud Professional Cloud Architect question bank has 877 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Google Cloud. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.