Selecting Storage Systems by Access Pattern

What it is

Selecting storage means matching each dataset to the managed service whose access pattern, consistency, latency and cost fit it. A good design often uses several stores: an operational database for transactions, a warehouse for analytics and object storage for raw files.

How it works

Analyzing data access patterns

Ask four questions about each dataset:

1. Read shape: full scans and aggregations, single-key lookups, or relational queries with joins?
2. Write rate: a few transactions, or millions of events per second?
3. Latency: seconds are fine, or single-digit milliseconds?
4. Consistency and scope: strong transactions, in one region or across the world?

Choosing managed services

Bigtable and Spanner key design

Both spread data by key, so sequential keys create hotspots. In Bigtable, start the row key with an identifier such as a device ID, then a timestamp, for example device#4711#20261016T1015. In Spanner, avoid monotonically increasing primary keys; use a UUID version 4 or a bit-reversed sequence.

Planning for storage costs and performance

Lifecycle management of data

{
  "lifecycle": {
    "rule": [
      {"action": {"type": "SetStorageClass", "storageClass": "COLDLINE"},
       "condition": {"age": 90}},
      {"action": {"type": "Delete"}, "condition": {"age": 2555}}
    ]
  }
}

gcloud storage buckets update gs://raw-logs --lifecycle-file=lifecycle.json

Autoclass moves each object between classes based on its own access, which suits data with unpredictable access. Fixed lifecycle rules suit predictable aging, such as logs read for 30 days and then kept for audit.

A worked example

An online retailer needs storage for five workloads.

How the exam asks about this

• "Store sensor readings at millions of writes per second and read the last hour per sensor in milliseconds." Bigtable with a row key that starts with the sensor ID.
• "A global inventory database needs strongly consistent transactions across continents." Spanner in a multi-region configuration.
• "Logs are read often for 30 days, rarely for a year, then deleted." Cloud Storage lifecycle rules: change class after 30 days, delete after 365.