IAM, Encryption Keys and PII Protection

What it is

Security design for a data platform answers three questions. Who may run jobs and read data (IAM and organization policies), how data is protected at rest and in transit (encryption and key management), and what happens to personal data such as names, emails and card numbers (privacy controls).

How it works

IAM for data services

Grant roles to groups and service accounts, never to individuals one by one. Grant at the narrowest level that works: table or dataset before project.

Querying needs two things: permission to create jobs in a project, and permission to read the data. An analyst with Data Viewer alone gets "does not have bigquery.jobs.create permission".

Pipelines run as service accounts. A Dataflow job uses a worker service account; give it only the roles its sources and sinks need, such as Dataflow Worker, Pub/Sub Subscriber and BigQuery Data Editor on one dataset.

Organization policies

Organization policies are guardrails set on the organization, a folder or a project. They apply whatever IAM says.

Encryption and key management

All data is encrypted at rest and in transit by default. The design choice is who controls the keys.

BigQuery, Cloud Storage, Pub/Sub, Dataflow, Dataproc and Cloud SQL all support CMEK. Each service's service agent needs the Cloud KMS CryptoKey Encrypter/Decrypter role on the key. The key ring must be in a location compatible with the data.

Privacy and PII strategies

Tokenization lets analysts count and join on a customer without seeing the real value. Only holders of the key can re-identify.

A worked example

A retailer must let analysts study purchase history but honour erasure requests. The engineer encrypts the email column in BigQuery with a separate keyset per customer, stored in a protected key table.

-- one keyset per customer, kept in a restricted dataset
CREATE TABLE secure.customer_keys AS
SELECT customer_id, KEYS.NEW_KEYSET('AEAD_AES_GCM_256') AS keyset
FROM raw.customers;

-- store encrypted email in the analytics table
SELECT c.customer_id,
       AEAD.ENCRYPT(k.keyset, c.email, CAST(c.customer_id AS STRING)) AS email_enc
FROM raw.customers AS c
JOIN secure.customer_keys AS k USING (customer_id);

To erase a customer, delete their row from secure.customer_keys. Once the key table's time travel and fail-safe periods pass, the ciphertext in every table and copy is unreadable, which is far easier than finding every copy.

How the exam asks about this

• "Analysts can see the dataset but every query fails with a jobs.create error." Grant BigQuery Job User on the project they query from.
• "No IAM grant may ever be made to accounts outside the company." Domain restricted sharing organization policy.
• "The security team must be able to revoke access to BigQuery data by disabling a key." CMEK with Cloud KMS.