Designing data processing systems · Topic 2 of 5
Data Sovereignty, Compliance and Environment Design
Data residency, regulations, audit logs, VPC Service Controls, project and dataset layout for governance, and separate development and production setups.
- 5 min
- Includes a diagram
- Dataset and bucket locations are fixed at creation; plan them first.
- Enforce locations with the resource locations organization policy.
- Admin Activity logs are always on; BigQuery Data Access logs are on by default.
What it is
Compliance design makes sure data stays where the law says, that access can be proven to auditors, and that the layout of projects, datasets and tables makes the right access easy. It also separates development work from production data so that a test can never leak or damage real records.
How it works
Regional considerations and data sovereignty
| Control | What it does |
|---|---|
| Dataset location | A BigQuery dataset is created in one region or multi-region (US or EU) and cannot be moved later |
| Bucket location | Cloud Storage buckets are regional, dual-region or multi-region |
gcp.resourceLocations policy | Stops anyone creating resources outside allowed locations |
| Assured Workloads | Applies a compliance control package (for example a sovereignty or government regime) to a folder |
| Access Transparency and Access Approval | Log and approve Google staff access to your content |
A BigQuery query runs in the location of the data it reads. Moving a dataset means copying it, for example with a BigQuery Data Transfer Service dataset copy, then deleting the original.
Residency rules cover every copy: backups, replicas, staging buckets, Dataflow temp locations and CMEK key rings. A pipeline that stages EU data in a US bucket breaks the rule even if the final table is in the EU.
Legal and regulatory compliance
| Requirement | Design response |
|---|---|
| Health data (HIPAA) | Sign a Business Associate Agreement and use covered services |
| Card data (PCI DSS) | Keep card data in a small, isolated scope; tokenize card numbers |
| Privacy laws (GDPR) | Residency, minimisation, erasure by deletion or crypto shredding |
| Records must not be altered | Cloud Storage retention policy with Bucket Lock |
| Prove who read what | Cloud Audit Logs: Data Access logs |
Admin Activity audit logs are always on. Data Access logs are off by default for most services, but BigQuery writes them by default. Route logs with a sink to a locked bucket or BigQuery for long retention.
Preventing exfiltration
VPC Service Controls draws a perimeter around projects. Supported APIs such as BigQuery and Cloud Storage inside it cannot be called from outside, and data cannot be copied to a project outside the perimeter. IAM still applies inside the perimeter, and a stolen credential used from outside it is blocked.
Project, dataset and table architecture
The layout is a governance tool. Access is easiest to manage when the boundaries match who may see what.
| Level | Use it for |
|---|---|
| Folder | Group projects by environment or business unit; attach organization policies |
| Project | Billing, quotas, environment boundary; separate storage projects from query projects when teams pay for their own queries |
| Dataset | The main access boundary: tables with the same audience go together |
| Table, column, row | Fine-grained exceptions: table IAM, policy tags, row access policies |
Common patterns are landing, curated and consumption datasets, and a separate dataset of authorized views that exposes a safe subset of a restricted dataset. Labels on projects, datasets and jobs allow cost reporting per team.
Multi-environment use cases
- One project per environment (dev, test, prod), grouped in folders with stricter policies on prod.
- Infrastructure as code (Terraform) so each environment is built the same way.
- A dedicated service account per pipeline per environment; dev accounts have no rights in prod.
- Dev and test use synthetic or de-identified data, never raw production PII.
- Code is promoted by CI/CD, not by hand edits in production.
A worked example
A European insurer needs analytics on claims. The rules: data stays in the EU, keys are managed by the security team, and developers never see real customer data.
bq --location=EU mk --dataset \
--default_kms_key=projects/sec-keys/locations/europe/keyRings/bq/cryptoKeys/claims \
prod-analytics:claims_curatedThe engineer creates a prod folder with a resource locations policy limited to EU locations. Keys live in a separate security project, and a nightly job de-identifies a sample with Cloud DLP into the dev project. A VPC Service Controls perimeter surrounds the prod projects.
How the exam asks about this
- "A dataset was created in US but must now be in europe-west3." Copy it to a new dataset in europe-west3 and switch consumers; the location cannot be changed.
- "Prevent data from being copied out of the analytics projects, even by users with valid credentials." VPC Service Controls perimeter.
- "Developers need realistic data without exposing customer identities." De-identify production data with Cloud DLP into a separate dev project.
What to remember for the exam
- Dataset and bucket locations are fixed at creation; plan them first.
- Enforce locations with the resource locations organization policy.
- Admin Activity logs are always on; BigQuery Data Access logs are on by default.
- Datasets are the main access boundary; use authorized views for safe subsets.
- Separate projects per environment, built with Terraform, with de-identified dev data.
IAM alone does not stop exfiltration. A user with read access can copy data to their own project unless a VPC Service Controls perimeter blocks it.
Common mistakes
Assuming a dataset location can be edited. It cannot; you copy the data to a new dataset.
Forgetting temporary and staging locations. Temp buckets and replicas must obey the same residency rule as the final table.
Sharing one project across dev and prod. A single mistaken job or grant then touches production data.
Check yourself
How do you stop any team from creating BigQuery datasets outside the EU?
Set the resource locations organization policy on the folder or organization to allow only EU locations.
Which audit logs show who read a BigQuery table?
Data Access audit logs, which BigQuery writes by default.
Why give each environment its own project?
Projects separate IAM, billing and quotas, so development jobs and accounts cannot reach production data.
Topic 2 of 5 in Designing data processing systems
Test yourself on Designing data processing systems
The full Google Cloud Professional Data Engineer question bank has 733 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Google Cloud. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.