Designing data processing systems · Topic 4 of 5
Flexible, Portable Designs and Data Discovery
Map business needs to architecture, design for multi-cloud and residency with BigLake and Omni, and stage, catalog, profile and discover data in Dataplex.
- 5 min
- Choose the design that also meets the future requirement without a rebuild.
- Keep raw data immutable so history can be reprocessed.
- BigLake governs open-format files; BigQuery Omni queries data in AWS or Azure in place.
What it is
A flexible design keeps working when requirements change: more data, new consumers, fresher results, another cloud. A portable design avoids locking data or code to one engine or one location. Governance through staging, cataloging, profiling and discovery makes sure people can find and trust the data as it grows.
How it works
Mapping current and future requirements
Translate each business phrase into an architecture property. The exam often hides the decisive requirement in one clause about the future.
| Business statement | Architecture response |
|---|---|
| "Volumes will grow tenfold" | Serverless, autoscaling services (BigQuery, Dataflow, Pub/Sub) |
| "Today daily reports, soon live dashboards" | A pipeline that can run in streaming mode, such as Apache Beam on Dataflow |
| "Other teams will reuse the data" | Keep raw data in Cloud Storage or BigQuery, separate storage from compute, catalog it |
| "We may need to reprocess history" | Keep an immutable raw landing zone, not only aggregated results |
| "Minimal operations staff" | Managed and serverless services over self-managed clusters |
When two answers both meet today's need, choose the one that also meets the stated future need without a redesign.
Designing for portability
| Need | Option |
|---|---|
| Data readable by many engines | Open file formats (Parquet, Avro, ORC) and Apache Iceberg tables |
| One governance layer over open files | BigLake tables: BigQuery security, including column and row controls, over Cloud Storage files |
| Query data that must stay in AWS or Azure | BigQuery Omni, which runs BigQuery compute in that cloud region |
| Run the same pipeline code on another runner | Apache Beam, which runs on Dataflow, Flink, Spark or locally |
| Move existing Spark and Hadoop jobs unchanged | Dataproc, which runs open source engines |
BigQuery Omni also helps with data residency. The data stays in its AWS or Azure region and only the query results move, and you can keep those results in the same region too.
Beam code is runner-portable
The same pipeline file runs anywhere; only the options change.
# local test
python orders_pipeline.py --runner=DirectRunner --input=sample.csv
# production on Dataflow
python orders_pipeline.py --runner=DataflowRunner \
--project=prod-data --region=europe-west1 \
--temp_location=gs://prod-data-temp/dataflow --input=gs://landing/orders/*.csvStaging, cataloging, profiling and discovery
| Activity | Google Cloud tool |
|---|---|
| Staging | Landing bucket or dataset for raw data, then curated zones after cleaning |
| Cataloging | Dataplex Catalog (renamed Dataplex Universal Catalog, and Knowledge Catalog since April 2026): search, business metadata, glossary terms |
| Profiling | Dataplex data profile scans: null rates, distinct counts, ranges, top values |
| Quality | Dataplex data quality scans with rules and thresholds |
| Discovery | Dataplex discovery of files in Cloud Storage, publishing tables you can query in BigQuery |
| Sensitive data discovery | Cloud DLP discovery profiles for BigQuery and Cloud Storage |
| Lineage | Data lineage for BigQuery jobs and supported pipelines |
The catalog automatically holds technical metadata for services such as BigQuery and Pub/Sub. Teams add business context: owners, descriptions, glossary terms and data quality results. That is what lets a new consumer find "the trusted customer table" instead of building another one.
A worked example
A media company runs BigQuery on Google Cloud, but a subsidiary's clickstream must stay in Amazon S3 for contractual reasons. Analysts want one place to search and query everything.
The engineer creates BigLake tables over the S3 data through BigQuery Omni, so queries run in the AWS region. Google Cloud data is registered in the same catalog, profiled with Dataplex scans and tagged with glossary terms.
How the exam asks about this
- "Data must remain in Azure, but analysts use BigQuery SQL." BigQuery Omni with BigLake tables on the Azure data.
- "Spark on Dataproc and BigQuery must both read the same tables with one set of fine-grained permissions." Open formats on Cloud Storage as BigLake or Iceberg tables.
- "Analysts cannot find which table is authoritative and trust is low." Catalog with business metadata, plus profile and data quality scans published in Dataplex.
What to remember for the exam
- Choose the design that also meets the future requirement without a rebuild.
- Keep raw data immutable so history can be reprocessed.
- BigLake governs open-format files; BigQuery Omni queries data in AWS or Azure in place.
- Beam pipelines move between runners; Dataproc runs existing open source jobs.
- Dataplex catalogs, profiles, checks quality and discovers files; Cloud DLP discovers sensitive data.
Copying multi-cloud data into Google Cloud to query it can break a residency or contract rule. Read whether the data may leave its cloud before choosing an ingestion answer.
Common mistakes
Storing only aggregates. New questions later need the detail, and it is gone.
Building a new catalog spreadsheet. Dataplex already harvests technical metadata; add business metadata there.
Rewriting Spark jobs for portability. Spark is already portable; Dataproc runs it without a rewrite.
Check yourself
Which product lets BigQuery query data that must stay in Amazon S3?
BigQuery Omni, which runs the query in the AWS region where the data lives.
What does a Dataplex data profile scan produce?
Column statistics such as null percentage, distinct values, ranges and top values, which help users judge a table before using it.
Why does Apache Beam help portability?
The same pipeline code runs on several runners, such as Dataflow, Flink or Spark, by changing only the pipeline options.
Topic 4 of 5 in Designing data processing systems
Test yourself on Designing data processing systems
The full Google Cloud Professional Data Engineer question bank has 733 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Google Cloud. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.