Ingesting and processing the data · Topic 1 of 4
Planning Data Pipelines on Google Cloud
Plan pipelines: sources and sinks, ETL or ELT logic and orchestration, Dataflow networking with private IPs and firewall rules, and CMEK encryption.
- 5 min
- Choose the sink by read pattern: BigQuery for analytics, Bigtable for key lookups, Cloud Storage for files.
- BigQuery and Cloud Storage subscriptions ingest Pub/Sub data with no pipeline code.
- ETL when data must be changed before storage; ELT when SQL in BigQuery is enough.
What it is
Planning a pipeline means deciding where data comes from (sources), where it lands (sinks), what happens in between (transformation and orchestration logic), and the network and encryption it runs with. Getting these right first avoids pipelines that work in a test project but fail in a locked-down production network.
How it works
Defining sources
| Source | Typical ingestion |
|---|---|
| Application or device events | Pub/Sub topic |
| Existing Kafka producers | Google Cloud Managed Service for Apache Kafka, or Dataflow reading Kafka |
| Files from partners or other clouds | Cloud Storage, via Storage Transfer Service |
| Operational databases (changes) | Datastream change data capture |
| Cloud SQL, AlloyDB or Spanner data needed occasionally | BigQuery federated queries with EXTERNAL_QUERY |
| SaaS applications | BigQuery Data Transfer Service |
Defining sinks
Pick the sink by how the data will be read, not by how it arrives.
| Read pattern | Sink |
|---|---|
| SQL analytics, dashboards, ML in SQL | BigQuery |
| Millisecond lookups by key at high volume | Bigtable |
| Raw archive, data lake, files for other engines | Cloud Storage |
| Transactions with strong consistency | Spanner, Cloud SQL or AlloyDB |
| Further event consumers | Another Pub/Sub topic |
Pub/Sub can deliver without any pipeline code: a BigQuery subscription writes messages straight into a table and a Cloud Storage subscription writes files. Use them when no transformation is needed.
Transformation and orchestration logic
| Approach | Meaning | Fits |
|---|---|---|
| ETL | Transform before loading (Dataflow, Dataproc, Data Fusion) | Heavy cleansing, streaming, PII removed before storage |
| ELT | Load raw into BigQuery, transform with SQL (Dataform, scheduled queries) | SQL teams, flexible reprocessing, simple sources |
Orchestration logic decides order, schedule, retries and backfills. Cloud Composer handles complex DAGs across services; Workflows chains API calls cheaply; BigQuery scheduled queries and Dataform workflow configurations run SQL on a timetable.
If PII must never be stored in raw form, transformation must happen before the sink. That rules out ELT for that field.
Networking fundamentals
Dataflow, Dataproc and Cloud Composer run worker VMs in a VPC network you choose.
| Requirement | Design |
|---|---|
| Workers must have no external IP addresses | Run with private IPs only and enable Private Google Access on the subnet so workers still reach Google APIs |
| Workers need the internet (for example a public API) without external IPs | Add Cloud NAT |
| Use a central network team's subnet | Shared VPC: pass the full subnetwork URL |
| Dataflow workers must talk to each other | Firewall rule allowing TCP 12345 (streaming) and 12346 (batch) between workers; needed only for jobs that do not use Dataflow Shuffle or Streaming Engine |
| Stop data leaving approved projects | VPC Service Controls perimeter |
| Reach on-premises sources | Cloud VPN or Cloud Interconnect |
Data encryption
Data is encrypted in transit and at rest by default. When the stem requires customer managed keys, set CMEK on every place data rests: the Pub/Sub topic, Dataflow job state, staging buckets and the BigQuery dataset.
python orders_pipeline.py --runner=DataflowRunner \
--project=prod-data --region=europe-west1 \
--subnetwork=https://www.googleapis.com/compute/v1/projects/net-host/regions/europe-west1/subnetworks/data-subnet \
--no_use_public_ips \
--service_account_email=df-orders@prod-data.iam.gserviceaccount.com \
--dataflow_kms_key=projects/sec-keys/locations/europe-west1/keyRings/df/cryptoKeys/orders \
--temp_location=gs://prod-data-temp/tmpA worked example
A logistics firm streams truck telemetry. Dashboards need data within a minute, a routing service needs the latest position per truck in milliseconds, and raw data must be kept for five years.
The plan: devices publish to Pub/Sub; one Dataflow streaming job writes aggregates to BigQuery and latest positions to Bigtable. A Cloud Storage subscription on the same topic archives raw messages, with a lifecycle rule moving them to Archive storage. Workers use private IPs in a Shared VPC subnet with Private Google Access.
How the exam asks about this
- "Dataflow workers have no external IPs and the job cannot reach BigQuery." Enable Private Google Access on the workers' subnet.
- "Load Pub/Sub messages into BigQuery as they are, with the least operational overhead." A BigQuery subscription on the topic.
- "A batch Dataflow job that does not use Dataflow Shuffle, in a custom VPC, hangs when workers exchange data." Add a firewall rule allowing TCP ports 12345 and 12346 between the worker VMs.
What to remember for the exam
- Choose the sink by read pattern: BigQuery for analytics, Bigtable for key lookups, Cloud Storage for files.
- BigQuery and Cloud Storage subscriptions ingest Pub/Sub data with no pipeline code.
- ETL when data must be changed before storage; ELT when SQL in BigQuery is enough.
- Private workers need Private Google Access; internet access also needs Cloud NAT.
- CMEK must cover every place data rests, including topics, job state and temp buckets.
A Shared VPC subnet must be passed as a full URL including the host project. A short subnet name makes Dataflow look for it in the service project and the job fails.
Common mistakes
Choosing the sink by ingestion format. Data arriving as events can still belong in BigQuery or Bigtable; ask how it will be read.
Building a Dataflow job just to copy messages. A BigQuery subscription or Cloud Storage subscription does it without code.
Forgetting temp and staging locations. CMEK and residency rules apply to them as well.
Check yourself
What must be enabled for Dataflow workers without external IPs to reach Google APIs?
Private Google Access on the subnet the workers use.
When is ELT a better fit than ETL?
When raw data can be stored as is and the team transforms it with SQL in BigQuery, for example with Dataform.
Which sink suits millisecond lookups of the latest value per device?
Bigtable, keyed by device ID.
Topic 1 of 4 in Ingesting and processing the data
Test yourself on Ingesting and processing the data
The full Google Cloud Professional Data Engineer question bank has 733 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Google Cloud. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.