Chapter 2 of 5 · Google Cloud Professional Data Engineer
Ingesting and processing the data
Pipeline planning, Dataflow, Dataproc, streaming and CI/CD.
- 4topics
- 20 minfast track
- 1diagrams and exhibits
- 25%of the exam
Topics in this chapter
Take them in order, or jump to the one you need.
- 1Planning Data Pipelines on Google CloudPlan pipelines: sources and sinks, ETL or ELT logic and orchestration, Dataflow networking with private IPs and firewall rules…5 min
- 2Choosing Processing Services and Building Batch PipelinesChoose Dataflow, Dataproc, Data Fusion, BigQuery, Pub/Sub or Kafka; cleanse data, build batch transforms in Beam and SQL, and…Diagram5 min
- 3Streaming, Windowing, Late Data and AI EnrichmentDataflow streaming: event time, watermarks, fixed, sliding and session windows, late data and triggers, processing logic, AI…5 min
- 4Deploying Pipelines with Orchestration and CI/CDRun pipelines in production: Cloud Composer or Workflows, Dataflow Flex Templates, CI/CD with tests and Cloud Build, and safe…5 min
About this chapter
What this chapter covers
Ingesting and processing the data is about 25% of the Google Cloud Professional Data Engineer exam, the largest share of any section. It covers how data gets from its sources into Google Cloud, how it is transformed in batch and streaming pipelines, and how those pipelines are deployed and run automatically.
It covers planning pipelines (sources and sinks, transformation and orchestration logic, networking and encryption), building pipelines (choosing between Dataflow, Apache Beam, Dataproc, Cloud Data Fusion, BigQuery, Pub/Sub, Spark, Hadoop and Kafka; cleansing; batch and streaming transformations with windows and late data; AI enrichment; data acquisition and new sources) and deploying and operationalizing pipelines (Cloud Composer, Workflows and CI/CD).
The exam has 40 to 50 multiple choice and multiple select questions in 2 hours. Questions in this section usually describe a source, a freshness need and a team's skills, then ask "What should you do?"; the answer is the service that fits all three.
How the topics fit together
Planning fixes the shape of the pipeline, building fills in the logic, and deployment makes it repeatable. Most wrong answers fail on one detail: wrong latency, too much operations work, or a rewrite the stem said to avoid.
Match the team and the code. Existing Spark or Hadoop jobs point to Dataproc, new batch and streaming pipelines to Dataflow, SQL teams to BigQuery and Dataform, and no-code teams to Cloud Data Fusion.
How to use these pages
Each topic page has the same shape, so you can skim for what you need.
- What it is: the idea in plain words.
- How it works: the mechanism, with tables and code examples.
- What to remember for the exam: the facts to memorise and how questions are phrased.
- Common mistakes: the classic traps.
- Check yourself: three questions to test your understanding.
Ready for more practice?
The full Google Cloud Professional Data Engineer question bank has 733 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Google Cloud. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.