Choosing Processing Services and Building Batch Pipelines

What it is

Building a pipeline starts with picking the service that fits the code, team and latency. Then you write the cleansing and batch transformation logic and decide how data is acquired and imported, including from sources the platform has never seen before.

How it works

Identifying the services

On Dataproc, keep data in Cloud Storage instead of HDFS. The cluster then holds no state and can be deleted after the job, which is the basis of cheap, job-based clusters.

Data cleansing

-- keep the latest version of each order
SELECT * FROM staging.orders
WHERE TRUE
QUALIFY ROW_NUMBER() OVER (PARTITION BY order_id ORDER BY updated_at DESC) = 1;

Batch transformations in Apache Beam

A batch Beam pipeline reads a bounded source, applies transforms and writes the result. The dead-letter pattern uses a tagged side output for rows that fail.

import apache_beam as beam

class ParseOrder(beam.DoFn):
    def process(self, line):
        try:
            order_id, amount, country = line.split(",")
            yield {"order_id": order_id, "amount": float(amount),
                   "country": country.strip().upper()}
        except ValueError:
            yield beam.pvalue.TaggedOutput("bad", line)

with beam.Pipeline() as p:
    rows = (p
            | beam.io.ReadFromText("gs://landing/orders/*.csv", skip_header_lines=1)
            | beam.ParDo(ParseOrder()).with_outputs("bad", main="good"))
    rows.good | beam.io.WriteToBigQuery(
        "prod-data:sales.orders",
        schema="order_id:STRING,amount:FLOAT,country:STRING",
        write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)
    rows.bad | beam.io.WriteToText("gs://landing/dead-letter/orders")

Data acquisition and import

bq load --source_format=PARQUET sales.orders "gs://landing/orders/2026-10-16/*.parquet"

Columnar, self-describing formats such as Parquet and Avro load faster than CSV and carry their schema.

Integrating new data sources

Look for a prebuilt path before writing code: a Google-provided Dataflow template (for example JDBC to BigQuery), a Cloud Data Fusion connector, a BigQuery Data Transfer Service connector or a Datastream source. Write custom Beam code only when no connector exists or the logic is unusual.

A worked example

A bank runs 200 nightly Spark jobs on an on-premises Hadoop cluster and adds a new partner who sends daily CSV files. The goal is to move quickly with a small team.

The engineer moves the Spark jobs to Dataproc, changing hdfs:// paths to gs:// paths. The partner files land in Cloud Storage, a Beam pipeline cleans them with a dead-letter output, and results load into BigQuery.

How the exam asks about this

• "Hundreds of existing Spark jobs must move to Google Cloud quickly with minimal code changes." Dataproc, with data in Cloud Storage.
• "Analysts with no coding skills must build pipelines from many SaaS and database sources." Cloud Data Fusion.
• "Malformed records currently crash the pipeline, but they must be kept for investigation." Catch them in the DoFn and write them to a dead-letter output.