Collaborating within and across teams to manage data and models · Topic 1 of 4
Organizing, Exploring and Preprocessing ML Data
Organize tabular, text and image data for ML and pick BigQuery, Dataflow, Apache Spark or in-memory Python for preprocessing by data scale and complexity.
- 5 min
- Tabular data in BigQuery; images, audio and video in Cloud Storage with a label manifest.
- Shard very large training data into many files for parallel reads.
- BigQuery for SQL-friendly structured work; Dataflow for scale, streaming and unstructured data.
What it is
Before training, data must be stored in a shape that suits its type, explored to find problems, and preprocessed into features. The exam focuses on two decisions: where each kind of data should live, and which tool should transform it given its size and complexity.
How it works
Organizing data by type
| Data type | Where it lives | Good practice |
|---|---|---|
| Tabular | BigQuery | Partition and cluster large tables; keep a snapshot or time column for reproducible training sets |
| Images, audio, video | Cloud Storage | Keep files in buckets and labels in a manifest (CSV or JSON Lines) or a BigQuery table that points to each file URI |
| Text and documents | Cloud Storage files or BigQuery string columns | Store raw text plus metadata (source, date, language); chunk long documents for gen AI use |
| Very large training sets | Cloud Storage, sharded | Many medium-sized files (for example TFRecord shards) let many workers read in parallel |
BigQuery object tables expose Cloud Storage files as rows with metadata, so you can list, filter and join images or PDFs with SQL. Managed datasets on Gemini Enterprise Agent Platform hold labels and splits for AutoML and custom training.
Exploring data
Exploration answers: how many rows, which values are missing, how classes are balanced, which features correlate with the label, and whether there are outliers or duplicates.
| Tool | Good for |
|---|---|
| BigQuery SQL | Counts, distributions and joins over billions of rows without moving data |
BigQuery DataFrames (bigframes) | pandas-style Python whose work runs inside BigQuery |
| pandas, Matplotlib, seaborn in a notebook | Detailed plots on a sample or a small dataset |
| TensorFlow Data Validation | Statistics and schema for a dataset, and comparison between datasets |
import bigframes.pandas as bpd
df = bpd.read_gbq("retail.customer_features")
print(df["churned"].value_counts()) # class balance
print(df.isna().sum()) # missing values per column
sample = df.sample(n=50_000).to_pandas() # small local copy for plottingChoosing the preprocessing tool
| Tool | Scale and complexity | Choose it when |
|---|---|---|
| BigQuery (SQL) | Very large structured data | Data already in BigQuery; joins, aggregations, window features; SQL team |
| Dataflow (Apache Beam) | Very large, batch or streaming, any format | Unstructured or streaming data, per-record Python logic, the same code for batch and streaming, autoscaling with no clusters |
| Apache Spark | Very large, cluster based | Existing Spark or PySpark code and skills, or Spark ML libraries |
| In-memory Python (pandas, scikit-learn, NumPy) | Fits in one machine's memory | Quick prototypes and small datasets in a notebook |
In order of complexity: start with SQL if it can express the logic, and reach for Beam or Spark when the data is unstructured, streaming or too complex for SQL.
"Preprocess images and text at scale with autoscaling and no cluster to manage" is Dataflow. "The team already has hundreds of PySpark jobs" is Apache Spark. A dataset of a few hundred megabytes does not need a distributed engine.
Common preprocessing steps
| Step | Example |
|---|---|
| Clean | Remove duplicates, fix types, drop impossible values |
| Impute | Fill missing numbers with the median, or add a "missing" flag |
| Encode | One-hot or embed categories; tokenize text |
| Scale | Standardize or min-max numeric values for neural networks and linear models |
| Balance | Downsample the majority class or upweight the minority class |
| Split | Train, validation and test sets; split by time or by entity to avoid leakage |
| Augment | Flip, crop or rotate images to add variety |
A worked example
A media company has 40 TB of clickstream events arriving continuously plus article text, and wants features for a recommendation model. SQL alone cannot tokenize the text cheaply, and data keeps arriving.
The engineer writes one Dataflow pipeline that parses events, joins article metadata and writes features to BigQuery, in streaming mode for fresh data and batch mode for backfills. Analysts then explore the feature tables with SQL and BigQuery DataFrames.
How the exam asks about this
- "A 2 TB customer table in BigQuery needs aggregated features; the team knows SQL." Compute the features with SQL in BigQuery.
- "Millions of images must be resized and labels attached, with autoscaling and no servers." A Dataflow (Apache Beam) pipeline reading from Cloud Storage.
- "A data scientist wants pandas syntax on a table too large for the notebook's memory." Use BigQuery DataFrames so the work runs in BigQuery.
What to remember for the exam
- Tabular data in BigQuery; images, audio and video in Cloud Storage with a label manifest.
- Shard very large training data into many files for parallel reads.
- BigQuery for SQL-friendly structured work; Dataflow for scale, streaming and unstructured data.
- Apache Spark when Spark code or skills already exist; pandas only when data fits in memory.
- Split by time or entity, not randomly, when rows are related.
Do not download a large BigQuery table into a notebook to preprocess it with pandas. It is slow, can exhaust memory and copies data out of its governed location.
Common mistakes
Choosing a cluster for small data. Distributed engines add setup and cost; a notebook is enough for data that fits in memory.
Storing images inside table cells. Keep files in Cloud Storage and reference their URIs.
Exploring only a convenient sample. A non-random sample can hide rare classes and outliers; check counts on the full data in SQL.
Check yourself
Where should training images and their labels be stored?
The image files in Cloud Storage, with labels in a manifest file or BigQuery table that references each file URI.
When is Dataflow a better choice than BigQuery SQL for preprocessing?
When the data is streaming or unstructured, or the logic needs per-record code that SQL cannot express well, with autoscaling and no clusters.
What does BigQuery DataFrames give a data scientist?
A pandas-style Python API whose operations run inside BigQuery, so large tables can be explored without loading them into memory.
Topic 1 of 4 in Collaborating within and across teams to manage data and models
Test yourself on Collaborating within and across teams to manage data and models
The full Google Cloud Professional Machine Learning Engineer question bank has 749 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Google Cloud. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.