Developing Apache Spark DataFrame/DataSet API Applications · Topic 5 of 6

Reading and Writing DataFrames with Schemas

Read and write DataFrames in PySpark with an explicit schema: StructType or DDL strings, read modes, save modes, overwrite, append and partitionBy.

4 min

In a nutshell

Define a schema with a StructType or a DDL string such as "id INT, name STRING".
Without a schema, CSV columns are strings unless inferSchema is set, and inference costs an extra job.
The default read mode is PERMISSIVE, and the default save mode is errorifexists.

What it is

Input and output operations move data between storage and DataFrames. Reading builds a DataFrame from files, and writing saves a DataFrame to a path or table in a chosen format and save mode.

A schema describes the column names, types and nullability. Giving Spark the schema yourself is more reliable and faster than letting it guess.

How it works

Reading with a schema

You can define the schema as a StructType or as a DDL string.

from pyspark.sql.types import StructType, StructField, IntegerType, StringType
schema = StructType([
    StructField("id", IntegerType(), False),
    StructField("name", StringType(), True)])
df = spark.read.schema(schema).csv("/data/in", header=True)

df = spark.read.schema("id INT, name STRING").json("/data/in")

spark.read.format("csv").option("header", True).load(path) is the long form of the same read. The third argument of StructField is nullable.

Why an explicit schema matters

Without a schema	With a schema
CSV columns are all strings unless `inferSchema` is on	Columns get the types you declare
Inferring a CSV or JSON schema runs an extra job over the data	No extra pass over the data
Inferred types can change when the data changes	Types stay stable

header=True makes the first CSV line the column names, and without it Spark names columns _c0, _c1 and so on. Parquet and Delta files carry their own schema, so no inference is needed.

Handling bad records on read

The mode option controls malformed rows in file sources such as CSV and JSON.

Read mode	Behavior
`PERMISSIVE` (default)	Keeps the row and puts nulls in fields that cannot be parsed
`DROPMALFORMED`	Drops the bad rows
`FAILFAST`	Raises an error on the first bad row

Writing and save modes

(df.write.mode("overwrite")
   .format("parquet")
   .partitionBy("country")
   .save("/data/out"))

The shortcut df.write.parquet("/data/out") does the same without format and save.

Save mode	When data already exists at the target
`errorifexists` or `error` (default)	Raises an error
`append`	Adds the new rows to the existing data
`overwrite`	Replaces the existing data
`ignore`	Does nothing and skips the write

Writes create a directory of part files, one per partition, not a single file. Calling coalesce(1) before the write produces a single part file, but it funnels all data through one task.

Overwrite and partitions

partitionBy("country") creates one sub directory per value, such as country=US. Filters on that column later read only the matching directories, which is called partition pruning. Choose low cardinality columns, because a column with millions of values creates millions of tiny directories.

A plain overwrite with partitionBy replaces the whole target path by default. Setting spark.sql.sources.partitionOverwriteMode to dynamic replaces only the partitions that the new data contains.

How the exam asks about this

"Read a CSV with a header and exact column types, without scanning the file to infer them." Pass the schema with spark.read.schema(...) and header=True.
"A job must fail if the output folder already has data." Use the default save mode, errorifexists.
"Write data so that queries on region read only part of the files." Use partitionBy("region").

What to remember for the exam

Define a schema with a StructType or a DDL string such as "id INT, name STRING".
Without a schema, CSV columns are strings unless inferSchema is set, and inference costs an extra job.
The default read mode is PERMISSIVE, and the default save mode is errorifexists.
overwrite replaces existing data, append adds to it, and ignore skips the write.
A write produces a folder of part files, and partitionBy creates column=value sub directories.
Parquet stores the schema in the file, so reading it needs no schema.

Watch out

mode("overwrite") deletes what is at the target path before writing. Pointing it at the wrong folder destroys data.

Common mistakes

Expecting a single output file. df.write.csv("out.csv") creates a folder named out.csv containing part files. Use coalesce(1) only when a single file is truly required.

Forgetting the header option. Reading a CSV without header=True turns the header line into a data row and names columns _c0, _c1. Setting the schema does not remove the header line by itself.

Partitioning on a high cardinality column. Partitioning by a user id produces a huge number of small files and slow reads. Pick a column with few distinct values that queries filter on.

Check yourself

What happens by default when you write to a path that already contains data?
The default save mode, errorifexists, raises an error and writes nothing.
Why pass a schema when reading a large CSV file?
It avoids the extra job that infers types, and it guarantees stable, correct column types.
What does partitionBy("country") produce on disk?
One sub directory per value, such as country=US, which lets later filters on country skip the other directories.

Topic 5 of 6 in Developing Apache Spark DataFrame/DataSet API Applications

Test yourself on Developing Apache Spark DataFrame/DataSet API Applications

The full Databricks Associate Developer for Apache Spark question bank has 600 questions, with timed practice exams and an explanation for every answer.

Get Full Access Try Sample Exam

This study guide is independent and is not affiliated with, endorsed by, or sponsored by Databricks. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.