Columns, Rows and Table Structures

What it is

A DataFrame is immutable. Every call that "changes" a column, a row set or the table structure returns a new DataFrame and leaves the original untouched.

This page covers the everyday transformations: adding, dropping, splitting and renaming columns, filtering rows and exploding arrays into rows. The exam asks which call fits a requirement and what the result looks like.

How it works

Adding, replacing and dropping columns

withColumn(name, expression) adds a column. If the name already exists, the column is replaced, not duplicated.

from pyspark.sql import functions as F
df2 = (df.withColumn("total", F.col("price") * F.col("qty"))
         .withColumn("tier", F.when(F.col("total") > 100, "big").otherwise("small"))
         .drop("qty"))

The second argument must be a Column. A plain Python value needs F.lit(5). drop("name") removes a column and silently does nothing if the column does not exist. select keeps only the columns you list, and selectExpr takes SQL strings.

Renaming

withColumnRenamed does nothing if the old name is missing. toDF needs exactly one name per column, or it fails.

Splitting a column

F.split(col, pattern) turns a string into an array column. The pattern is a regular expression, so a literal dot or pipe must be escaped.

parts = df.withColumn("p", F.split("full_name", " "))
parts = parts.withColumn("first", F.col("p").getItem(0))

getItem(0) reads the first element of the array. Array positions start at 0 with getItem.

Filtering rows

filter and where are aliases. They accept a Column expression or a SQL string.

df.filter((F.col("age") > 30) & (F.col("country") == "US"))
df.where("age > 30 AND country = 'US'")

In Column expressions use &, | and ~, and put each condition in parentheses. The Python words and, or and not raise an error because they cannot be applied to a Column. A comparison with a null value yields null, and filter drops rows that are not true, so use isNull() and isNotNull() to find nulls.

Exploding arrays

explode turns each array element into its own row and repeats the other columns.

orders.select("id", F.explode("items").alias("item"))

An array of three items turns one input row into three output rows. For a map column, explode produces key and value columns.

Chaining and order

Because each call returns a new DataFrame, transformations chain naturally, and order can matter. Filtering early shrinks the data before costly steps such as joins. A column created by withColumn can be used by the next call in the chain, but a column you dropped cannot be referenced afterward.

Spark also evaluates lazily, so none of these transformations run until an action such as count or show is called.

Handling nulls in rows

dropna(how="any", subset=[...]) removes rows with nulls, and how="all" removes a row only when every checked column is null. fillna(0, subset=["qty"]) replaces nulls with a value of a matching type.

How the exam asks about this

• "A column holds comma separated tags and each tag needs its own row." Use split to build an array, then explode.
• "Keep orders that have no items instead of losing them when exploding." Use explode_outer.
• "Rename every column in one call." Use toDF with the full list of names.