Developing Apache Spark DataFrame/DataSet API Applications · Topic 3 of 6

Dates, Sorting and DataFrame Conversion

Spark date functions (from_unixtime, to_date, year, date_format), sorting, printSchema, collect and iteration, and DataFrame to list and pandas conversion.

4 min

In a nutshell

from_unixtime returns a string and unix_timestamp returns a long.
to_date returns a DateType column and date_format returns a string.
Use MM for month and mm for minutes in format patterns.

What it is

This page covers two sets of everyday skills. The first is working with dates: converting between Unix epoch numbers, strings, dates and timestamps, and pulling out parts such as the year. The second is getting around a DataFrame: sorting it, looking at its schema, iterating over its rows and converting it to and from Python lists.

The exam checks that you know which function returns which type and which calls bring all the data back to the driver.

How it works

Converting between epoch, string, date and timestamp

Goal	Function	Returns
Epoch seconds to a string	`F.from_unixtime(col, fmt)`	String
String to epoch seconds	`F.unix_timestamp(col, fmt)`	Long
String to date	`F.to_date(col, fmt)`	DateType
String to timestamp	`F.to_timestamp(col, fmt)`	TimestampType
Date or timestamp to string	`F.date_format(col, fmt)`	String

from_unixtime uses the format yyyy-MM-dd HH:mm:ss when you omit it, and it renders the time in the session time zone. A common two step pattern turns epoch seconds into a real date.

from pyspark.sql import functions as F
df = df.withColumn("d", F.to_date(F.from_unixtime("epoch")))

Format patterns are case sensitive. MM is month and mm is minutes, and HH is a 24 hour clock. A string that does not match the pattern gives null, or an error when ANSI mode is on.

Extracting date components

Function	Returns
`F.year`, `F.month`, `F.dayofmonth`	Integer parts of the date
`F.dayofweek`	1 for Sunday through 7 for Saturday
`F.dayofyear`, `F.weekofyear`, `F.quarter`	Position in the year
`F.hour`, `F.minute`, `F.second`	Parts of a timestamp

Date arithmetic uses F.datediff(end, start) for whole days, F.date_add(col, n) and F.date_sub(col, n) for days, and F.add_months(col, n). F.current_date() gives today's date. Note the argument order of datediff: the later date comes first.

Sorting

orderBy and sort are aliases and both sort the whole DataFrame.

df.orderBy(F.col("dept").asc(), F.col("salary").desc())
df.sort(F.desc("salary"))

A global sort needs a shuffle that range partitions the data. sortWithinPartitions sorts each partition on its own without a full shuffle, which is cheaper when a global order is not needed. Ascending order is the default, and asc_nulls_last() and desc_nulls_first() control where nulls fall.

Inspecting structure

df.printSchema() prints a tree and returns nothing. df.schema returns a StructType, df.dtypes a list of name and type tuples, and df.columns a list of names. df.show(n, truncate=False) prints rows and shows 20 by default.

Iterating and converting

Call	Result	Where the work happens
`df.collect()`	List of Row objects	All rows to the driver
`df.take(n)` or `df.head(n)`	List of the first n rows	Small result on the driver
`df.toLocalIterator()`	Iterator of rows	Fetched one partition at a time
`df.toPandas()`	pandas DataFrame	All rows to the driver
`df.foreach(f)`	Nothing	Function runs on the executors

collect on a large DataFrame can exhaust driver memory. A Row can be read as row["name"], row.name or row.asDict().

To go from a DataFrame to a plain list, select the column first.

names = [r["name"] for r in df.select("name").collect()]

To go the other way, spark.createDataFrame(data, schema) accepts a list of tuples, a list of Rows or a pandas DataFrame. A DDL string such as "id INT, name STRING" works as the schema.

How the exam asks about this

"Convert a column of epoch seconds into a readable yyyy-MM-dd string." Use from_unixtime with that format.
"Get the month number from a date column." Use F.month.
"Loop over a very large result without loading it all into driver memory." Use toLocalIterator(), not collect().

What to remember for the exam

from_unixtime returns a string and unix_timestamp returns a long.
to_date returns a DateType column and date_format returns a string.
Use MM for month and mm for minutes in format patterns.
dayofweek counts from 1 for Sunday.
printSchema() prints and returns None, and show() prints 20 rows by default.
collect() and toPandas() pull the whole result into driver memory.

Watch out

foreach runs on the executors, so any print inside it appears in executor logs, not in your notebook output.

Common mistakes

Swapping the datediff arguments. F.datediff("start", "end") gives a negative number when end is later. The end date is the first argument.

Treating a date string as a date. A column of "2026-11-02" strings has no date functions available until you convert it with to_date. Check printSchema() to see the real type.

Collecting big data to loop in Python. Calling collect() on a huge table moves everything to the driver. Prefer built in functions, or toLocalIterator when you must read every row.

Check yourself

Which function turns epoch seconds into a formatted date string?
F.from_unixtime(col, fmt), which returns a string, and to_date can then convert that string to a DateType column.
What does df.printSchema() return?
Nothing (None). It prints the schema as a tree to the output.
Why is df.collect() risky on a large DataFrame?
It returns every row to the driver at once, which can run the driver out of memory.

Topic 3 of 6 in Developing Apache Spark DataFrame/DataSet API Applications

Test yourself on Developing Apache Spark DataFrame/DataSet API Applications

The full Databricks Associate Developer for Apache Spark question bank has 600 questions, with timed practice exams and an explanation for every answer.

Get Full Access Try Sample Exam

This study guide is independent and is not affiliated with, endorsed by, or sponsored by Databricks. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.