Apache Spark Architecture and Components · Topic 2 of 4
DataFrames, SparkSession, Caching and Garbage Collection
Databricks Spark Developer: DataFrame and Dataset concepts, the SparkSession lifecycle, cache and persist storage levels, and JVM garbage collection.
- 4 min
- PySpark has DataFrames only. Typed Datasets are Scala and Java.
- DataFrames are immutable. Each transformation returns a new one.
- getOrCreate reuses an active session. spark.stop() ends it.
What it is
A DataFrame is a distributed table with named columns and a schema. A Dataset is the typed version of the same idea, available in Scala and Java. In PySpark you work only with DataFrames, which are Datasets of Row objects.
The SparkSession is the entry point to all of this. Caching keeps a DataFrame's partitions in memory or on disk so later actions do not recompute them. Garbage collection is how the JVM frees memory on the driver and executors.
How it works
DataFrame and Dataset
| DataFrame | Dataset | |
|---|---|---|
| Structure | Rows with named, typed columns | Typed objects such as case classes |
| Languages | Python, Scala, Java, R, SQL | Scala and Java only |
| Type checking | At run time against the schema | At compile time |
| Optimizer | Catalyst and Tungsten | Catalyst and Tungsten |
Both are immutable. Every transformation returns a new DataFrame and leaves the original unchanged.
SparkSession lifecycle
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("etl").getOrCreate()
other = spark.newSession()
df = spark.read.parquet("/data/sales")
spark.stop()getOrCreate() returns the existing session if one is active and creates one otherwise. Databricks notebooks already provide spark. newSession() shares the same SparkContext and cached data but has its own temporary views and SQL configuration. spark.stop() ends the application and releases its executors.
Caching and storage levels
cache() and persist() are lazy. Nothing is stored until an action runs, and an action that touches only some partitions, such as show(), caches only those. A full count() fills the cache.
from pyspark import StorageLevel
df = spark.read.parquet("/data/events")
df.persist(StorageLevel.DISK_ONLY)
df.count()
df.unpersist()| Storage level | Behavior |
|---|---|
| MEMORY_ONLY | Keeps partitions in memory. Partitions that do not fit are recomputed when needed |
| MEMORY_AND_DISK | Keeps partitions in memory and spills the rest to disk. The default for DataFrame cache() |
| DISK_ONLY | Keeps partitions on disk only |
| Levels ending in _2 | Same as above but with two replicas on different nodes |
A second persist() with a different level is not applied while the first is still in place. Call unpersist() first.
Cache a DataFrame when several actions reuse it, such as an expensive join that feeds two reports. Do not cache data used once, since the cache costs memory and a pass over the data.
Garbage collection
Spark runs in JVMs, and the JVM frees unused objects through garbage collection. Many short lived objects, or a cache that fills the heap, make collections long and frequent. Tasks then stall while the JVM pauses.
The Spark UI shows GC time per task and per executor. Common remedies are:
- more executor memory, or fewer cores per executor so tasks share the heap less
- built in functions instead of Python UDFs, which create extra objects
- a smaller cache, or a level that spills to disk
- a different collector such as G1GC, set through
spark.executor.extraJavaOptions
How the exam asks about this
- "Two reports read the same joined DataFrame. How do you avoid computing the join twice?" Cache or persist it before the first action.
- "A cached DataFrame is no longer needed." Call unpersist() to free executor memory.
- "The Spark UI shows tasks spending a large share of their time in GC." This signals memory pressure: add memory, reduce cached data or avoid UDFs.
What to remember for the exam
- PySpark has DataFrames only. Typed Datasets are Scala and Java.
- DataFrames are immutable. Each transformation returns a new one.
- getOrCreate reuses an active session. spark.stop() ends it.
- cache and persist are lazy and fill on the first action that reads the data.
- DataFrame cache() uses MEMORY_AND_DISK.
- High GC time in the Spark UI points to too many objects or too little memory.
Calling df.cache() does not run anything. If no action follows, nothing is cached, and a show() caches only the partitions it read.
Common mistakes
Assuming cache changes the data in place. Caching stores the partitions of that DataFrame object. A different DataFrame built by another transformation is not cached unless you cache it too.
Caching everything. Cached data competes with execution for the same memory. Too much caching causes eviction, spilling and longer GC pauses.
Expecting a second persist to change the level. Without unpersist the original level stays, so switching from MEMORY_AND_DISK to DISK_ONLY needs an unpersist first.
Check yourself
What does
getOrCreate()do when a SparkSession already exists?It returns the existing session instead of building a new one.
Why does a cached DataFrame still recompute on the first action?
Cache is lazy, so the first action both computes the result and fills the cache. Later actions read from it.
Which symptom in the Spark UI suggests garbage collection trouble?
A large share of task time spent in GC, which points to memory pressure or too many objects.
Topic 2 of 4 in Apache Spark Architecture and Components
Test yourself on Apache Spark Architecture and Components
The full Databricks Associate Developer for Apache Spark question bank has 600 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Databricks. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.