Apache Spark Architecture and Components · Topic 4 of 4
Actions, Transformations, Lazy Evaluation and Spark Modules
Databricks Spark Developer: which calls are lazy transformations and which are actions, plus what Spark Core, SQL, Streaming, Pandas API and MLlib offer.
- 4 min
- Transformations are lazy and return a DataFrame. Actions run a job and return a value or write output.
- Common actions: count, collect, take, first, show, toPandas, foreach and write operations.
- printSchema, explain and cache do not trigger a job over the data.
What it is
Transformations describe a new DataFrame from an existing one, such as filter or select. Actions ask Spark for a result or an output, such as count or write. Spark runs nothing for a transformation. It records it in a plan and waits.
This is lazy evaluation. When an action finally arrives, the Catalyst optimizer rewrites the whole plan and Spark runs it in one go. The same engine is delivered as a set of modules, each a different API on top of it.
How it works
Transformations and actions
| Transformations (lazy) | Actions (run a job) |
|---|---|
| select, filter, where, withColumn, drop | count, collect, take, first, head |
| groupBy then agg, orderBy, join, union | show, toPandas |
| distinct, dropDuplicates, repartition | write.save, write.parquet, saveAsTable |
| read with an explicit schema | foreach, foreachPartition |
Some calls look like actions but are not. printSchema(), explain(), columns, dtypes and cache() do not run a job over the data. A read that infers the schema, such as JSON or CSV with inferSchema, does run a job.
from pyspark.sql import functions as F
df = spark.read.parquet("/data/orders")
big = df.filter(F.col("amount") > 100)
big.count()
big.show(5)Nothing runs on the first three lines. count() and show(5) each trigger a job, and each rereads the source unless big is cached.
Why laziness helps
- Optimization: Spark sees the whole chain, so it can push a filter down to the file scan, prune unused columns and combine steps into one stage.
- Less work:
take(5)after a long chain can stop early instead of processing everything. - Fault tolerance: the plan is also the lineage, so lost partitions can be rebuilt.
Laziness has a cost for debugging. Errors that depend on the data, such as a failing cast (with ANSI mode on, the default in Spark 4) or a corrupt record, appear only when an action runs, not at the line that caused them. Name and type problems are caught earlier, because the analyzer resolves columns as soon as you call the transformation.
Actions that return data to the driver
collect() returns every row as a list of Row objects, take(n) and head(n) return the first n rows, and first() returns one row. toPandas() converts the whole DataFrame into a pandas DataFrame in driver memory.
These are safe on small results and risky on large ones. Filter, aggregate or limit() first, or write the result to storage instead of pulling it back.
Spark modules
| Module | What it provides |
|---|---|
| Spark Core | The base engine: scheduling, memory management, fault recovery and the RDD API |
| Spark SQL and DataFrames | Structured data with schemas, SQL, the Catalyst optimizer and many data sources |
| Pandas API on Spark | The pandas style API (pyspark.pandas) running distributed on Spark |
| Structured Streaming | Continuous processing of unbounded data with the same DataFrame API |
| MLlib | Distributed machine learning: algorithms, features and pipelines. The DataFrame based API is pyspark.ml |
All modules share the same driver, executors and optimizer. A pipeline can read with SQL, transform with DataFrames, score with MLlib and write a stream without leaving Spark.
How the exam asks about this
- "Which line triggers execution?" The one with an action, such as
count(),collect(),show()orwrite. Chained filter, select and join lines do not. - "A filter is written after a join in code, yet the scan reads fewer files." Catalyst pushed the filter down because evaluation is lazy and the plan is optimized as a whole.
- "Which module processes a continuous stream of events with DataFrame code?" Structured Streaming.
What to remember for the exam
- Transformations are lazy and return a DataFrame. Actions run a job and return a value or write output.
- Common actions: count, collect, take, first, show, toPandas, foreach and write operations.
- printSchema, explain and cache do not trigger a job over the data.
- Every action recomputes the plan from the source unless the DataFrame is cached.
- Spark Core is the base engine. SQL and DataFrames, Pandas API on Spark, Structured Streaming and MLlib build on it.
explain()prints the plan Spark will run without running it.
groupBy("k") alone returns a grouped object, not a DataFrame result. Nothing is computed until you add an aggregation and then an action.
Common mistakes
Calling a missing action and expecting output. A chain of transformations with no action runs nothing and shows nothing. Add show(), count() or a write.
Placing an action in a loop. Each count() inside a loop is a full job. Cache the DataFrame first or avoid the repeated action.
Thinking Pandas API on Spark is the same as a pandas DataFrame. It is distributed and lazy in the same way as other Spark code, whereas a plain pandas DataFrame lives in driver memory.
Check yourself
Which of
select,filterandcountstarts a job?Only
count, because it is an action. Select and filter are lazy transformations.Why can the same DataFrame be computed twice by two actions?
Spark does not keep results by default, so each action reruns the plan from the source unless the DataFrame is cached.
Which module lets you apply DataFrame code to unbounded data?
Structured Streaming.
Topic 4 of 4 in Apache Spark Architecture and Components
Test yourself on Apache Spark Architecture and Components
The full Databricks Associate Developer for Apache Spark question bank has 600 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Databricks. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.