Apache Spark Architecture and Components · Topic 3 of 4
Execution Hierarchy, Partitions and Shuffles
Databricks Spark Developer: how jobs, stages and tasks relate, what a shuffle is, and how repartition, coalesce and shuffle partitions set the counts.
- 5 min
- Includes a diagram
- Each action starts a job. Wide transformations split a job into stages. A stage has one task per partition.
- Narrow transformations stay in one stage. Wide ones need a shuffle.
- spark.sql.shuffle.partitions defaults to 200 and spark.sql.files.maxPartitionBytes defaults to 128 MB.
What it is
Spark runs work in a fixed hierarchy: an application contains jobs, a job contains stages, and a stage contains tasks. Each level has one clear trigger, and exam questions ask you to count them.
A partition is a slice of the data that one task processes. A shuffle is the redistribution of data across partitions and machines, needed when rows with the same key must end up together.
How it works
The hierarchy
| Level | Created by | Count rule |
|---|---|---|
| Application | One SparkSession and its driver | One per spark-submit or notebook session |
| Job | Each action such as count, collect or write | Usually one job per action |
| Stage | Shuffle boundaries inside a job | Number of shuffles plus one, in a simple plan |
| Task | One unit of work on one partition | One per partition per stage |
Take df.filter(...).groupBy("k").count().collect(). The collect() is the action, so there is one job. The groupBy needs a shuffle, so the job has two stages: read, filter and partial count before the shuffle, then final count after it. If the input has 40 partitions and shuffle partitions are 200, stage one has 40 tasks and stage two has 200, unless adaptive execution coalesces the shuffle partitions.
Narrow and wide transformations
| Type | Needs a shuffle | Examples |
|---|---|---|
| Narrow | No. Each output partition reads one input partition | select, filter, withColumn, union, coalesce |
| Wide | Yes. Output partitions read many input partitions | groupBy, join (unless broadcast), distinct, orderBy, repartition |
Narrow transformations are pipelined inside one stage. A wide transformation ends a stage, writes shuffle files and starts a new stage. This is why shuffles are costly: they use disk, network and a wait for every task in the earlier stage.
Configuring partitions
| Setting or call | Effect |
|---|---|
| spark.sql.files.maxPartitionBytes | Target size of an input partition when reading files. Default 128 MB |
| spark.sql.shuffle.partitions | Number of partitions after a shuffle in DataFrame and SQL. Default 200 |
| repartition(n) | Full shuffle to exactly n roughly equal partitions. Can increase or decrease |
| repartition(n, "col") | Full shuffle that hashes rows by column, so equal keys land together |
| coalesce(n) | Merges partitions without a full shuffle. Can only decrease |
spark.conf.set("spark.sql.shuffle.partitions", "64")
df = spark.read.parquet("/data/events")
by_user = df.repartition(32, "user_id")
small = by_user.coalesce(8)
print(small.rdd.getNumPartitions())The last line prints 8. Use coalesce to reduce partitions cheaply, for example before writing fewer output files. Use repartition when you need more partitions, an even spread, or co-location by key.
Choosing a partition count
Too few partitions leave cores idle and risk memory pressure per task. Too many create scheduling overhead and tiny tasks. The Spark tuning guide recommends two to three tasks per CPU core in the cluster.
Adaptive Query Execution, on by default since Spark 3.2, can coalesce small shuffle partitions at run time, so the 200 default is often a starting point rather than the final count. Tuning AQE is covered in the troubleshooting chapter.
Seeing the hierarchy in the Spark UI
The Spark UI lists each job, the stages inside it and the number of tasks per stage, so you can check your count against reality. A stage with one very slow task among many fast ones points to data skew. A stage with thousands of tiny tasks points to too many partitions.
Shuffle data is written to the local disks of the executors and fetched by the next stage. That is why a lost executor during a shuffle can force earlier work to be recomputed.
How the exam asks about this
- "A job calls
count()once andcollect()once on two different DataFrames. How many jobs?" Two, one per action. - "A groupBy followed by a join of two large tables." Each wide step adds a stage boundary, so there are several stages.
- "Reduce 500 partitions to 10 before writing, avoiding a full shuffle." Use coalesce(10).
What to remember for the exam
- Each action starts a job. Wide transformations split a job into stages. A stage has one task per partition.
- Narrow transformations stay in one stage. Wide ones need a shuffle.
- spark.sql.shuffle.partitions defaults to 200 and spark.sql.files.maxPartitionBytes defaults to 128 MB.
- repartition always shuffles and can go up or down. coalesce avoids a full shuffle and only goes down.
- Partitioning by a column with repartition(n, "col") puts equal keys in the same partition.
- Tasks per stage is bounded by partitions, not by cores.
coalesce cannot increase the partition count. df.coalesce(100) on a DataFrame with 20 partitions still has 20.
Common mistakes
Counting stages by the number of transformations. Only wide transformations create a stage boundary. Ten chained narrow steps are still one stage.
Using repartition just to shrink output. It works, but it pays for a full shuffle that coalesce avoids. Coalesce can leave uneven partitions, which is the trade off.
Expecting changes to shuffle partitions to affect narrow steps. The setting applies after a shuffle, not to the partitions created when reading files.
Check yourself
Why does
repartition(8)cost more thancoalesce(8)on a DataFrame with 100 partitions?Repartition performs a full shuffle of the data, while coalesce merges neighboring partitions without one.
How many stages does
df.groupBy("k").count().show()have in a simple plan?Two, because the groupBy shuffle splits the job into a stage before it and a stage after it.
What decides how many tasks a stage runs?
The number of partitions in that stage, with one task per partition.
Topic 3 of 4 in Apache Spark Architecture and Components
Test yourself on Apache Spark Architecture and Components
The full Databricks Associate Developer for Apache Spark question bank has 600 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Databricks. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.