Apache Spark Architecture and Components · Topic 1 of 4
Spark Benefits, Challenges and Core Components
Databricks Spark Developer: why teams use Spark, where it hurts, and what the driver, executors, worker nodes, cores and memory each do in a cluster.
- 4 min
- Includes a diagram
- The driver plans and schedules. Executors run tasks and cache data.
- Worker nodes are machines. Executors are processes on them.
- One core runs one task at a time, so slots equal executors times cores.
What it is
Apache Spark is a distributed engine that splits data into partitions and processes them in parallel on many machines. You write DataFrame or SQL code once, and Spark plans and runs it across a cluster.
A cluster is a set of machines managed together. One process, the driver, plans the work. Many executors on worker nodes do it.
How it works
Advantages
| Advantage | Why it matters |
|---|---|
| Parallel processing | Data is split into partitions, so adding machines adds speed |
| In memory processing | Intermediate data and cached DataFrames stay in memory instead of being rewritten to disk between steps |
| Fault tolerance | A lost partition is recomputed from its lineage, the recorded chain of steps that built it |
| One engine for many workloads | SQL, DataFrames, streaming and machine learning share one API and one optimizer |
| Optimization | The Catalyst optimizer rewrites your plan before anything runs |
Challenges
| Challenge | What it looks like |
|---|---|
| Shuffles | Joins and aggregations move data across the network, which is the most expensive step |
| Data skew | One partition holds far more data, so one task runs long while others sit idle |
| Memory pressure | Too little executor memory causes spilling to disk or out of memory errors |
| Small data or low latency lookups | Planning and scheduling overhead is wasted on tiny inputs, and Spark is not a transactional database |
| Tuning effort | Partition counts, memory and cores need adjusting for each workload |
| Python overhead | Python UDFs move data between the JVM and Python, which is slower than built in functions |
Core components
| Component | Role |
|---|---|
| Driver node | Runs your program, holds the SparkSession, builds the plan, splits it into tasks and schedules them. Receives results from collect |
| Cluster manager | Hands out machines and resources (standalone, YARN or Kubernetes) |
| Worker node | A machine that hosts executors |
| Executor | A JVM process that runs tasks and stores cached data for one application |
| CPU core | One core runs one task at a time (spark.task.cpus defaults to 1) |
| Memory | Executor memory holds task data and cached data. Driver memory holds the plan and any collected results |
Executors belong to one application and normally live until it ends, although dynamic allocation can release idle ones. The driver is a single point of failure for the application: if it dies, the application dies.
Cores and memory
The number of tasks that can run at once is executors multiplied by cores per executor. Four executors with four cores each give 16 task slots. A stage with 200 tasks runs in about 13 waves on 16 slots.
from pyspark.sql import SparkSession
spark = (SparkSession.builder
.config("spark.executor.cores", "4")
.config("spark.executor.memory", "8g")
.config("spark.driver.memory", "4g")
.getOrCreate())Executor memory is shared between execution (shuffles, joins, sorts) and storage (cached data) in one unified region. Spark can evict cached blocks when execution needs room, down to a protected storage share, but execution memory is never evicted for storage.
How the exam asks about this
- "A job runs
df.collect()on a huge table and the application fails with an out of memory error." The driver ran out of memory, because collect pulls every row to the driver. - "Which component runs tasks and keeps cached partitions?" The executor.
- "A cluster has 10 executors with 5 cores each. How many tasks run at once?" 50.
What to remember for the exam
- The driver plans and schedules. Executors run tasks and cache data.
- Worker nodes are machines. Executors are processes on them.
- One core runs one task at a time, so slots equal executors times cores.
- Shuffles, skew and memory pressure are the main challenges.
- Spark gains speed through partition parallelism and in memory reuse, and recovers lost partitions from lineage.
- collect, toPandas and large take calls can overload the driver.
More executors do not speed up a stage that has fewer tasks than slots. A stage with 8 partitions uses at most 8 cores, however large the cluster is.
Common mistakes
Mixing up worker node and executor. A worker node is hardware or a virtual machine, and an executor is a process on it. One worker can host several executors.
Treating Spark as always faster. For small data the overhead of planning and scheduling can make a single machine tool quicker. Spark pays off when data or computation outgrows one machine.
Blaming executors for driver failures. A failure after collect or a very large broadcast points to driver memory, not to executor memory.
Check yourself
What happens to a partition when the executor holding it is lost?
Spark recomputes it from its lineage on another executor, so no data is lost.
How many tasks can run at the same time on 6 executors with 2 cores each?
12, because task slots equal executors multiplied by cores per executor.
Which side runs out of memory when you call collect on a very large DataFrame?
The driver, since all rows are returned to it.
Topic 1 of 4 in Apache Spark Architecture and Components
Test yourself on Apache Spark Architecture and Components
The full Databricks Associate Developer for Apache Spark question bank has 600 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Databricks. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.