Apache Spark Architecture and Components

What this chapter covers

Apache Spark Architecture and Components is 20% of the Databricks Associate Developer for Apache Spark exam, tied with Using Spark SQL as the second largest chapter. It covers how Spark is built and how it runs your code.

The chapter has four parts: why teams use Spark and what the driver, executors, cores and memory do, how DataFrames, the SparkSession, caching and garbage collection work, how jobs, stages, tasks, partitions and shuffles relate, and how lazy evaluation separates actions from transformations across the Spark modules.

Questions are single-answer multiple choice. Many show a short PySpark snippet in the stem or the options and ask what it returns, how many jobs or stages it creates, which line triggers execution, or which call meets a requirement. There are no exhibits.

How the topics fit together

The driver and executors are the machines and processes everything else runs on. The session builds a plan, an action turns it into jobs, shuffles cut jobs into stages, and partitions decide how many tasks each stage runs.

• Component questions ask which part plans, which part runs tasks and where an out of memory error comes from.
• Caching questions ask when cache fills, which storage level fits and when to unpersist.
• Hierarchy questions ask how many jobs, stages or tasks a snippet creates.
• Partition questions ask whether repartition or coalesce fits and what the default shuffle partition count is.
• Execution questions ask which line is an action and why a chain of transformations does nothing.

How to use these pages

Each topic page has the same shape, so you can skim for what you need.

• What it is: the idea in plain words.
• How it works: the mechanism, with tables and examples.
• What to remember for the exam: the facts to memorize and how questions are phrased.
• Common mistakes: the classic traps.
• Check yourself: three questions to test your understanding.