Troubleshooting and Tuning · Topic 2 of 2
Logging and Monitoring Spark Applications
Read driver and executor logs and the Spark UI to diagnose out of memory errors and cluster underutilization, and how Databricks delivers compute logs.
- 4 min
- Driver log shows application level failures; executor logs show task level failures.
- Output from code running inside UDFs goes to executor logs.
setLogLevelchanges log verbosity; event logs and a history server keep finished jobs viewable.
What it is
When a Spark job fails or runs slowly, the answer is in two places: the logs of the driver and the executors, and the Spark UI. The driver log shows what the application itself did. The executor logs show what each task did on the workers.
Monitoring means watching those sources while a job runs. Logging means producing and keeping them, so you can still analyze a job after the cluster is gone.
How it works
Driver logs and executor logs
| Source | Holds | Typical question it answers |
|---|---|---|
| Driver log | Application errors, stack traces, your own log and print output on the driver | Why did the job fail? |
| Executor log (stdout, stderr) | Task level errors, JVM errors, output from code inside UDFs | Why did a task die? |
| Spark UI | Jobs, stages, tasks, storage, executors, SQL plans | Where is time and memory going? |
Code that runs inside a UDF or a function applied on the workers prints to the executor logs, not to the driver. A print in driver code appears in the driver output.
Customizing and publishing logs
Spark uses log4j. In a session you can change verbosity with spark.sparkContext.setLogLevel("WARN"), which affects the logging of that application. Python code on the driver can also use the standard logging module.
Executor logs disappear with the cluster unless they are kept. For finished applications, enable event logging with spark.eventLog.enabled and spark.eventLog.dir, then read them in the Spark history server. On Databricks, set a log path under the compute Advanced options, Logging tab, and the platform delivers driver and worker logs there every five minutes. The destination can be a Unity Catalog volume.
Diagnosing out of memory errors
| Symptom | Likely cause | Typical fix |
|---|---|---|
Driver fails on collect() or toPandas() | Result too large for the driver | Aggregate or limit first, write to storage, raise driver memory |
Executor OutOfMemoryError on one task | Skewed or oversized partition | Repartition, fix skew, raise executor memory |
| Long GC time and heavy spill | Partitions too big for execution memory | More partitions, less cached data |
| Container killed for exceeding memory | Off heap use, often Python workers | Raise memory overhead |
In the stage page of the Spark UI, spill to memory and disk and a high GC time per task point to memory pressure. Compare the maximum task value with the median to tell skew from a general lack of memory.
Diagnosing cluster underutilization
Underutilization means executors sit idle. Look at the Executors tab for cores that never receive tasks, and at the stage page for the number of tasks.
- Fewer tasks than cores: too few partitions. Increase them with
repartition. - One long task at the end: skew. Most cores finish and wait.
- Driver busy while executors idle: work done on the driver, such as a large
collect()or a Python loop over rows. - Idle gaps between jobs: time spent in driver side code or slow reads.
df = spark.read.parquet("/data/events")
print(df.rdd.getNumPartitions())
spark.sparkContext.setLogLevel("WARN")How the exam asks about this
- "A task fails with OutOfMemoryError only for one key." The partition is skewed; rebalance or fix the key distribution.
- "The driver crashes after
df.collect()." The result is too big for the driver; aggregate first or write it out. - "Where do you see errors raised inside a UDF?" In the executor logs, reachable from the Spark UI Executors tab.
What to remember for the exam
- Driver log shows application level failures; executor logs show task level failures.
- Output from code running inside UDFs goes to executor logs.
setLogLevelchanges log verbosity; event logs and a history server keep finished jobs viewable.- Databricks delivers configured compute logs to the chosen destination every five minutes.
- Spill, high GC time and one oversized task indicate memory pressure or skew.
- Too few partitions or driver side work leaves executors idle.
More executor memory is not the fix for a driver out of memory error. The driver and executors have separate memory settings, so look at which one actually failed.
Common mistakes
Reading only the driver log. A failed job often shows only "Job aborted" on the driver. The real cause, such as an executor out of memory error, is in the executor log of the failed task.
Collecting to debug. Calling collect() on a large DataFrame to inspect it can itself cause the driver crash. Use limit or show for samples.
Ignoring idle cores. A cluster with many cores but a few partitions cannot use them. Check the task count before scaling the cluster.
Check yourself
Where does a
printinside a UDF appear?In the executor logs, because the function runs on the workers, not on the driver.
Which settings keep a finished application viewable?
Event logging with
spark.eventLog.enabledand a log directory, read through the Spark history server.A cluster has 64 cores but a stage has 8 tasks. What is the likely fix?
Increase the partitions, for example with repartition, so more tasks run in parallel.
Topic 2 of 2 in Troubleshooting and Tuning
Test yourself on Troubleshooting and Tuning
The full Databricks Associate Developer for Apache Spark question bank has 600 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Databricks. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.