Databricks Associate Developer for Apache Spark Practice Exam

• A. Calling join() with another DataFrame on a key
• B. Calling count() on the DataFrame Correct answer
• C. Calling select() with two column names
• D. Calling filter() with a column condition

Option B is correct because count() is an action, so it forces Spark to execute the plan and run a job. join(), select() and filter() are lazy transformations that only extend the plan until an action is called.

• A. The select that keeps fewer columns
• B. The shuffle needed to group rows by country Correct answer
• C. The filter that removes rows from input partitions
• D. The collect() call that returns rows to the driver

Option B is correct because groupBy("country").count() needs all rows with the same country in the same partition, which requires a shuffle, and a shuffle splits the job into two stages. filter() and select() are narrow transformations that run in the same stage as the file read, and collect() is the action that triggers the job rather than a stage boundary.

• A. df.persist(StorageLevel.MEMORY_ONLY_2)
• B. df.persist(StorageLevel.MEMORY_AND_DISK) Correct answer
• C. df.persist(StorageLevel.DISK_ONLY)
• D. df.persist(StorageLevel.MEMORY_ONLY)

Option B is correct because MEMORY_AND_DISK keeps the partitions that fit in memory and spills the rest to disk, so repeated actions never go back to the slow JDBC source. MEMORY_ONLY would drop the partitions that do not fit and recompute them from the source, MEMORY_ONLY_2 also needs a second replica in memory, and DISK_ONLY gives up the faster in-memory reads.

• A. df.write.mode("append").format("parquet").save("/mnt/out")
• B. df.write.format("parquet").save("/mnt/out")
• C. df.write.mode("overwrite").parquet("/mnt/out") Correct answer
• D. df.write.mode("ignore").parquet("/mnt/out")

Option C is correct because mode("overwrite") replaces any existing data at the path before writing the Parquet files. Option A adds new files next to the existing data, option D skips the write when data already exists, and option B uses the default mode, which raises an error if the path already exists.

• A. sales_df.write.partitionBy(16, "customer_id").saveAsTable("sales")
• B. sales_df.write.bucketBy("customer_id").sortBy("customer_id").saveAsTable("sales")
• C. sales_df.write.bucketBy(16, "customer_id").sortBy("customer_id").saveAsTable("sales") Correct answer
• D. sales_df.write.bucketBy(16, "customer_id").sortBy("customer_id").parquet("/data/sales")

Option C is correct because bucketBy(numBuckets, col) takes the bucket count first, sortBy sorts within each bucket, and saveAsTable stores the bucketing in the table metadata so later joins on customer_id can avoid a shuffle. Option A uses partitionBy, which does not take a bucket count, option B omits the number of buckets, and option D writes to a path with parquet() instead of saving a table.

• A. Register it with createTempView instead, which scopes the view to the application
• B. Call spark.catalog.cacheTable("orders") so the cached data is shared across sessions
• C. Register it with createGlobalTempView and query it as global_temp.orders Correct answer
• D. Keep createOrReplaceTempView and query it as default.orders from the new session

Option C is correct because a global temporary view is shared by all sessions of the same Spark application and is queried through the global_temp database as global_temp.orders. A view created with createOrReplaceTempView or createTempView belongs only to the session that created it, and caching a table does not make its view visible to another session.

• A. df.withColumnRenamed("customer_name", "cust_nm")
• B. df.withColumnRenamed("cust_nm", "customer_name") Correct answer
• C. df.withColumn("cust_nm", "customer_name")
• D. df.rename("cust_nm", "customer_name")

Option B is correct because withColumnRenamed takes the existing column name first and the new name second, and it leaves all other columns unchanged. Option A reverses the arguments, option C is withColumn, which expects a Column expression as its second argument rather than a name, and option D uses a rename method that DataFrame does not have.

• A. df.dropDuplicates(["event_id"]) Correct answer
• B. df.dropna(subset=["event_id"])
• C. df.distinct()
• D. df.select("event_id").distinct()

Option A is correct because dropDuplicates(["event_id"]) compares rows on event_id only and keeps one full row per value, so all four columns are retained even though ts differs. distinct() compares whole rows, so re-delivered events with a different ts all stay, dropna() removes rows with nulls rather than duplicates, and selecting only event_id loses the other columns.

• A. The driver sends an updated copy to all executors after each task completes
• B. Tasks on executors can read its running value to branch their logic
• C. Updates made inside transformations are applied exactly once, even if a task is retried
• D. Tasks on executors add to it, and the driver reads its value Correct answer

Option D is correct because tasks on executors can only add to an accumulator, and only the driver program can read its value. Tasks cannot read the running value, and Spark does not send updated copies back to executors after each task. Updates are guaranteed to be applied exactly once only inside actions, while updates made inside transformations may be applied more than once if tasks or stages are re-executed.

• A. Call repartitionByRange(8, "ts") on the DataFrame before the write
• B. Call coalesce(8) on the DataFrame before the write Correct answer
• C. Set spark.sql.shuffle.partitions to 8 before the write
• D. Call repartition(8) on the DataFrame before the write

Option B is correct because coalesce(8) merges existing partitions into fewer partitions without a full shuffle, so the write produces about 8 files. repartition(8) and repartitionByRange(8, "ts") both perform a full shuffle, and spark.sql.shuffle.partitions only affects the partition count after later shuffles, not the number of files written from the existing 2,000 partitions.

• A. Enable AQE with a high initial shuffle partition count so partitions coalesce at runtime Correct answer
• B. Set spark.sql.shuffle.partitions to 20 so small days run with fewer, larger tasks
• C. Set spark.sql.shuffle.partitions to 2000 so partitions stay small enough on the largest days
• D. Call repartition(200) on the input before each aggregation so partitions are evenly sized

Option A is correct because with adaptive query execution enabled, Spark can start with a high number of shuffle partitions and coalesce small ones at runtime based on the actual data size, so the small days get few tasks and the large days get enough partitions to avoid spilling. A fixed value such as 20 or 2000 fixes one of the two cases and worsens the other, and repartition(200) just keeps the same fixed count.

• A. Spark 3.5
• B. Spark 2.4
• C. Spark 3.0
• D. Spark 3.2 Correct answer

Option D is correct because adaptive query execution has been enabled by default (spark.sql.adaptive.enabled=true) since Apache Spark 3.2.0. It was introduced in 3.0 but was off by default there, and it stayed off in 3.1. Spark 2.4 predates AQE entirely.

• A. Update mode
• B. Complete mode Correct answer
• C. Continuous mode
• D. Append mode

Option B is correct because complete mode writes the whole updated result table, here the counts for all pages, on every trigger. Update mode emits only the rows that changed, and append mode is not supported for an aggregation without a watermark. Continuous is a trigger type for low-latency processing, not an output mode.

• A. A StreamingQueryManager bound to the session
• B. A DataFrame with the first micro-batch rows
• C. A StreamingQuery handle for monitoring and stopping Correct answer
• D. None, because the query runs detached

Option C is correct because start() launches the query in the background and returns a StreamingQuery object, which you use to check status, read progress, await termination or call stop(). Option A is wrong because the StreamingQueryManager comes from spark.streams, not from start(). Option B is wrong because a streaming DataFrame cannot be materialized as rows this way. Option D is wrong because start() does return a handle even though the query runs in the background.

• A. The memory sink
• B. The foreachBatch sink
• C. The console sink Correct answer
• D. The parquet file sink

Option C is correct because the console sink prints the output of each micro-batch to standard output and is meant for debugging, not production. The memory sink (A) stores results in an in-memory table that can be queried by name. The foreachBatch sink (B) runs your own function on each batch, and the parquet sink (D) writes files.

• A. Standalone mode
• B. Cluster mode Correct answer
• C. Local mode
• D. Client mode

Option B is correct because in cluster mode the driver is launched on a node inside the cluster, whereas in client mode the driver runs on the machine that ran the submit command. Standalone is a cluster manager type rather than a deploy mode, and local mode runs everything in a single process on the local machine.

• A. In the stdout log of the ApplicationMaster container that hosts the driver, on a cluster node Correct answer
• B. In the terminal of the edge node that ran spark-submit, as the job completes
• C. In the stdout logs of the executor containers that counted the rows of df
• D. In the event log that the Spark History Server shows for the finished application

Option A is correct because with --deploy-mode cluster the driver runs inside the ApplicationMaster container on a cluster node, so print output goes to that container's stdout log, not to the submitting terminal. Option B describes client mode. Option C is wrong because the print runs on the driver; executors only count partitions. Option D is wrong because the event log records scheduler events, not application stdout.

• A. gRPC Correct answer
• B. Py4J sockets
• C. Thrift over HTTP
• D. JDBC

Option A is correct because Spark Connect is a client-server design that sends requests over gRPC, using protobuf messages. Py4J sockets are the classic PySpark bridge to the driver JVM, which Connect replaces. Thrift over HTTP and JDBC are used by the Thrift server and SQL clients, not by Connect.

• A. @pandas_udf("double") def add_tax(s: float) -> float: return s * 1.2
• B. @pandas_udf("double") def add_tax(s: pd.Series) -> pd.Series: return s * 1.2 Correct answer
• C. @udf("double") def add_tax(s: pd.Series) -> pd.Series: return s * 1.2
• D. @pandas_udf("double") def add_tax(df: pd.DataFrame) -> pd.DataFrame: return df * 1.2

Option B is correct because a Series to Series pandas UDF is declared with @pandas_udf and the type hints pd.Series -> pd.Series, so Spark passes the column in as batches of values and expects a Series of the same length back. Option C uses @udf, which runs row by row. Option A takes and returns a single float rather than batches, and option D takes and returns a pd.DataFrame, which does not fit a single double column.

• A. JSON lines sent over a local socket
• B. Parquet files written to executor local disk
• C. Apache Arrow columnar record batches Correct answer
• D. Pickled Python objects, one row at a time

Option C is correct because Spark uses Apache Arrow to transfer data between the JVM and the Python worker in columnar record batches, which is what lets a pandas UDF work on whole Series instead of single rows. Option D describes the slow row-at-a-time pickle path of ordinary Python UDFs, and JSON lines or Parquet files on local disk are not used for this transfer.