Structured Streaming · Topic 1 of 3
The Structured Streaming Engine, Output Modes and Sinks
Structured Streaming engine for the Databricks Spark exam: micro batches, exactly once, checkpoints, output modes, sinks, triggers and fault tolerance.
- 4 min
- A stream is treated as an unbounded table and processed in micro batches by default.
- The checkpoint stores offsets and state; reuse the same location to resume.
- Exactly once end to end needs a replayable source, checkpointing and an idempotent or transactional sink.
What it is
Structured Streaming is the Spark engine for processing data that keeps arriving. You write the query as if the input were a table that grows forever, and Spark runs it incrementally on the new rows.
This is the programming model: a stream is an unbounded input table, your query defines a result table, and each trigger updates the result and writes changes to a sink. You use the same DataFrame API as for batch.
How it works
Micro batch processing
By default the engine runs the query in micro batches. It reads a small batch of new data, processes it with ordinary Spark jobs, writes the output, and starts the next batch as soon as the previous one finishes. The default gives low latency without a fixed schedule.
The trigger option changes this:
| Trigger | Behavior |
|---|---|
| Default | Next micro batch starts when the previous one ends |
processingTime="1 minute" | A micro batch at a fixed interval |
availableNow=True | Process all data available now in one or more batches, then stop |
Exactly once and fault tolerance
Spark tracks progress in the checkpoint location: the offsets of each batch and the state of aggregations. After a failure or a restart with the same checkpoint, the query resumes from the last recorded position and replays the unfinished batch.
End to end exactly once needs three parts:
- A replayable source, so a batch can be read again from its offsets (files, Kafka).
- Checkpointing, so the engine knows which offsets it finished.
- An idempotent or transactional sink, so replaying a batch does not duplicate output. The file sink is exactly once; the Kafka sink is at least once.
Creating a streaming DataFrame
Use spark.readStream instead of spark.read. Common sources are files in a directory, Kafka and the built in rate source for testing. A file stream needs a schema unless schema inference is turned on.
df.isStreaming returns true for a streaming DataFrame.
Writing a stream
Use writeStream, then start(), which returns a StreamingQuery.
stream = (spark.readStream.format("rate")
.option("rowsPerSecond", 10).load())
query = (stream.writeStream.format("parquet")
.option("path", "/out/events")
.option("checkpointLocation", "/chk/events")
.outputMode("append").start())Nothing runs until start(). query.awaitTermination() blocks the program while the query runs, and query.stop() ends it. A streaming DataFrame cannot use show() or collect(); use the console sink for debugging.
Output modes
| Mode | Writes | Needs |
|---|---|---|
| append (default) | Only new rows, each once | Rows that never change later |
| update | Only rows changed since the last trigger | Nothing extra; aggregates write only changed groups |
| complete | The whole result table every trigger | An aggregation |
Aggregations without a watermark cannot use append mode, because their results can still change. Complete mode keeps all state, so it suits small result tables.
Output sinks
| Sink | Modes | Use |
|---|---|---|
| File (parquet, json, csv, orc, text) | append | Exactly once file output |
| Kafka | append, update, complete | Publish to a topic, at least once |
| foreach and foreachBatch | append, update, complete | Custom logic per row or per micro batch |
| Console | all | Debugging only |
| Memory | append, complete | Debugging only |
How the exam asks about this
- "A streaming aggregation must write the full result each trigger." Use complete output mode.
- "The job restarts after a crash and must not reprocess finished data." Set a checkpointLocation and reuse it.
- "Which sink is for debugging and not fault tolerant?" The console (or memory) sink.
What to remember for the exam
- A stream is treated as an unbounded table and processed in micro batches by default.
- The checkpoint stores offsets and state; reuse the same location to resume.
- Exactly once end to end needs a replayable source, checkpointing and an idempotent or transactional sink.
- append is the default mode; complete requires an aggregation.
- The file sink supports only append; console and memory sinks are for debugging.
- Nothing runs until
start();awaitTermination()keeps the application alive.
Do not pick complete mode for a plain select or filter stream. Complete mode needs an aggregation, so the query fails.
Common mistakes
Forgetting the checkpoint location. Without it a query cannot recover its position after a restart. Each query needs its own location; sharing one between queries corrupts progress.
Calling batch actions on a stream. show(), count() and collect() are not allowed on a streaming DataFrame. Write to a sink instead.
Assuming Kafka output is exactly once. The Kafka sink is at least once, so duplicates are possible after a retry unless you deduplicate downstream.
Check yourself
What does the checkpoint location store?
The offsets processed in each batch and the state of stateful operations, so a restarted query resumes where it stopped.
Which output mode requires an aggregation?
Complete mode, which rewrites the whole result table at every trigger.
Name the three requirements for end to end exactly once.
A replayable source, checkpointing, and an idempotent or transactional sink.
Topic 1 of 3 in Structured Streaming
Test yourself on Structured Streaming
The full Databricks Associate Developer for Apache Spark question bank has 600 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Databricks. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.