CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice question 10 of 25

A data engineer is building a Structured Streaming pipeline and wants the pipeline to recover from failures or intentional shutdowns by continuing…

Choose your answer, then check it against the explanation and the full breakdown below.

A data engineer is building a Structured Streaming pipeline and wants the pipeline to recover from failures or intentional shutdowns by continuing where the pipeline left off. How can this be achieved?

Answer options

Correct Answer:

Explanation

To ensure a Structured Streaming pipeline can recover from failures and continue from where it left off, you must enable checkpointing. This is achieved by specifying a path in a fault-tolerant file system (e.g., HDFS, S3, ADLS Gen2) using the checkpointLocation option on the DataStreamWriter (the object returned by writeStream). Spark uses this location to save all progress information for the query, including the range of offsets processed for each batch and the state of any running aggregations. When the query is restarted, it uses this checkpoint data to resume processing from the exact point it stopped, guaranteeing end-to-end, exactly-once fault tolerance.

Why Incorrect

A. By configuring the option checkpointLocation during readStream: The checkpointLocation option is a configuration for the data sink (writeStream), which manages the overall query progress, not the data source (readStream).

B. By configuring the option recoveryLocation during the SparkSession initialization: Fault tolerance is configured on a per-query basis, not globally on the SparkSession. Furthermore, recoveryLocation is not a valid configuration option for this purpose.

C. By configuring the option recoveryLocation during writeStream: The correct option name to specify the checkpoint directory for a streaming query is checkpointLocation, not recoveryLocation.

References

1. Apache Spark Official Documentation: The Structured Streaming Programming Guide explicitly states

"In case of a failure or intentional shutdown

you can recover the previous progress and state of a previous query and continue where it left off. This is done using checkpointing... you can set a checkpoint location for a query by setting the checkpointLocation option in the DataStreamWriter."

Source: Apache Spark 3.5.1 Documentation

Structured Streaming Programming Guide

Section: "Recovering from Failures with Checkpointing".

2. Databricks Official Documentation: The Databricks documentation on productionizing Structured Streaming jobs confirms this method. "To recover from failures

you must configure your streaming query with a checkpoint location... You set the path for the checkpoint using the checkpointLocation option."

Source: Databricks Documentation

Structured Streaming

Section: "Configure Structured Streaming checkpointing to recover from query failures".

3. Learning Spark

2nd Edition (O'Reilly): This book

co-authored by Databricks engineers

is a key resource. In Chapter 10

it details

"To enable this recovery

you have to set a checkpoint location for your query... You can set this location using the checkpointLocation option in the DataStreamWriter."

Source: Chambers

& Zaharia

M. (2020). Learning Spark

2nd Edition. O'Reilly Media

Inc. Chapter 10

"Structured Streaming"

p. 261

Section: "Recovering from Failures with Checkpointing".

Question 10 of 25

Question breakdown

Tests: Configure fault tolerance in Structured Streaming with checkpointing
Difficulty: Easy

Key idea: A streaming query records its offsets and state in a checkpoint directory, which you set with the checkpointLocation option on the DataStreamWriter so a restarted query resumes where it stopped.

How to reach the answer

The clue is the requirement to recover from failures or shutdowns by continuing where the pipeline left off, which describes checkpointing.
Checkpointing stores the offsets and state of a streaming query in a checkpoint directory.
The query is created when writeStream...start() is called, so checkpointLocation is set on the writer side.
recoveryLocation is not a valid Spark option, which rules out B and C.
That leaves D: set checkpointLocation on writeStream.

Each option

AIncorrect. checkpointLocation belongs to the DataStreamWriter (the query and sink side), not to the DataStreamReader source configuration.
BIncorrect. recoveryLocation is not a real option, and recovery is configured per query rather than at SparkSession initialization.
CIncorrect. writeStream is the right place, but the option is named checkpointLocation, not recoveryLocation.
DCorrect. Setting checkpointLocation on writeStream persists offsets and state so a restarted query resumes from its last committed progress.

Common trap: Option A is tempting because it sounds logical that the reader would track how far it has read from the source, but Spark tracks progress for the whole query on the writer side.

In practice: Every production streaming job, such as Kafka or Auto Loader into Delta, needs a unique, durable checkpoint path per query. Deleting or sharing that path causes reprocessing or corrupted state.

Remember: Fault-tolerant streaming means using .option("checkpointLocation", path) on writeStream, with one dedicated path per query.

Official documentation: Structured Streaming checkpoints (Databricks)

Drafted with AI assistance and checked against the answer key.

Keep practicing

Take the free CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice test

Ten exam-style questions with answers and explanations, plus the exam facts and study guides.

Start the free practice test