CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice question 10 of 25
A data engineer is building a Structured Streaming pipeline and wants the pipeline to recover from failures or intentional shutdowns by continuing where the pipeline left off. How can this be achieved?
Full set: 135 questionsTo ensure a Structured Streaming pipeline can recover from failures and continue from where it left off, you must enable checkpointing. This is achieved by specifying a path in a fault-tolerant file system (e.g., HDFS, S3, ADLS Gen2) using the checkpointLocation option on the DataStreamWriter (the object returned by writeStream). Spark uses this location to save all progress information for the query, including the range of offsets processed for each batch and the state of any running aggregations. When the query is restarted, it uses this checkpoint data to resume processing from the exact point it stopped, guaranteeing end-to-end, exactly-once fault tolerance.
A. By configuring the option checkpointLocation during readStream: The checkpointLocation option is a configuration for the data sink (writeStream), which manages the overall query progress, not the data source (readStream).
B. By configuring the option recoveryLocation during the SparkSession initialization: Fault tolerance is configured on a per-query basis, not globally on the SparkSession. Furthermore, recoveryLocation is not a valid configuration option for this purpose.
C. By configuring the option recoveryLocation during writeStream: The correct option name to specify the checkpoint directory for a streaming query is checkpointLocation, not recoveryLocation.
1. Apache Spark Official Documentation: The Structured Streaming Programming Guide explicitly states
"In case of a failure or intentional shutdown
you can recover the previous progress and state of a previous query and continue where it left off. This is done using checkpointing... you can set a checkpoint location for a query by setting the checkpointLocation option in the DataStreamWriter."
Source: Apache Spark 3.5.1 Documentation
Structured Streaming Programming Guide
Section: "Recovering from Failures with Checkpointing".
2. Databricks Official Documentation: The Databricks documentation on productionizing Structured Streaming jobs confirms this method. "To recover from failures
you must configure your streaming query with a checkpoint location... You set the path for the checkpoint using the checkpointLocation option."
Source: Databricks Documentation
Structured Streaming
Section: "Configure Structured Streaming checkpointing to recover from query failures".
3. Learning Spark
2nd Edition (O'Reilly): This book
co-authored by Databricks engineers
is a key resource. In Chapter 10
it details
"To enable this recovery
you have to set a checkpoint location for your query... You can set this location using the checkpointLocation option in the DataStreamWriter."
Source: Chambers
B.
& Zaharia
M. (2020). Learning Spark
2nd Edition. O'Reilly Media
Inc. Chapter 10
"Structured Streaming"
p. 261
Section: "Recovering from Failures with Checkpointing".
Question breakdown
Key idea: A streaming query records its offsets and state in a checkpoint directory, which you set with the checkpointLocation option on the DataStreamWriter so a restarted query resumes where it stopped.
How to reach the answer
- The clue is the requirement to recover from failures or shutdowns by continuing where the pipeline left off, which describes checkpointing.
- Checkpointing stores the offsets and state of a streaming query in a checkpoint directory.
- The query is created when writeStream...start() is called, so checkpointLocation is set on the writer side.
- recoveryLocation is not a valid Spark option, which rules out B and C.
- That leaves D: set checkpointLocation on writeStream.
Each option
Common trap: Option A is tempting because it sounds logical that the reader would track how far it has read from the source, but Spark tracks progress for the whole query on the writer side.
In practice: Every production streaming job, such as Kafka or Auto Loader into Delta, needs a unique, durable checkpoint path per query. Deleting or sharing that path causes reprocessing or corrupted state.
Remember: Fault-tolerant streaming means using .option("checkpointLocation", path) on writeStream, with one dedicated path per query.
Official documentation: Structured Streaming checkpoints (Databricks)
Drafted with AI assistance and checked against the answer key.
Keep practicing
Take the free CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice test
Ten exam-style questions with answers and explanations, plus the exam facts and study guides.
More questions
Other CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice questions
- Question 1You have: DataFrame A: 128 GB of transactions DataFrame B: 1 GB user lookup table Which s…
- Question 2A Spark DataFrame df is cached using the MEMORY_AND_DISK storage level, but the DataFrame…
- Question 344 of 55. A data engineer is working on a real-time analytics pipeline using Spark Struct…
- Question 7A developer wants to test Spark Connect with an existing Spark application. What are the …
- Question 8A data engineer is running a Spark job to process a dataset of 1 TB stored in distributed…
- Question 11Given a DataFrame df that has 10 partitions, after running the code: result = df.coalesce…
