A data engineer is building a Structured Streaming pipeline and wants the pipeline to recover from failures or intentional shutdowns by continuing…

To ensure a Structured Streaming pipeline can recover from failures and continue from where it left off, you must enable checkpointing. This is achieved by specifying a path in a fault-tolerant file system (e.g., HDFS, S3, ADLS Gen2) using the checkpointLocation option on the DataStreamWriter (the object returned by writeStream). Spark uses this location to save all progress information for the query, including the range of offsets processed for each batch and the state of any running aggregations. When the query is restarted, it uses this checkpoint data to resume processing from the exact point it stopped, guaranteeing end-to-end, exactly-once fault tolerance.

A. By configuring the option checkpointLocation during readStream: The checkpointLocation option is a configuration for the data sink (writeStream), which manages the overall query progress, not the data source (readStream).

B. By configuring the option recoveryLocation during the SparkSession initialization: Fault tolerance is configured on a per-query basis, not globally on the SparkSession. Furthermore, recoveryLocation is not a valid configuration option for this purpose.

C. By configuring the option recoveryLocation during writeStream: The correct option name to specify the checkpoint directory for a streaming query is checkpointLocation, not recoveryLocation.

1. Apache Spark Official Documentation: The Structured Streaming Programming Guide explicitly states

"In case of a failure or intentional shutdown

you can recover the previous progress and state of a previous query and continue where it left off. This is done using checkpointing... you can set a checkpoint location for a query by setting the checkpointLocation option in the DataStreamWriter."

Source: Apache Spark 3.5.1 Documentation

Structured Streaming Programming Guide

Section: "Recovering from Failures with Checkpointing".

2. Databricks Official Documentation: The Databricks documentation on productionizing Structured Streaming jobs confirms this method. "To recover from failures

you must configure your streaming query with a checkpoint location... You set the path for the checkpoint using the checkpointLocation option."

Source: Databricks Documentation

Structured Streaming

Section: "Configure Structured Streaming checkpointing to recover from query failures".

3. Learning Spark

2nd Edition (O'Reilly): This book

co-authored by Databricks engineers

is a key resource. In Chapter 10

it details

"To enable this recovery

you have to set a checkpoint location for your query... You can set this location using the checkpointLocation option in the DataStreamWriter."

Source: Chambers

B.

& Zaharia

M. (2020). Learning Spark

2nd Edition. O'Reilly Media

Inc. Chapter 10

"Structured Streaming"

p. 261

Section: "Recovering from Failures with Checkpointing".