CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice question 3 of 25
44 of 55. A data engineer is working on a real-time analytics pipeline using Spark Structured Streaming. They want the system to process incoming data in micro-batches at a fixed interval of 5 seconds. Which code snippet fulfills this requirement?
Full set: 135 questionsThe trigger(processingTime="5 seconds") configuration explicitly sets up a micro-batch processing model where the Spark engine initiates a new batch at a fixed interval. This means the query will check for new data and start processing it every 5 seconds, regardless of whether the previous batch has finished. This directly fulfills the requirement for a real-time analytics pipeline that processes data in micro-batches at a fixed 5-second interval.
B. trigger(continuous="5 seconds") enables continuous processing, a low-latency mode, not the requested fixed-interval micro-batch model. The interval here is for checkpointing, not batching.
C. trigger(once=True) processes only a single batch of all available data and then stops the query, which is not suitable for a continuous, real-time pipeline.
D. The default trigger (no trigger specified) starts a new micro-batch immediately after the previous one completes. This does not guarantee a fixed interval; the timing depends on processing duration.
1. Apache Spark 3.5.0 Documentation
Structured Streaming Programming Guide
Section: Triggers.
"If you specify a processing time trigger
Spark will check for new data when the previous processing has completed. If new data is available
Spark will start a new processing. If no new data is available
Spark will sleep until the processing time is reached
and then check for new data again." This describes the behavior of Trigger.ProcessingTime("5 seconds")
confirming option A. The documentation also explicitly contrasts this with the default behavior (option D)
Trigger.Once() (option C)
and Trigger.Continuous() (option B).
2. Databricks Documentation
Structured Streaming
"Configure Structured Streaming trigger intervals".
Under the heading "Fixed interval micro-batches
" the documentation states: "You can configure structured streaming queries to run with a fixed batch interval using processingTime. If you do not specify a processingTime trigger
queries run using the default micro-batch setting...". This directly validates that processingTime is the correct parameter for achieving a fixed interval.
Question breakdown
Key idea: trigger(processingTime="5 seconds") runs micro-batches on a fixed 5-second schedule.
How to reach the answer
- The deciding clues are 'micro-batches' and 'fixed interval of 5 seconds'.
- processingTime is the trigger type that schedules micro-batches at a fixed interval.
- continuous is a different, non-micro-batch execution mode, and its interval is a checkpoint interval.
- once=True runs a single batch and stops, and leaving out the trigger starts each batch as soon as the previous one finishes.
Each option
Common trap: Option B also contains "5 seconds", so it looks equivalent, but the continuous trigger is not micro-batch execution and the interval means something different.
In practice: Fixed processingTime intervals are used to control cost and batch sizes for sinks such as Delta tables. availableNow is the usual choice for scheduled incremental runs.
Remember: Use processingTime for fixed-interval micro-batches, continuous for low-latency non-batch mode, and once or availableNow for one-off runs.
Official documentation: Configure Structured Streaming trigger intervals
Drafted with AI assistance and checked against the answer key.
Keep practicing
Take the free CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice test
Ten exam-style questions with answers and explanations, plus the exam facts and study guides.
More questions
Other CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice questions
- Question 1You have: DataFrame A: 128 GB of transactions DataFrame B: 1 GB user lookup table Which s…
- Question 2A Spark DataFrame df is cached using the MEMORY_AND_DISK storage level, but the DataFrame…
- Question 7A developer wants to test Spark Connect with an existing Spark application. What are the …
- Question 8A data engineer is running a Spark job to process a dataset of 1 TB stored in distributed…
- Question 10A data engineer is building a Structured Streaming pipeline and wants the pipeline to rec…
- Question 11Given a DataFrame df that has 10 partitions, after running the code: result = df.coalesce…
