CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice question 19 of 25

39 of 55. A Spark developer is developing a Spark application to monitor task performance across a cluster. One requirement is to track the maximum processing time for tasks on each worker node and consolidate this information on the driver for further analysis. Which technique should the developer use?

Full set: 135 questions
Answer options
A: Broadcast a variable to share the maximum time among workers.
B: Configure the Spark UI to automatically collect maximum times.
C: Use an RDD action like reduce() to compute the maximum time. Correct answer
D: Use an accumulator to record the maximum time on the driver.
Correct Answer:
C. Use an RDD action like reduce() to compute the maximum time.
Explanation

An RDD (or Dataset) action such as reduce() executes on every partition, applies a commutative/associative function (e.g., max), and returns the single aggregated result to the driver. This exactly satisfies “track the maximum processing time … and consolidate this information on the driver.”

Why Incorrect

A. Broadcast variables are read-only values sent from the driver to executors; they cannot collect or update data back to the driver.
B. The Spark UI passively displays metrics; it is not a programmatic technique for collecting values inside an application.
D. Standard accumulators are intended for sums/counters and are write-only from tasks; although custom AccumulatorV2 can implement max, Spark docs discourage relying on accumulators to retrieve computation results.

References

1. Apache Spark 3.4.2 Programming Guide – Actions: reduce (returns aggregated value to driver) (docs/latest/rdd-programming-guide.html#actions).

2. Apache Spark 3.4.2 Programming Guide – Broadcast Variables (read-only, not for returning results) (docs/latest/rdd-programming-guide.html#broadcast-variables).

3. Apache Spark 3.4.2 Programming Guide – Accumulators (write-only from tasks; recommended only for debugging) (docs/latest/rdd-programming-guide.html#accumulators).

Scroll to Top