CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice question 24 of 25

A Spark application suffers from too many small tasks due to excessive partitioning. How can this be fixed without a full shuffle? Options:

Full set: 135 questions
Answer options
A: Use the distinct() transformation to combine similar partitions
B: Use the coalesce() transformation with a lower number of partitions Correct answer
C: Use the sortBy() transformation to reorganize the data
D: Use the repartition() transformation with a lower number of partitions
Correct Answer:
B. Use the coalesce() transformation with a lower number of partitions
Explanation

The coalesce() transformation is specifically designed to decrease the number of partitions in a DataFrame efficiently. It achieves this by combining existing partitions on the same executor, which minimizes data movement across the network and avoids a full shuffle. This makes it the ideal solution for consolidating an excessive number of small partitions into a smaller number of larger, more manageable ones, thereby reducing task overhead without incurring the high cost of a wide transformation.

Why Incorrect

A. Use the distinct() transformation to combine similar partitions The distinct() transformation is used to find unique records, which requires a full shuffle to compare all records. It is not a tool for managing partition count.
C. Use the sortBy() transformation to reorganize the data The sortBy() transformation sorts the entire dataset, which is a wide operation that necessitates a full shuffle to move data between executors.
D. Use the repartition() transformation with a lower number of partitions The repartition() transformation always triggers a full shuffle of the data across all executors, even when decreasing the partition count. This violates the core requirement of the question. ---

References

1. Databricks Documentation, "Learning Spark, 2nd Edition": In Chapter 10, "Spark SQL and Performance," the authors state, "To reduce the number of partitions, we recommend using coalesce(), which is a more efficient version of repartition() that avoids a full shuffle of the data. It does this by having a smaller number of partitions 'swallow' a larger number of partitions, with data movement minimized to only what is necessary." (Karau, H., Konwinski, A., Wendell, P., & Zaharia, M. (2020). Learning Spark, 2nd Ed. O'Reilly Media, Inc. Chapter 10, Section: "Coalesce and Repartition").

2. Apache Spark 3.5.0 Official Documentation: The documentation for the coalesce function in the pyspark.sql.DataFrame API notes its efficiency: "This operation results in a narrow dependency, e.g. if you go from 1000 partitions to 100 partitions, there will not be a shuffle, instead each of the 100 new partitions will claim 10 of the current partitions." This confirms it avoids a full shuffle. (Source: Apache Spark Project, pyspark.sql.DataFrame.coalesce API documentation).

3. Databricks Documentation, "Repartition vs Coalesce": "Coalesce uses existing partitions to minimize the amount of data that's shuffled. Repartition creates new partitions and runs a full shuffle. Coalesce is faster, but it can lead to unevenly sized partitions. Repartition is slower, but it creates evenly sized partitions." (Source: Databricks Documentation > Apache Spark > Development > Repartition).

Scroll to Top