Given a DataFrame df that has 10 partitions, after running the code: result = df.coalesce(20) How many partitions will the result DataFrame have?

The coalesce(n) transformation in Apache Spark is an optimized method used to decrease the number of partitions. It avoids a full data shuffle by combining existing partitions on the same worker node, making it a "narrow" transformation. A key characteristic of coalesce is that it cannot be used to increase the number of partitions. When the number n provided to coalesce is greater than the current number of partitions, the operation has no effect, and the DataFrame retains its original number of partitions. In this case, the DataFrame df has 10 partitions, and coalesce(20) is called. Since 20 is greater than 10, the number of partitions remains 10.

B. Same number as the cluster executors: The number of partitions is not automatically adjusted to match the number of executors by the coalesce operation.

C. 1: This would be the result of calling df.coalesce(1), which would collapse all partitions into a single one.

D. 20: To increase the number of partitions to 20, the repartition(20) transformation must be used, as it performs a full shuffle to redistribute the data.

1. Apache Spark Official Documentation: The pyspark.sql.DataFrame.repartition documentation clarifies its relationship with coalesce. repartition(n) is equivalent to coalesce(n

shuffle=True). The default for coalesce is shuffle=False

which prevents it from creating new partitions

thus it cannot increase the partition count.

Source: Apache Spark 3.4.1 Documentation

pyspark.sql.DataFrame.repartition.

2. Databricks Documentation: "The coalesce method reduces the number of partitions. [...] Because it doesn’t trigger a shuffle

coalesce cannot be used to increase the number of partitions."

Source: Databricks Documentation > Apache Spark > Development > Best practices: repartition and coalesce.

3. Learning Spark

2nd Edition (O'Reilly): This book

authored by Databricks engineers

is widely used in academic settings. It states

"A common question is when to use repartition() versus coalesce(). Use repartition() when you want to increase or decrease the number of partitions... Use coalesce() only when you want to decrease the number of partitions..."

Source: Chambers

B.

& Zaharia

M. (2020). Learning Spark

2nd Edition. O'Reilly Media

Inc. Chapter 4

"Spark SQL and DataFrames

" Section on "Partitions."