CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice question 11 of 25
Given a DataFrame df that has 10 partitions, after running the code: result = df.coalesce(20) How many partitions will the result DataFrame have?
Full set: 135 questionsThe coalesce(n) transformation in Apache Spark is an optimized method used to decrease the number of partitions. It avoids a full data shuffle by combining existing partitions on the same worker node, making it a "narrow" transformation. A key characteristic of coalesce is that it cannot be used to increase the number of partitions. When the number n provided to coalesce is greater than the current number of partitions, the operation has no effect, and the DataFrame retains its original number of partitions. In this case, the DataFrame df has 10 partitions, and coalesce(20) is called. Since 20 is greater than 10, the number of partitions remains 10.
B. Same number as the cluster executors: The number of partitions is not automatically adjusted to match the number of executors by the coalesce operation.
C. 1: This would be the result of calling df.coalesce(1), which would collapse all partitions into a single one.
D. 20: To increase the number of partitions to 20, the repartition(20) transformation must be used, as it performs a full shuffle to redistribute the data.
1. Apache Spark Official Documentation: The pyspark.sql.DataFrame.repartition documentation clarifies its relationship with coalesce. repartition(n) is equivalent to coalesce(n
shuffle=True). The default for coalesce is shuffle=False
which prevents it from creating new partitions
thus it cannot increase the partition count.
Source: Apache Spark 3.4.1 Documentation
pyspark.sql.DataFrame.repartition.
2. Databricks Documentation: "The coalesce method reduces the number of partitions. [...] Because it doesn’t trigger a shuffle
coalesce cannot be used to increase the number of partitions."
Source: Databricks Documentation > Apache Spark > Development > Best practices: repartition and coalesce.
3. Learning Spark
2nd Edition (O'Reilly): This book
authored by Databricks engineers
is widely used in academic settings. It states
"A common question is when to use repartition() versus coalesce(). Use repartition() when you want to increase or decrease the number of partitions... Use coalesce() only when you want to decrease the number of partitions..."
Source: Chambers
B.
& Zaharia
M. (2020). Learning Spark
2nd Edition. O'Reilly Media
Inc. Chapter 4
"Spark SQL and DataFrames
" Section on "Partitions."
Keep practicing
Take the free CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice test
Ten exam-style questions with answers and explanations, plus the exam facts and study guides.
More questions
Other CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice questions
- Question 1You have: DataFrame A: 128 GB of transactions DataFrame B: 1 GB user lookup table Which s…
- Question 2A Spark DataFrame df is cached using the MEMORY_AND_DISK storage level, but the DataFrame…
- Question 344 of 55. A data engineer is working on a real-time analytics pipeline using Spark Struct…
- Question 7A developer wants to test Spark Connect with an existing Spark application. What are the …
- Question 8A data engineer is running a Spark job to process a dataset of 1 TB stored in distributed…
- Question 10A data engineer is building a Structured Streaming pipeline and wants the pipeline to rec…
