CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice question 8 of 25
A data engineer is running a Spark job to process a dataset of 1 TB stored in distributed storage. The cluster has 10 nodes, each with 16 CPUs. Spark UI shows: Low number of Active Tasks Many tasks complete in milliseconds Fewer tasks than available CPUs Which approach should be used to adjust the partitioning for optimal resource allocation?
Full set: 135 questionsThe symptoms described—low number of active tasks and fewer tasks than available CPUs—are classic indicators of under-partitioning. With 160 available CPU cores (10 nodes 16 CPUs), the cluster is significantly underutilized. The most effective and scalable approach to resolve this is to determine the number of partitions based on the data size. By dividing the total dataset size (1 TB) by a recommended partition size (e.g., 128 MB), you ensure that each task processes a manageable chunk of data. This creates a high number of partitions (approx. 8,192), allowing Spark to achieve maximum parallelism by keeping all CPU cores busy, thus optimizing resource allocation and job performance.
A. This method ignores the data size. For a 1 TB dataset, 160 partitions would be ~6.25 GB each, which is excessively large and likely to cause out-of-memory (OOM) errors.
B. Using a fixed, arbitrary value like 200 is not a best practice. It fails to account for the specific characteristics of the dataset size and cluster resources, leading to suboptimal performance.
C. This would result in only 10 partitions for 160 cores, leading to extreme under-utilization and unmanageably large partitions (~100 GB each), which would certainly fail.
1. Book: "Spark: The Definitive Guide" by Bill Chambers and Matei Zaharia (creator of Apache Spark).
Reference: Part V
Chapter 17: Performance Tuning. The chapter explains that the number of partitions is the key to parallelism. For input data from file-based sources
Spark by default creates one partition per block of the file in HDFS (defaulting to 128 MB). This establishes 128 MB as a sound
fundamental partition size. The strategy in option D directly applies this principle.
2. Book: "Learning Spark
2nd Edition" by Jules S. Damji
et al.
Reference: Chapter 10
"Performance Tuning
" section on "Number of Partitions." The authors state
"For a large dataset of 1 TB
you would want more partitions... A good starting point is to have partitions of size 128 MB." This directly endorses the calculation method described in option D as a best practice for large datasets.
3. Databricks Documentation
"Best practices: performance tuning".
Reference: In sections discussing optimization
Databricks frequently advises on ideal file and partition sizes. The general guidance is to aim for partition sizes between 128 MB and 1 GB. Option D's approach of calculating the partition count to achieve a size of 128 MB aligns perfectly with this official vendor recommendation for optimal performance.
Question breakdown
Key idea: Size partitions from the data volume (about 128 MB each) so there are many more tasks than cores, keeping every core busy while each task stays manageable.
How to reach the answer
- The symptoms are few active tasks and fewer tasks than the 160 available cores, which points to too few partitions for the cluster.
- A count tied only to hardware (160 or 10 partitions) ignores the 1 TB size and produces multi-GB partitions.
- A fixed value such as 200 also ignores data size, giving roughly 5 GB per partition.
- 1 TB divided by 128 MB gives about 8,192 partitions, which run in many waves across 160 cores at a safe per-task size.
Each option
Common trap: Option A sounds like a natural match of one task per core, but it ignores how much data each task would have to process.
In practice: Engineers tune spark.sql.files.maxPartitionBytes, spark.sql.shuffle.partitions, or repartition() based on data volume, and rely on AQE to coalesce small shuffle partitions.
Remember: Derive the partition count from data size divided by a target partition size, not from core or node counts alone.
Drafted with AI assistance and checked against the answer key.
Keep practicing
Take the free CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice test
Ten exam-style questions with answers and explanations, plus the exam facts and study guides.
More questions
Other CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice questions
- Question 1You have: DataFrame A: 128 GB of transactions DataFrame B: 1 GB user lookup table Which s…
- Question 2A Spark DataFrame df is cached using the MEMORY_AND_DISK storage level, but the DataFrame…
- Question 344 of 55. A data engineer is working on a real-time analytics pipeline using Spark Struct…
- Question 7A developer wants to test Spark Connect with an existing Spark application. What are the …
- Question 10A data engineer is building a Structured Streaming pipeline and wants the pipeline to rec…
- Question 11Given a DataFrame df that has 10 partitions, after running the code: result = df.coalesce…
