CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice question 8 of 25

A data engineer is running a Spark job to process a dataset of 1 TB stored in distributed storage. The cluster has 10 nodes, each with 16 CPUs. Spark…

Choose your answer, then check it against the explanation and the full breakdown below.

A data engineer is running a Spark job to process a dataset of 1 TB stored in distributed storage. The cluster has 10 nodes, each with 16 CPUs. Spark UI shows: Low number of Active Tasks Many tasks complete in milliseconds Fewer tasks than available CPUs Which approach should be used to adjust the partitioning for optimal resource allocation?

Answer options

Correct Answer:

Explanation

The symptoms described—low number of active tasks and fewer tasks than available CPUs—are classic indicators of under-partitioning. With 160 available CPU cores (10 nodes 16 CPUs), the cluster is significantly underutilized. The most effective and scalable approach to resolve this is to determine the number of partitions based on the data size. By dividing the total dataset size (1 TB) by a recommended partition size (e.g., 128 MB), you ensure that each task processes a manageable chunk of data. This creates a high number of partitions (approx. 8,192), allowing Spark to achieve maximum parallelism by keeping all CPU cores busy, thus optimizing resource allocation and job performance.

Why Incorrect

A. This method ignores the data size. For a 1 TB dataset, 160 partitions would be ~6.25 GB each, which is excessively large and likely to cause out-of-memory (OOM) errors.

B. Using a fixed, arbitrary value like 200 is not a best practice. It fails to account for the specific characteristics of the dataset size and cluster resources, leading to suboptimal performance.

C. This would result in only 10 partitions for 160 cores, leading to extreme under-utilization and unmanageably large partitions (~100 GB each), which would certainly fail.

References

1. Book: "Spark: The Definitive Guide" by Bill Chambers and Matei Zaharia (creator of Apache Spark).

Reference: Part V

Chapter 17: Performance Tuning. The chapter explains that the number of partitions is the key to parallelism. For input data from file-based sources

Spark by default creates one partition per block of the file in HDFS (defaulting to 128 MB). This establishes 128 MB as a sound

fundamental partition size. The strategy in option D directly applies this principle.

2. Book: "Learning Spark

2nd Edition" by Jules S. Damji

et al.

Reference: Chapter 10

"Performance Tuning

" section on "Number of Partitions." The authors state

"For a large dataset of 1 TB

you would want more partitions... A good starting point is to have partitions of size 128 MB." This directly endorses the calculation method described in option D as a best practice for large datasets.

3. Databricks Documentation

"Best practices: performance tuning".

Reference: In sections discussing optimization

Databricks frequently advises on ideal file and partition sizes. The general guidance is to aim for partition sizes between 128 MB and 1 GB. Option D's approach of calculating the partition count to achieve a size of 128 MB aligns perfectly with this official vendor recommendation for optimal performance.

Question 8 of 25

Question breakdown

Tests: Tune partition counts for parallelism
Difficulty: Medium

Key idea: Size partitions from the data volume (about 128 MB each) so there are many more tasks than cores, keeping every core busy while each task stays manageable.

How to reach the answer

The symptoms are few active tasks and fewer tasks than the 160 available cores, which points to too few partitions for the cluster.
A count tied only to hardware (160 or 10 partitions) ignores the 1 TB size and produces multi-GB partitions.
A fixed value such as 200 also ignores data size, giving roughly 5 GB per partition.
1 TB divided by 128 MB gives about 8,192 partitions, which run in many waves across 160 cores at a safe per-task size.

Each option

AIncorrect. 160 partitions of a 1 TB dataset are about 6 GB each, which risks spills and memory errors.
BIncorrect. A fixed 200 ignores data volume and still gives multi-GB partitions.
CIncorrect. 10 partitions would leave 150 cores idle and produce partitions around 100 GB each.
DCorrect. Basing the count on a target size of about 128 MB yields roughly 8,192 partitions, enough to saturate all cores with manageable tasks.

Common trap: Option A sounds like a natural match of one task per core, but it ignores how much data each task would have to process.

In practice: Engineers tune spark.sql.files.maxPartitionBytes, spark.sql.shuffle.partitions, or repartition() based on data volume, and rely on AQE to coalesce small shuffle partitions.

Remember: Derive the partition count from data size divided by a target partition size, not from core or node counts alone.

Drafted with AI assistance and checked against the answer key.

Keep practicing

Take the free CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice test

Ten exam-style questions with answers and explanations, plus the exam facts and study guides.

Start the free practice test