A data engineer is running a Spark job to process a dataset of 1 TB stored in distributed storage. The cluster has 10 nodes, each with 16 CPUs. Spark…

The symptoms described—low number of active tasks and fewer tasks than available CPUs—are classic indicators of under-partitioning. With 160 available CPU cores (10 nodes 16 CPUs), the cluster is significantly underutilized. The most effective and scalable approach to resolve this is to determine the number of partitions based on the data size. By dividing the total dataset size (1 TB) by a recommended partition size (e.g., 128 MB), you ensure that each task processes a manageable chunk of data. This creates a high number of partitions (approx. 8,192), allowing Spark to achieve maximum parallelism by keeping all CPU cores busy, thus optimizing resource allocation and job performance.

A. This method ignores the data size. For a 1 TB dataset, 160 partitions would be ~6.25 GB each, which is excessively large and likely to cause out-of-memory (OOM) errors.

B. Using a fixed, arbitrary value like 200 is not a best practice. It fails to account for the specific characteristics of the dataset size and cluster resources, leading to suboptimal performance.

C. This would result in only 10 partitions for 160 cores, leading to extreme under-utilization and unmanageably large partitions (~100 GB each), which would certainly fail.

1. Book: "Spark: The Definitive Guide" by Bill Chambers and Matei Zaharia (creator of Apache Spark).

Reference: Part V

Chapter 17: Performance Tuning. The chapter explains that the number of partitions is the key to parallelism. For input data from file-based sources

Spark by default creates one partition per block of the file in HDFS (defaulting to 128 MB). This establishes 128 MB as a sound

fundamental partition size. The strategy in option D directly applies this principle.

2. Book: "Learning Spark

2nd Edition" by Jules S. Damji

et al.

Reference: Chapter 10

"Performance Tuning

" section on "Number of Partitions." The authors state

"For a large dataset of 1 TB

you would want more partitions... A good starting point is to have partitions of size 128 MB." This directly endorses the calculation method described in option D as a best practice for large datasets.

3. Databricks Documentation

"Best practices: performance tuning".

Reference: In sections discussing optimization

Databricks frequently advises on ideal file and partition sizes. The general guidance is to aim for partition sizes between 128 MB and 1 GB. Option D's approach of calculating the partition count to achieve a size of 128 MB aligns perfectly with this official vendor recommendation for optimal performance.