CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice question 21 of 25
A data engineer is running a batch processing job on a Spark cluster with the following configuration: 10 worker nodes 16 CPU cores per worker node 64 GB RAM per node The data engineer wants to allocate four executors per node, each executor using four cores. What is the total number of CPU cores used by the application?
Full set: 135 questionsThe total number of CPU cores utilized by the Spark application is determined by multiplying the number of worker nodes by the number of executors allocated per node, and then by the number of CPU cores assigned to each executor. The calculation is as follows: Total Executors = (Number of worker nodes) × (Executors per node) = 10 × 4 = 40 executors. Total CPU Cores = (Total Executors) × (Cores per executor) = 40 × 4 = 160 cores. This configuration fully utilizes the 16 available cores on each of the 10 worker nodes.
B. 64: This value might be derived from an incorrect calculation, such as multiplying the cores per node (16) by the executors per node (4), which is not the correct formula.
C. 80: This result cannot be logically derived from the given parameters. It may represent a miscalculation, such as using only half the number of nodes (5 nodes 4 executors/node 4 cores/executor).
D. 40: This represents the total number of executors (10 nodes × 4 executors per node), not the total number of CPU cores used by the application.
1. Official Apache Spark Documentation
"Application Properties": The documentation specifies the configuration properties for a Spark application. The total core count is the product of the number of executors and the cores per executor (spark.executor.cores). The number of executors is determined by the cluster setup. In this scenario
it's 10 nodes 4 executors/node.
Reference: Apache Spark 3.x Documentation
Configuration > Application Properties
see descriptions for spark.executor.cores.
2. Databricks Documentation
"Cluster configuration": The Databricks documentation explains that a cluster consists of a driver and worker nodes. Executors are processes that run on worker nodes
and each executor is allocated a specific number of cores. The total core usage for the application is the sum of cores allocated to all executors.
Reference: Databricks Documentation
Compute > Clusters > Cluster configuration.
3. Karau
H.
Konwinski
A.
Wendell
P.
& Zaharia
M. (2015). Learning Spark: Lightning-Fast Big Data Analysis. O'Reilly Media
Inc.
Chapter 4
"Spark's Execution Model
" explains that the number of cores available to an application is the number of executors multiplied by the cores per executor. This fundamental principle directly supports the calculation. (Note: While a commercial book
its authors are the creators of Spark
making it an authoritative source on its architecture).
Keep practicing
Take the free CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice test
Ten exam-style questions with answers and explanations, plus the exam facts and study guides.
More questions
Other CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice questions
- Question 1You have: DataFrame A: 128 GB of transactions DataFrame B: 1 GB user lookup table Which s…
- Question 2A Spark DataFrame df is cached using the MEMORY_AND_DISK storage level, but the DataFrame…
- Question 344 of 55. A data engineer is working on a real-time analytics pipeline using Spark Struct…
- Question 7A developer wants to test Spark Connect with an existing Spark application. What are the …
- Question 8A data engineer is running a Spark job to process a dataset of 1 TB stored in distributed…
- Question 10A data engineer is building a Structured Streaming pipeline and wants the pipeline to rec…
