CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice question 2 of 25
A Spark DataFrame df is cached using the MEMORY_AND_DISK storage level, but the DataFrame is too large to fit entirely in memory. What is the likely behavior when Spark runs out of memory to store the DataFrame?
Full set: 135 questionsThe MEMORYANDDISK storage level instructs Spark to first attempt to store the DataFrame's partitions in memory as deserialized objects. If the DataFrame is too large to fit entirely in memory, Spark will store the partitions that fit and "spill" the remaining partitions to disk. When an action requires a partition stored on disk, Spark reads it from there. This process avoids job failure due to insufficient memory but introduces performance overhead because disk I/O is significantly slower than memory access. This behavior ensures that the computation can proceed even when the data exceeds available RAM.
A. Spark does not duplicate the entire DataFrame on disk. It only stores the partitions that do not fit in memory. The 2 replication levels would create duplicates, but that is not specified here.
B. The split is not "even." Spark fills memory to its capacity first and only then writes subsequent partitions to disk. It is a spill-over mechanism, not a balanced distribution.
D. The MEMORYANDDISK storage level operates on entire partitions, not on a row-by-row basis according to access frequency. It fills memory with partitions until it is full, then spills the rest.
1. Apache Spark 3.4.1 Documentation
RDD Programming Guide
"RDD Persistence" section: The official documentation describes the MEMORYANDDISK storage level as: "Store RDD as deserialized Java objects in the JVM. If the RDD does not fit in memory
store the partitions that don't fit on disk
and read them from there when they're needed." This directly supports the "spill to disk" behavior.
(Source: https://spark.apache.org/docs/latest/rdd-programming-guide.html#rdd-persistence)
2. Chambers
B.
& Zaharia
M. (2018). Spark: The Definitive Guide. O'Reilly Media
Inc. In Chapter 13
"Caching
" the section on "Storage Levels" explains that MEMORYANDDISK will "spill to disk if the RDD is too large to fit in memory." This confirms that disk is used as an overflow for data that cannot be held in memory.
3. Databricks Documentation
"Optimize performance with caching": This guide details caching strategies in Databricks. It reiterates the Spark documentation
stating for MEMORYANDDISK: "If the RDD does not fit in memory
store the partitions that don't fit on disk
and read them from there when they're needed." This confirms the behavior of storing as much as possible in memory and spilling the rest to disk.
(Source: https://docs.databricks.com/en/optimizations/cache.html)
Question breakdown
Key idea: MEMORY_AND_DISK keeps as many partitions in memory as fit and writes the remaining partitions to disk, reading them back when needed.
How to reach the answer
- The clue is the MEMORY_AND_DISK level combined with data that is larger than available memory.
- This level fills memory first and puts the overflow on disk; it does not keep a full duplicate copy on disk.
- Spark works at partition granularity, not individual rows, and does not split data evenly or by access frequency.
- Processing continues, but disk reads add latency, which matches option C.
Each option
Common trap: Option D sounds like a smart caching policy, but Spark persistence does not track per-row access frequency; it is a simple memory-first, overflow-to-disk scheme.
In practice: When the Storage tab in the Spark UI shows a cached DataFrame split between memory and disk, expect slower reuse. Consider a serialized storage level, more executor memory, or caching only the columns you need.
Remember: MEMORY_AND_DISK means memory first, with whole partitions overflowing to disk, so the job keeps running but more slowly.
Drafted with AI assistance and checked against the answer key.
Keep practicing
Take the free CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice test
Ten exam-style questions with answers and explanations, plus the exam facts and study guides.
More questions
Other CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice questions
- Question 1You have: DataFrame A: 128 GB of transactions DataFrame B: 1 GB user lookup table Which s…
- Question 344 of 55. A data engineer is working on a real-time analytics pipeline using Spark Struct…
- Question 7A developer wants to test Spark Connect with an existing Spark application. What are the …
- Question 8A data engineer is running a Spark job to process a dataset of 1 TB stored in distributed…
- Question 10A data engineer is building a Structured Streaming pipeline and wants the pipeline to rec…
- Question 11Given a DataFrame df that has 10 partitions, after running the code: result = df.coalesce…
