CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice question 2 of 25

A Spark DataFrame df is cached using the MEMORY_AND_DISK storage level, but the DataFrame is too large to fit entirely in memory. What is the likely…

Choose your answer, then check it against the explanation and the full breakdown below.

A Spark DataFrame df is cached using the MEMORY_AND_DISK storage level, but the DataFrame is too large to fit entirely in memory. What is the likely behavior when Spark runs out of memory to store the DataFrame?

Answer options

Correct Answer:

Explanation

The MEMORYANDDISK storage level instructs Spark to first attempt to store the DataFrame's partitions in memory as deserialized objects. If the DataFrame is too large to fit entirely in memory, Spark will store the partitions that fit and "spill" the remaining partitions to disk. When an action requires a partition stored on disk, Spark reads it from there. This process avoids job failure due to insufficient memory but introduces performance overhead because disk I/O is significantly slower than memory access. This behavior ensures that the computation can proceed even when the data exceeds available RAM.

Why Incorrect

A. Spark does not duplicate the entire DataFrame on disk. It only stores the partitions that do not fit in memory. The 2 replication levels would create duplicates, but that is not specified here.

B. The split is not "even." Spark fills memory to its capacity first and only then writes subsequent partitions to disk. It is a spill-over mechanism, not a balanced distribution.

D. The MEMORYANDDISK storage level operates on entire partitions, not on a row-by-row basis according to access frequency. It fills memory with partitions until it is full, then spills the rest.

References

1. Apache Spark 3.4.1 Documentation

RDD Programming Guide

"RDD Persistence" section: The official documentation describes the MEMORYANDDISK storage level as: "Store RDD as deserialized Java objects in the JVM. If the RDD does not fit in memory

store the partitions that don't fit on disk

and read them from there when they're needed." This directly supports the "spill to disk" behavior.

(Source: https://spark.apache.org/docs/latest/rdd-programming-guide.html#rdd-persistence)

2. Chambers

& Zaharia

M. (2018). Spark: The Definitive Guide. O'Reilly Media

Inc. In Chapter 13

"Caching

" the section on "Storage Levels" explains that MEMORYANDDISK will "spill to disk if the RDD is too large to fit in memory." This confirms that disk is used as an overflow for data that cannot be held in memory.

3. Databricks Documentation

"Optimize performance with caching": This guide details caching strategies in Databricks. It reiterates the Spark documentation

stating for MEMORYANDDISK: "If the RDD does not fit in memory

store the partitions that don't fit on disk

and read them from there when they're needed." This confirms the behavior of storing as much as possible in memory and spilling the rest to disk.

(Source: https://docs.databricks.com/en/optimizations/cache.html)

Question 2 of 25

Question breakdown

Tests: Understand storage level behavior for persisted DataFrames
Difficulty: Easy

Key idea: MEMORY_AND_DISK keeps as many partitions in memory as fit and writes the remaining partitions to disk, reading them back when needed.

How to reach the answer

The clue is the MEMORY_AND_DISK level combined with data that is larger than available memory.
This level fills memory first and puts the overflow on disk; it does not keep a full duplicate copy on disk.
Spark works at partition granularity, not individual rows, and does not split data evenly or by access frequency.
Processing continues, but disk reads add latency, which matches option C.

Each option

AIncorrect. Spark does not mirror the data in both places or move the whole DataFrame to disk; only the partitions that do not fit go to disk.
BIncorrect. There is no even split; memory is filled first and only the overflow goes to disk.
CCorrect. Partitions that fit stay in memory, the rest are spilled to disk, and the job continues with extra I/O cost.
DIncorrect. Placement is not driven by how often rows are accessed; storage happens at partition level as memory allows.

Common trap: Option D sounds like a smart caching policy, but Spark persistence does not track per-row access frequency; it is a simple memory-first, overflow-to-disk scheme.

In practice: When the Storage tab in the Spark UI shows a cached DataFrame split between memory and disk, expect slower reuse. Consider a serialized storage level, more executor memory, or caching only the columns you need.

Remember: MEMORY_AND_DISK means memory first, with whole partitions overflowing to disk, so the job keeps running but more slowly.

Drafted with AI assistance and checked against the answer key.

Keep practicing

Take the free CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice test

Ten exam-style questions with answers and explanations, plus the exam facts and study guides.

Start the free practice test