CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice question 12 of 25

What is the difference between df.cache() and df.persist() in Spark DataFrame?

Full set: 135 questions
Answer options
A: Both cache() and persist() can be used to set the default storage level (MEMORY_AND_DISK_SER)
B: Both functions perform the same operation. The persist() function provides improved performance as its default storage level is DISK_ONLY.
C: persist() - Persists the DataFrame with the default storage level (MEMORY_AND_DISK_SER) and cache() - Can be used to set different storage levels to persist the contents of the DataFrame.
D: cache() - Persists the DataFrame with the default storage level (MEMORY_AND_DISK) and persist() - Can be used to set different storage levels to persist the contents of the DataFrame Correct answer
Correct Answer:
D. cache() - Persists the DataFrame with the default storage level (MEMORY_AND_DISK) and persist() - Can be used to set different storage levels to persist the contents of the DataFrame
Explanation

The cache() method is a specific, parameter-less convenience function that persists a DataFrame using the default storage level, which is MEMORYANDDISK. In contrast, the persist() method is more versatile. While calling persist() with no arguments is equivalent to cache(), its primary purpose is to allow the developer to specify a different storage level. This provides granular control over how the DataFrame is stored, enabling optimizations based on memory availability and performance requirements by choosing levels like MEMORYONLY, DISKONLY, or serialized versions.

Why Incorrect

A: The cache() method does not accept any arguments and cannot be used to set a storage level; it always uses the default. B: The default storage level for persist() is MEMORYANDDISK, not DISKONLY. DISKONLY would typically offer lower performance. C: This statement incorrectly reverses the roles of the two functions. persist() is the function that allows for setting different storage levels.

References

1. Apache Spark 3.4.1 Documentation

pyspark.sql.DataFrame.cache: "Persists this :class:DataFrame with the default storage level (MEMORYANDDISK)."

Source: pyspark.sql.DataFrame.cache

2. Apache Spark 3.4.1 Documentation

pyspark.sql.DataFrame.persist: "Sets the storage level to persist the contents of the :class:DataFrame... This can be used to assign a new storage level..." The method signature is persist(storageLevel=StorageLevel(True

True

False

False

1)).

Source: pyspark.sql.DataFrame.persist

3. Databricks Documentation

Caching: "You can choose to cache data using df.cache() or df.persist(). .cache() will cache to MEMORYANDDISK storage level. .persist() can cache to various storage levels

which you can see in the Spark API docs."

Source: Databricks Documentation > Workspace guide > Caching.

4. Learning Spark

2nd Edition (O'Reilly)

Chapter 4: Caching and Persistence: "The cache() method is a shorthand for persist() with the default storage level

MEMORYANDDISK." (Note: While a commercial book

its content is directly aligned with and explains the official Spark documentation for the exam's scope).

Scroll to Top