What is the difference between df.cache() and df.persist() in Spark DataFrame?

The cache() method is a specific, parameter-less convenience function that persists a DataFrame using the default storage level, which is MEMORYANDDISK. In contrast, the persist() method is more versatile. While calling persist() with no arguments is equivalent to cache(), its primary purpose is to allow the developer to specify a different storage level. This provides granular control over how the DataFrame is stored, enabling optimizations based on memory availability and performance requirements by choosing levels like MEMORYONLY, DISKONLY, or serialized versions.

A: The cache() method does not accept any arguments and cannot be used to set a storage level; it always uses the default.

B: The default storage level for persist() is MEMORYANDDISK, not DISKONLY. DISKONLY would typically offer lower performance.

C: This statement incorrectly reverses the roles of the two functions. persist() is the function that allows for setting different storage levels.

1. Apache Spark 3.4.1 Documentation

pyspark.sql.DataFrame.cache: "Persists this :class:DataFrame with the default storage level (MEMORYANDDISK)."

Source: pyspark.sql.DataFrame.cache

2. Apache Spark 3.4.1 Documentation

pyspark.sql.DataFrame.persist: "Sets the storage level to persist the contents of the :class:DataFrame... This can be used to assign a new storage level..." The method signature is persist(storageLevel=StorageLevel(True

True

False

False

1)).

Source: pyspark.sql.DataFrame.persist

3. Databricks Documentation

Caching: "You can choose to cache data using df.cache() or df.persist(). .cache() will cache to MEMORYANDDISK storage level. .persist() can cache to various storage levels

which you can see in the Spark API docs."

Source: Databricks Documentation > Workspace guide > Caching.

4. Learning Spark

2nd Edition (O'Reilly)

Chapter 4: Caching and Persistence: "The cache() method is a shorthand for persist() with the default storage level

MEMORYANDDISK." (Note: While a commercial book

its content is directly aligned with and explains the official Spark documentation for the exam's scope).