CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice question 12 of 25
What is the difference between df.cache() and df.persist() in Spark DataFrame?
Full set: 135 questionsThe cache() method is a specific, parameter-less convenience function that persists a DataFrame using the default storage level, which is MEMORYANDDISK. In contrast, the persist() method is more versatile. While calling persist() with no arguments is equivalent to cache(), its primary purpose is to allow the developer to specify a different storage level. This provides granular control over how the DataFrame is stored, enabling optimizations based on memory availability and performance requirements by choosing levels like MEMORYONLY, DISKONLY, or serialized versions.
A: The cache() method does not accept any arguments and cannot be used to set a storage level; it always uses the default. B: The default storage level for persist() is MEMORYANDDISK, not DISKONLY. DISKONLY would typically offer lower performance. C: This statement incorrectly reverses the roles of the two functions. persist() is the function that allows for setting different storage levels.
1. Apache Spark 3.4.1 Documentation
pyspark.sql.DataFrame.cache: "Persists this :class:DataFrame with the default storage level (MEMORYANDDISK)."
Source: pyspark.sql.DataFrame.cache
2. Apache Spark 3.4.1 Documentation
pyspark.sql.DataFrame.persist: "Sets the storage level to persist the contents of the :class:DataFrame... This can be used to assign a new storage level..." The method signature is persist(storageLevel=StorageLevel(True
True
False
False
1)).
Source: pyspark.sql.DataFrame.persist
3. Databricks Documentation
Caching: "You can choose to cache data using df.cache() or df.persist(). .cache() will cache to MEMORYANDDISK storage level. .persist() can cache to various storage levels
which you can see in the Spark API docs."
Source: Databricks Documentation > Workspace guide > Caching.
4. Learning Spark
2nd Edition (O'Reilly)
Chapter 4: Caching and Persistence: "The cache() method is a shorthand for persist() with the default storage level
MEMORYANDDISK." (Note: While a commercial book
its content is directly aligned with and explains the official Spark documentation for the exam's scope).
Keep practicing
Take the free CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice test
Ten exam-style questions with answers and explanations, plus the exam facts and study guides.
More questions
Other CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice questions
- Question 1You have: DataFrame A: 128 GB of transactions DataFrame B: 1 GB user lookup table Which s…
- Question 2A Spark DataFrame df is cached using the MEMORY_AND_DISK storage level, but the DataFrame…
- Question 344 of 55. A data engineer is working on a real-time analytics pipeline using Spark Struct…
- Question 7A developer wants to test Spark Connect with an existing Spark application. What are the …
- Question 8A data engineer is running a Spark job to process a dataset of 1 TB stored in distributed…
- Question 10A data engineer is building a Structured Streaming pipeline and wants the pipeline to rec…
