CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice question 20 of 25
A data engineer is reviewing a Spark application that applies several transformations to a DataFrame but notices that the job does not start executing immediately. Which two characteristics of Apache Spark's execution model explain this behavior? Choose 2 answers:
Full set: 135 questionsApache Spark's execution model is fundamentally based on the principle of lazy evaluation. Transformations, such as select(), filter(), or groupBy(), are not executed immediately when they are defined. Instead, Spark builds a Directed Acyclic Graph (DAG) representing the logical plan of computations. This plan is only compiled, optimized, and executed when an action, such as count(), collect(), or write(), is called on the DataFrame. This deferred execution allows Spark's Catalyst Optimizer to create a more efficient physical execution plan by analyzing the entire transformation pipeline at once.
A. The Spark engine is automated and triggers jobs based on code execution (specifically, actions), not manual intervention.
C. Transformations are not executed to build the lineage graph; they define the lineage graph, which is then executed later.
D. Optimization occurs after an action is called and before execution begins, not during the definition of transformations. The delay is due to lazy evaluation, not the optimization process itself.
1. Databricks Documentation
"Introduction to the Spark programming model": This document explicitly states
"Apache Spark uses lazy evaluation for DataFrame and Dataset operations. It waits until an action is called before executing the graph of computations... Because of lazy evaluation
if you define a DataFrame transformation
Spark does not execute it immediately." This directly supports options B and E.
Source: Databricks Documentation > Get started > Apache Spark > Introduction to the Spark programming model.
2. Apache Spark™ Documentation
"RDD Programming Guide": In the "Lazy Evaluation" section
the guide explains
"All transformations in Spark are lazy
in that they do not compute their results right away. Instead
they just remember the transformations applied to some base dataset... The transformations are only computed when an action requires a result to be returned to the driver program." This confirms that transformations are lazy (E) and actions trigger computation (B).
Source: spark.apache.org > Documentation > Latest > RDD Programming Guide > Lazy Evaluation.
3. Zaharia
M.
Chowdhury
M.
Franklin
M. J.
Shenker
S.
& Stoica
I. (2012). Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. NSDI'12. In Section 3.1
"Resilient Distributed Datasets (RDDs)
" the foundational paper for Spark states: "By default
RDDs are lazily evaluated—that is
they are not materialized until an action (e.g.
count) is invoked on them." This academic source establishes lazy evaluation (E) and the role of actions (B) as core principles.
DOI: https://www.usenix.org/system/files/conference/nsdi12/nsdi12-final138.pdf (Page 3
Section 3.1)
Keep practicing
Take the free CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice test
Ten exam-style questions with answers and explanations, plus the exam facts and study guides.
More questions
Other CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice questions
- Question 1You have: DataFrame A: 128 GB of transactions DataFrame B: 1 GB user lookup table Which s…
- Question 2A Spark DataFrame df is cached using the MEMORY_AND_DISK storage level, but the DataFrame…
- Question 344 of 55. A data engineer is working on a real-time analytics pipeline using Spark Struct…
- Question 7A developer wants to test Spark Connect with an existing Spark application. What are the …
- Question 8A data engineer is running a Spark job to process a dataset of 1 TB stored in distributed…
- Question 10A data engineer is building a Structured Streaming pipeline and wants the pipeline to rec…
