A data engineer is reviewing a Spark application that applies several transformations to a DataFrame but notices that the job does not start…

Apache Spark's execution model is fundamentally based on the principle of lazy evaluation. Transformations, such as select(), filter(), or groupBy(), are not executed immediately when they are defined. Instead, Spark builds a Directed Acyclic Graph (DAG) representing the logical plan of computations. This plan is only compiled, optimized, and executed when an action, such as count(), collect(), or write(), is called on the DataFrame. This deferred execution allows Spark's Catalyst Optimizer to create a more efficient physical execution plan by analyzing the entire transformation pipeline at once.

A. The Spark engine is automated and triggers jobs based on code execution (specifically, actions), not manual intervention.

C. Transformations are not executed to build the lineage graph; they define the lineage graph, which is then executed later.

D. Optimization occurs after an action is called and before execution begins, not during the definition of transformations. The delay is due to lazy evaluation, not the optimization process itself.

1. Databricks Documentation

"Introduction to the Spark programming model": This document explicitly states

"Apache Spark uses lazy evaluation for DataFrame and Dataset operations. It waits until an action is called before executing the graph of computations... Because of lazy evaluation

if you define a DataFrame transformation

Spark does not execute it immediately." This directly supports options B and E.

Source: Databricks Documentation > Get started > Apache Spark > Introduction to the Spark programming model.

2. Apache Spark™ Documentation

"RDD Programming Guide": In the "Lazy Evaluation" section

the guide explains

"All transformations in Spark are lazy

in that they do not compute their results right away. Instead

they just remember the transformations applied to some base dataset... The transformations are only computed when an action requires a result to be returned to the driver program." This confirms that transformations are lazy (E) and actions trigger computation (B).

Source: spark.apache.org > Documentation > Latest > RDD Programming Guide > Lazy Evaluation.

3. Zaharia

M.

Chowdhury

M.

Franklin

M. J.

Shenker

S.

& Stoica

I. (2012). Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. NSDI'12. In Section 3.1

"Resilient Distributed Datasets (RDDs)

" the foundational paper for Spark states: "By default

RDDs are lazily evaluated—that is

they are not materialized until an action (e.g.

count) is invoked on them." This academic source establishes lazy evaluation (E) and the role of actions (B) as core principles.

DOI: https://www.usenix.org/system/files/conference/nsdi12/nsdi12-final138.pdf (Page 3

Section 3.1)