H13-711 practice question 18 of 30

Each stage of a Spark task can be divided into jobs, and the score is marked as shuffle.

Full set: 201 questions
Answer options
A: True
B: False
Correct Answer:
[FALSE]
Explanation

The statement incorrectly describes the hierarchical relationship within Spark's execution model. The correct hierarchy is that a Spark Job is divided into one or more Stages. A Stage is a collection of parallel Tasks that can be executed without a data shuffle. The statement inverts this relationship by claiming a stage is divided into jobs. An action (e.g., count(), collect()) on a Resilient Distributed Dataset (RDD) triggers a single Job. The Spark DAGScheduler then examines the RDD lineage graph and divides the job into a set of stages. The boundaries between these stages are defined by wide dependencies, which require a shuffle of data across the cluster.

Why Incorrect

The option "True" is incorrect because it fundamentally misrepresents the Spark execution hierarchy. A Job is the top-level unit of work triggered by a user's action, and it is composed of smaller units called Stages, not the other way around.

References

1. Apache Spark Official Documentation, Job Scheduling Overview: The official documentation states, "The final result of a DAGScheduler is a set of stages. Stages are passed on to the TaskScheduler." It further clarifies, "The DAGScheduler computes a directed acyclic graph (DAG) of stages for each job." This explicitly defines the relationship as Job -> Stages.

Source: Apache Spark 3.5.1 Documentation, "Cluster Mode Overview > Job Scheduling".

2. Zaharia, M., et al. (2012). Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. This foundational peer-reviewed paper on Spark describes the scheduling process.

Reference: Section 3.2, "Job Scheduling," explains: "Our scheduler is similar to the Dryad scheduler. When a user runs an action... on an RDD, the scheduler examines that RDD’s lineage graph to build a DAG of stages... It then launches tasks to compute missing partitions from each stage until it has computed the target RDD." This confirms that a user action creates a job, which is then broken into a DAG of stages.

Source: Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation (NSDI'12), Page 4, Section 3.2.

3. University of California, Berkeley, CS 186/286 Courseware: Lecture materials on Spark architecture consistently illustrate this hierarchy.

Reference: Lecture slides on Spark execution typically contain a diagram showing: Action -> Job -> Stages -> Tasks. The slides explain that a job is a computation triggered by an action, and the DAG of this computation is broken into stages at shuffle boundaries.

Source: UC Berkeley EECS, CS 186/286 Introduction to Database Systems, Lecture 18: "Spark", Fall 2019. (Illustrates the Job -> Stage -> Task breakdown).

Scroll to Top