You have: DataFrame A: 128 GB of transactions DataFrame B: 1 GB user lookup table Which strategy is correct for broadcasting?

The most effective join optimization strategy in this scenario is the broadcast hash join. This strategy involves sending the smaller DataFrame (DataFrame B, 1 GB) to every executor node in the cluster. By doing this, each executor has a complete local copy of the user lookup table. The join can then be performed on each partition of the large DataFrame (DataFrame A, 128 GB) locally, without requiring the costly network operation of shuffling the massive 128 GB DataFrame. This significantly improves performance by minimizing data movement across the network.

A. The primary performance gain from a broadcast join comes from avoiding the shuffle of the large DataFrame, not the small one being broadcasted.

C. Attempting to broadcast the large DataFrame (128 GB) is incorrect. This would likely cause an OutOfMemoryError on the driver node as it tries to collect the entire large DataFrame.

D. This statement is factually incorrect; DataFrame A is the larger of the two DataFrames and should not be broadcasted.

---

1. Apache Spark Official Documentation

SQL Tuning Guide: In the section on Join Optimization

the documentation states: "Spark uses broadcast join instead of hash join when the size of one side of the join data is smaller than the spark.sql.autoBroadcastJoinThreshold... Spark broadcasts the smaller table to all executors... This avoids a shuffle of the data in the larger table." This directly supports broadcasting the smaller table (B) to avoid shuffling the larger one (A).

Source: Apache Spark 3.5.0 Documentation

Performance Tuning > SQL Tuning Guide > Join Optimization.

2. Databricks Official Documentation

Join optimization: The Databricks documentation on optimizations explains: "When you join a large table to a small table

a broadcast join can give you better performance than a merge join. A broadcast join broadcasts the small table to all nodes in the cluster and performs a hash join locally on each node

which reduces network I/O." This confirms that the smaller table is the candidate for broadcasting.

Source: Databricks Documentation > Workloads > Data engineering > Delta Lake > Optimizations > Join optimization.

3. Karau

H.

Konwinski

A.

Wendell

P.

& Zaharia

M. (2015). Learning Spark: Lightning-Fast Big Data Analysis. O'Reilly Media

Inc. In Chapter 5

"Working with Key/Value Data

" the authors explain the broadcast join mechanism: "If one of the RDDs is much smaller than the other... we can do a broadcast hash join... This avoids the cost of the shuffle." This principle is fundamental to Spark's architecture and directly applies to DataFrames.

Source: Chapter 5

section on "Joins".