The code block shown below should set the number of partitions that Spark uses when shuffling data for joins or aggregations to 100. Choose the…

Correct code block:

spark.conf.set("spark.sql.shuffle.partitions", 100)

The conf interface is part of the SparkSession, so you need to call it through spark and not pyspark.

To configure spark, you need to use the set method, not the get method. get reads a property,

but does not write it. The correct property to achieve what is outlined in the Question: is spark

.sql.aggregate.partitions,

which

needs

to

be

passed

to

set

as

a

string.

Properties

spark.shuffle.partitions and spark.sql.aggregate.partitions do not exist in Spark.

Static notebook | Dynamic notebook: See test 2,