DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-0 Practice Questions

Correct code block: transactionsDf.join(itemsDf, itemsDf.itemId == transactionsDf.productId, "outer") Static notebook | Dynamic notebook: See test 1, ( Databricks import instructions) (<a href="https://flrs.github.io/spark_practice_tests_code/#1/33.html" rel="noopener noreferrer" target="_blank" style="color: rgb(29, 78, 216);">https://flrs.github.io/spark_practice_tests_code/#1/33.html , <a href="https://bit.ly/sparkpracticeexams_import_instructions)" rel="noopener noreferrer" target="_blank" style="color: rgb(29, 78, 216);">https://bit.ly/sparkpracticeexams_import_instructions)

itemsDf.printSchema()

Correct! Here is an example of what itemsDf.printSchema() shows, you can see the tree-like structure

containing both column names and types:

root

|-- itemId: integer (nullable = true)

|-- attributes: array (nullable = true)

|

|-- element: string (containsNull = true)

|-- supplier: string (nullable = true)

itemsDf.rdd.printSchema()

No, the DataFrame's underlying RDD does not have a printSchema() method.

spark.schema(itemsDf)

Incorrect, there is no spark.schema command.

print(itemsDf.columns)

print(itemsDf.dtypes)

Wrong. While the output of this code blocks contains both column names and column types, the

information is not arranges in a tree-like way.

itemsDf.print.schema()

No, DataFrame does not have a print method.

Static notebook | Dynamic notebook: See test 3,

The challenge in this Question: comes from

there being an array variable in the schem

a. In addition, you should know how to pass a schema to the DataFrameReader that is invoked by

spark.read.

The correct way to define an array of strings in a schema is through ArrayType(StringType()). A

schema can be passed to the DataFrameReader by simply appending schema(structType) to the

read() operator. Alternatively, you can also define a schema as a string. For example, for the schema

of itemsDf, the following string would make sense: itemId integer, attributes array,

supplier string.

A thing to keep in mind is that in schema definitions, you always need to instantiate the types, like

so: StringType(). Just using StringType does not work in pySpark and will fail.

Another concern with schemas is whether columns should be nullable, so allowed to have null

values. In the case at hand, this is not a concern however, since the Question: just asks for a

"valid"

schema. Both non-nullable and nullable column schemas would be valid here, since no null value

appears in the DataFrame sample.

More info: Learning Spark, 2nd Edition, Chapter 3

Static notebook | Dynamic notebook: See test 3,

transactionsDf.groupBy("productId").count()

Correct. This code block first groups DataFrame transactionsDf by column productId and then counts

the rows in each group.

transactionsDf.groupBy("productId").select(count("value"))

Incorrect. You cannot call select on a GroupedData object (the output of a groupBy) statement.

transactionsDf.count("productId")

No. DataFrame.count() does not take any arguments.

transactionsDf.count("productId").distinct()

Wrong. Since DataFrame.count() does not take any arguments, this option cannot be right.

transactionsDf.groupBy("productId").agg(col("value").count())

False. A Column object, as returned by col("value"), does not have a count() method. You can see all

available methods for Column object linked in the Spark documentation below.

More info: pyspark.sql.DataFrame.count — PySpark 3.1.2 documentation, pyspark.sql.Column —

PySpark 3.1.2 documentation

Static notebook | Dynamic notebook: See test 3,

transactionsDf.where(transactionsDf.storeId!=25)

Correct. DataFrame.where() is an alias for the DataFrame.filter() method. Using this method, it is

straightforward to filter out rows that do not have value 25 in column storeId.

transactionsDf.select(transactionsDf.storeId!=25)

Wrong. The select operator allows you to build DataFrames column-wise, but when using it as

shown, it does not filter out rows.

transactionsDf.filter(transactionsDf.storeId==25)

Incorrect. Although the filter expression works for filtering rows, the == in the filtering condition is

inappropriate. It should be != instead.

transactionsDf.drop(transactionsDf.storeId==25)

No. DataFrame.drop() is used to remove specific columns, but not rows, from the DataFrame.

transactionsDf.remove(transactionsDf.storeId==25)

False. There is no DataFrame.remove() operator in PySpark.

More info: pyspark.sql.DataFrame.where — PySpark 3.1.2 documentation

Static notebook | Dynamic notebook: See test 3,

Correct code block:

spark.conf.set("spark.sql.shuffle.partitions", 20)

The code block expresses the option incorrectly.

Correct! The option should be expressed as a string.

The code block sets the wrong option.

No, spark.sql.shuffle.partitions is the correct option for the use case in the question.

The code block sets the incorrect number of parts.

Wrong, the code block correctly states 20 parts.

The code block uses the wrong command for setting an option.

No, in PySpark spark.conf.set() is the correct command for setting an option.

The code block is missing a parameter.

Incorrect, spark.conf.set() takes two parameters.

More info: Configuration - Spark 3.1.2 Documentation

Correct code block:

transactionsDf.drop("predError", "productId", "value")

Static notebook | Dynamic notebook: See test 1,

Tasks run in parallel via slots.

Correct. Given the assumption, an executor then has one or more "slots", defined by the equation

spark.executor.cores / spark.task.cpus. With the executor's resources divided into slots, each task

takes up a slot and multiple tasks can be executed in parallel.

Slot is another name for executor.

No, a slot is part of an executor.

An executor runs on a single core.

No, an executor can occupy multiple cores. This is set by the spark.executor.cores option.

There must be more slots than tasks.

No. Slots just process tasks. One could imagine a scenario where there was just a single slot for

multiple tasks, processing one task at a time. Granted – this is the opposite of what Spark should be

used for, which is distributed data processing over multiple cores and machines, performing many

tasks in parallel.

There must be less executors than tasks.

No, there is no such requirement.

More info: Spark Architecture | Distributed Systems Architecture (https://bit.ly/3x4MZZt)

Output of correct code block:

+----------------------------+

|size(split(supplier, , -1))|

+----------------------------+

|

3|

|

1|

|

3|

+----------------------------+

This Question: shows a

typical use case for the split command: Splitting a string into words. An additional difficulty is that

you

are

asked

to

count

the

words.

Although

it

is

tempting

to

use

the

count method here, the size method (as in: size of an array) is actually the correct one to use.

Familiarize yourself with the split and the size methods using the linked documentation below.

More info:

Split method: pyspark.sql.functions.split — PySpark 3.1.2 documentation

Size method: pyspark.sql.functions.size — PySpark 3.1.2 documentation

Static notebook | Dynamic notebook: See test 2,

transactionsDf.unpersist()

Correct. The DataFrame.unpersist() command does exactly what the Question: asks for

- it removes all cached parts of the DataFrame from memory and disk.

del transactionsDf

False. While this option can help remove the DataFrame from memory and disk, it does not do so

immediately. The reason is that this command just notifies the Python garbage collector that the

transactionsDf now may be deleted from memory. However, the garbage collector does not do so

immediately and, if you wanted it to run immediately, would need to be specifically triggered to do

so. Find more information linked below.

array_remove(transactionsDf, "*")

Incorrect. The array_remove method from pyspark.sql.functions is used for removing elements from

arrays in columns that match a specific condition. Also, the first argument would be a column, and

not a DataFrame as shown in the code block.

transactionsDf.persist()

No. This code block does exactly the opposite of what is asked for: It caches (writes) DataFrame

transactionsDf to memory and disk. Note that even though you do not pass in a specific storage

level here, Spark will use the default storage level (MEMORY_AND_DISK).

transactionsDf.clearCache()

Wrong. Spark's DataFrame does not have a clearCache() method.

More info: pyspark.sql.DataFrame.unpersist — PySpark 3.1.2 documentation, python - How to delete

an RDD in PySpark for the purpose of releasing resources? - Stack Overflow

Static notebook | Dynamic notebook: See test 3,