Free DATABRICKS-CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-0 Practice Test

Correct code block: transactionsDf.join(itemsDf, itemsDf.itemId == transactionsDf.productId, "outer") Static notebook | Dynamic notebook: See test 1, ( Databricks import instructions) (<a href="https://flrs.github.io/spark_practice_tests_code/#1/33.html" rel="noopener noreferrer" target="_blank" style="color: rgb(29, 78, 216);">https://flrs.github.io/spark_practice_tests_code/#1/33.html , <a href="https://bit.ly/sparkpracticeexams_import_instructions)" rel="noopener noreferrer" target="_blank" style="color: rgb(29, 78, 216);">https://bit.ly/sparkpracticeexams_import_instructions)

itemsDf.printSchema() Correct! Here is an example of what itemsDf.printSchema() shows, you can see the tree-like structure containing both column names and types: root |-- itemId: integer (nullable = true) |-- attributes: array (nullable = true) | |-- element: string (containsNull = true) |-- supplier: string (nullable = true) itemsDf.rdd.printSchema() No, the DataFrame's underlying RDD does not have a printSchema() method. spark.schema(itemsDf) Incorrect, there is no spark.schema command. print(itemsDf.columns) print(itemsDf.dtypes) Wrong. While the output of this code blocks contains both column names and column types, the information is not arranges in a tree-like way. itemsDf.print.schema() No, DataFrame does not have a print method. Static notebook | Dynamic notebook: See test 3,

The challenge in this Question: comes from there being an array variable in the schem
a. In addition, you should know how to pass a schema to the DataFrameReader that is invoked by spark.read. The correct way to define an array of strings in a schema is through ArrayType(StringType()). A schema can be passed to the DataFrameReader by simply appending schema(structType) to the read() operator. Alternatively, you can also define a schema as a string. For example, for the schema of itemsDf, the following string would make sense: itemId integer, attributes array, supplier string. A thing to keep in mind is that in schema definitions, you always need to instantiate the types, like so: StringType(). Just using StringType does not work in pySpark and will fail. Another concern with schemas is whether columns should be nullable, so allowed to have null values. In the case at hand, this is not a concern however, since the Question: just asks for a "valid" schema. Both non-nullable and nullable column schemas would be valid here, since no null value appears in the DataFrame sample. More info: Learning Spark, 2nd Edition, Chapter 3 Static notebook | Dynamic notebook: See test 3,

transactionsDf.groupBy("productId").count() Correct. This code block first groups DataFrame transactionsDf by column productId and then counts the rows in each group. transactionsDf.groupBy("productId").select(count("value")) Incorrect. You cannot call select on a GroupedData object (the output of a groupBy) statement. transactionsDf.count("productId") No. DataFrame.count() does not take any arguments. transactionsDf.count("productId").distinct() Wrong. Since DataFrame.count() does not take any arguments, this option cannot be right. transactionsDf.groupBy("productId").agg(col("value").count()) False. A Column object, as returned by col("value"), does not have a count() method. You can see all available methods for Column object linked in the Spark documentation below. More info: pyspark.sql.DataFrame.count — PySpark 3.1.2 documentation, pyspark.sql.Column — PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 3,

transactionsDf.where(transactionsDf.storeId!=25) Correct. DataFrame.where() is an alias for the DataFrame.filter() method. Using this method, it is straightforward to filter out rows that do not have value 25 in column storeId. transactionsDf.select(transactionsDf.storeId!=25) Wrong. The select operator allows you to build DataFrames column-wise, but when using it as shown, it does not filter out rows. transactionsDf.filter(transactionsDf.storeId==25) Incorrect. Although the filter expression works for filtering rows, the == in the filtering condition is inappropriate. It should be != instead. transactionsDf.drop(transactionsDf.storeId==25) No. DataFrame.drop() is used to remove specific columns, but not rows, from the DataFrame. transactionsDf.remove(transactionsDf.storeId==25) False. There is no DataFrame.remove() operator in PySpark. More info: pyspark.sql.DataFrame.where — PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 3,

Correct code block: spark.conf.set("spark.sql.shuffle.partitions", 20) The code block expresses the option incorrectly. Correct! The option should be expressed as a string. The code block sets the wrong option. No, spark.sql.shuffle.partitions is the correct option for the use case in the question. The code block sets the incorrect number of parts. Wrong, the code block correctly states 20 parts. The code block uses the wrong command for setting an option. No, in PySpark spark.conf.set() is the correct command for setting an option. The code block is missing a parameter. Incorrect, spark.conf.set() takes two parameters. More info: Configuration - Spark 3.1.2 Documentation

Correct code block: transactionsDf.drop("predError", "productId", "value") Static notebook | Dynamic notebook: See test 1,

Tasks run in parallel via slots. Correct. Given the assumption, an executor then has one or more "slots", defined by the equation spark.executor.cores / spark.task.cpus. With the executor's resources divided into slots, each task takes up a slot and multiple tasks can be executed in parallel. Slot is another name for executor. No, a slot is part of an executor. An executor runs on a single core. No, an executor can occupy multiple cores. This is set by the spark.executor.cores option. There must be more slots than tasks. No. Slots just process tasks. One could imagine a scenario where there was just a single slot for multiple tasks, processing one task at a time. Granted – this is the opposite of what Spark should be used for, which is distributed data processing over multiple cores and machines, performing many tasks in parallel. There must be less executors than tasks. No, there is no such requirement. More info: Spark Architecture | Distributed Systems Architecture (https://bit.ly/3x4MZZt)

Output of correct code block: +----------------------------+ |size(split(supplier, , -1))| +----------------------------+ | 3| | 1| | 3| +----------------------------+ This Question: shows a typical use case for the split command: Splitting a string into words. An additional difficulty is that you are asked to count the words. Although it is tempting to use the count method here, the size method (as in: size of an array) is actually the correct one to use. Familiarize yourself with the split and the size methods using the linked documentation below. More info: Split method: pyspark.sql.functions.split — PySpark 3.1.2 documentation Size method: pyspark.sql.functions.size — PySpark 3.1.2 documentation Static notebook | Dynamic notebook: See test 2,

transactionsDf.unpersist() Correct. The DataFrame.unpersist() command does exactly what the Question: asks for
- it removes all cached parts of the DataFrame from memory and disk. del transactionsDf False. While this option can help remove the DataFrame from memory and disk, it does not do so immediately. The reason is that this command just notifies the Python garbage collector that the transactionsDf now may be deleted from memory. However, the garbage collector does not do so immediately and, if you wanted it to run immediately, would need to be specifically triggered to do so. Find more information linked below. array_remove(transactionsDf, "*") Incorrect. The array_remove method from pyspark.sql.functions is used for removing elements from arrays in columns that match a specific condition. Also, the first argument would be a column, and not a DataFrame as shown in the code block. transactionsDf.persist() No. This code block does exactly the opposite of what is asked for: It caches (writes) DataFrame transactionsDf to memory and disk. Note that even though you do not pass in a specific storage level here, Spark will use the default storage level (MEMORY_AND_DISK). transactionsDf.clearCache() Wrong. Spark's DataFrame does not have a clearCache() method. More info: pyspark.sql.DataFrame.unpersist — PySpark 3.1.2 documentation, python - How to delete an RDD in PySpark for the purpose of releasing resources? - Stack Overflow Static notebook | Dynamic notebook: See test 3,