CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice question 25 of 25

A Data Analyst needs to retrieve employees with 5 or more years of tenure. Which code snippet filters and shows the list?

Full set: 135 questions
Answer options
A: employees_df.filter(employees_df.tenure >= 5).show() Correct answer
B: employees_df.where(employees_df.tenure >= 5)
C: filter(employees_df.tenure >= 5)
D: employees_df.filter(employees_df.tenure >= 5).collect()
Correct Answer:
A. employees_df.filter(employees_df.tenure >= 5).show()
Explanation

The code employeesdf.filter(employeesdf.tenure >= 5) is a transformation that creates a new DataFrame containing only the rows where the 'tenure' column is greater than or equal to 5. However, transformations in Spark are lazy, meaning they don't execute until an action is called. The .show() method is an action that triggers the execution of the preceding transformations and displays the resulting DataFrame in a user-friendly tabular format. This two-step process of applying a transformation (filter) followed by an action (show) correctly fulfills the request to filter the data and display the list.

Why Incorrect

B. This code only defines the transformation. It does not include an action like .show() or .collect() to trigger the computation and display the results.
C. This is syntactically incorrect. The filter function is a method of the DataFrame class and must be called on a DataFrame instance (e.g., employeesdf.filter(...)).
D. The .collect() action retrieves the entire filtered dataset to the driver node's memory as a list of Row objects. This is not ideal for "showing" data and can cause an OutOfMemoryError with large datasets. ---

References

1. Apache Spark 3.4.1 Documentation, pyspark.sql.DataFrame.filter: "Filters rows using the given condition. where() is an alias for filter()." This confirms that filter is the correct method for this operation.

2. Apache Spark 3.4.1 Documentation, pyspark.sql.DataFrame.show: "Prints the first n rows to the console. ... This is a lazy operation; use cache or eager evaluation if you want to force eager evaluation." This identifies .show() as the appropriate action to display the results of the transformation.

3. Databricks Documentation, "Transformations and actions": "Transformations are lazy operations, meaning that they don’t execute until you run an action. ... An action is an operation that returns a value, such as count, collect, or show." This reference clarifies why option B is incomplete, as it lacks an action to trigger the lazy filter transformation.

4. Apache Spark 3.4.1 Documentation, pyspark.sql.DataFrame.collect: "Returns all the records as a list of Row." The documentation also implicitly warns against its use on large datasets: "Collecting large datasets can result in an OutOfMemoryError." This supports why .show() is preferred over .collect() for displaying data.

Scroll to Top