A Data Analyst needs to retrieve employees with 5 or more years of tenure. Which code snippet filters and shows the list?

The code employeesdf.filter(employeesdf.tenure >= 5) is a transformation that creates a new DataFrame containing only the rows where the 'tenure' column is greater than or equal to 5. However, transformations in Spark are lazy, meaning they don't execute until an action is called. The .show() method is an action that triggers the execution of the preceding transformations and displays the resulting DataFrame in a user-friendly tabular format. This two-step process of applying a transformation (filter) followed by an action (show) correctly fulfills the request to filter the data and display the list.

B. This code only defines the transformation. It does not include an action like .show() or .collect() to trigger the computation and display the results.

C. This is syntactically incorrect. The filter function is a method of the DataFrame class and must be called on a DataFrame instance (e.g., employeesdf.filter(...)).

D. The .collect() action retrieves the entire filtered dataset to the driver node's memory as a list of Row objects. This is not ideal for "showing" data and can cause an OutOfMemoryError with large datasets.

---

1. Apache Spark 3.4.1 Documentation

pyspark.sql.DataFrame.filter: "Filters rows using the given condition. where() is an alias for filter()." This confirms that filter is the correct method for this operation.

2. Apache Spark 3.4.1 Documentation

pyspark.sql.DataFrame.show: "Prints the first n rows to the console. ... This is a lazy operation; use cache or eager evaluation if you want to force eager evaluation." This identifies .show() as the appropriate action to display the results of the transformation.

3. Databricks Documentation

"Transformations and actions": "Transformations are lazy operations

meaning that they don’t execute until you run an action. ... An action is an operation that returns a value

such as count

collect

or show." This reference clarifies why option B is incomplete

as it lacks an action to trigger the lazy filter transformation.

4. Apache Spark 3.4.1 Documentation

pyspark.sql.DataFrame.collect: "Returns all the records as a list of Row." The documentation also implicitly warns against its use on large datasets: "Collecting large datasets can result in an OutOfMemoryError." This supports why .show() is preferred over .collect() for displaying data.