CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice question 25 of 25
A Data Analyst needs to retrieve employees with 5 or more years of tenure. Which code snippet filters and shows the list?
Full set: 135 questionsThe code employeesdf.filter(employeesdf.tenure >= 5) is a transformation that creates a new DataFrame containing only the rows where the 'tenure' column is greater than or equal to 5. However, transformations in Spark are lazy, meaning they don't execute until an action is called. The .show() method is an action that triggers the execution of the preceding transformations and displays the resulting DataFrame in a user-friendly tabular format. This two-step process of applying a transformation (filter) followed by an action (show) correctly fulfills the request to filter the data and display the list.
B. This code only defines the transformation. It does not include an action like .show() or .collect() to trigger the computation and display the results.
C. This is syntactically incorrect. The filter function is a method of the DataFrame class and must be called on a DataFrame instance (e.g., employeesdf.filter(...)).
D. The .collect() action retrieves the entire filtered dataset to the driver node's memory as a list of Row objects. This is not ideal for "showing" data and can cause an OutOfMemoryError with large datasets. ---
1. Apache Spark 3.4.1 Documentation, pyspark.sql.DataFrame.filter: "Filters rows using the given condition. where() is an alias for filter()." This confirms that filter is the correct method for this operation.
2. Apache Spark 3.4.1 Documentation, pyspark.sql.DataFrame.show: "Prints the first n rows to the console. ... This is a lazy operation; use cache or eager evaluation if you want to force eager evaluation." This identifies .show() as the appropriate action to display the results of the transformation.
3. Databricks Documentation, "Transformations and actions": "Transformations are lazy operations, meaning that they don’t execute until you run an action. ... An action is an operation that returns a value, such as count, collect, or show." This reference clarifies why option B is incomplete, as it lacks an action to trigger the lazy filter transformation.
4. Apache Spark 3.4.1 Documentation, pyspark.sql.DataFrame.collect: "Returns all the records as a list of Row." The documentation also implicitly warns against its use on large datasets: "Collecting large datasets can result in an OutOfMemoryError." This supports why .show() is preferred over .collect() for displaying data.
Keep practicing
Take the free CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice test
Ten exam-style questions with answers and explanations, plus the exam facts and study guides.
More questions
Other CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice questions
- Question 1You have: DataFrame A: 128 GB of transactions DataFrame B: 1 GB user lookup table Which s…
- Question 2A Spark DataFrame df is cached using the MEMORY_AND_DISK storage level, but the DataFrame…
- Question 344 of 55. A data engineer is working on a real-time analytics pipeline using Spark Struct…
- Question 7A developer wants to test Spark Connect with an existing Spark application. What are the …
- Question 8A data engineer is running a Spark job to process a dataset of 1 TB stored in distributed…
- Question 10A data engineer is building a Structured Streaming pipeline and wants the pipeline to rec…
