DATABRICKS-MACHINE-LEARNING-ASSOCIATE practice question 15 of 15

A data scientist has been given an incomplete notebook from the data engineering team. The notebook uses a Spark DataFrame spark_df on which the data…

Choose your answer, then check it against the explanation.

A data scientist has been given an incomplete notebook from the data engineering team. The notebook uses a Spark DataFrame spark_df on which the data scientist needs to perform further feature engineering. Unfortunately, the data scientist has not yet learned the PySpark DataFrame API. Which of the following blocks of code can the data scientist run to be able to use the pandas API on Spark?

Answer options

Correct Answer:

Explanation

The pyspark.pandas library is designed to provide a familiar pandas-like API that executes on a Spark cluster. To work with an existing Spark DataFrame (sparkdf) using this API, it must first be converted into a pandas-on-Spark DataFrame. The correct and idiomatic way to do this is by passing the Spark DataFrame directly to the pyspark.pandas.DataFrame constructor. This creates a new DataFrame object that supports pandas syntax but leverages Spark's distributed engine for computation, avoiding the need to collect the entire dataset onto the driver node.

Why Incorrect

B. ps.topandas() is not a valid top-level function in the pyspark.pandas library. The topandas() method exists on a pandas-on-Spark DataFrame object itself.

C. sparkdf.tosql() is used to register the DataFrame as a temporary view for SQL queries, not to enable the pandas API.

D. The standard pandas library (import pandas as pd) constructor pd.DataFrame() does not natively accept a distributed Spark DataFrame as input.

E. sparkdf.topandas() converts the Spark DataFrame to a standard, in-memory pandas DataFrame. This collects all data to the driver, losing the benefits of distributed processing for subsequent operations.

---

References

1. Apache Spark Documentation

pyspark.pandas.DataFrame: The official API documentation for the pyspark.pandas.DataFrame constructor explicitly lists Spark DataFrame as a valid input type for the data parameter.

Source: Apache Spark 3.4.1 Documentation > API Reference > pyspark.pandas > DataFrame > pyspark.pandas.DataFrame.

Quote: "Create a pandas-on-Spark DataFrame. ... Parameters: data : numpy.ndarray

dict

pandas.DataFrame

pandas.Series or Spark DataFrame"

2. Databricks Documentation

Pandas API on Spark: The official Databricks guide demonstrates converting between Spark DataFrames and pandas-on-Spark DataFrames.

Source: Databricks Documentation > Docs > Development > Libraries

notebooks

and Git > Pandas API on Spark.

Section: "From/to Spark DataFrames". This section shows that you can create a pandas-on-Spark DataFrame from a Spark DataFrame using ps.DataFrame(sparkdf) or ps.fromspark(sparkdf).

3. Databricks Documentation

toPandas() Method: The documentation clarifies that the .toPandas() method (aliased as .topandas() for consistency) on a Spark DataFrame collects data to the driver

which is fundamentally different from using the distributed pandas API on Spark.

Source: Databricks Documentation > Docs > Apache Spark > Spark SQL > DataFrames and Datasets > pyspark.sql.DataFrame.toPandas.

Quote: "Returns the contents of this DataFrame as Pandas pandas.DataFrame. This method should only be used if the resulting Pandas pandas.DataFrame is expected to be small

as all the data is loaded into the driver’s memory."

Question 15 of 15

Keep practicing

Take the free DATABRICKS-MACHINE-LEARNING-ASSOCIATE practice test

Ten exam-style questions with answers and explanations, plus the exam facts and study guides.

Start the free practice test