DATABRICKS-MACHINE-LEARNING-ASSOCIATE practice question 11 of 15

A data scientist has written a feature engineering notebook that utilizes the pandas library. As the size of the data processed by the notebook…

Choose your answer, then check it against the explanation.

A data scientist has written a feature engineering notebook that utilizes the pandas library. As the size of the data processed by the notebook increases, the notebook's runtime is drastically increasing, but it is processing slowly as the size of the data included in the process increases. Which of the following tools can the data scientist use to spend the least amount of time refactoring their notebook to scale with big data?

Answer options

Correct Answer:

Explanation

The pandas API on Spark is specifically designed to provide a familiar, pandas-like interface that executes on a distributed Spark engine. This allows data scientists to scale their existing pandas code to handle large datasets with minimal code modifications. The primary change required is often just altering the import statement from import pandas as pd to import pyspark.pandas as ps. This directly addresses the core requirement of minimizing refactoring time while achieving scalability, making it the most efficient solution for the described problem.

Why Incorrect

A. PySpark DataFrame API: Requires a significant rewrite of the code because its API, syntax, and execution model are fundamentally different from the pandas API.

C. Spark SQL: Involves translating imperative pandas code into declarative SQL queries, which constitutes a major refactoring effort and a change in programming paradigm.

D. Feature Store: Is a centralized repository for storing, managing, and serving features, not a tool for executing the feature engineering computation itself. The computation still needs to be scaled.

References

1. Databricks Official Documentation

"What is the pandas API on Spark?": "The pandas API on Spark allows you to scale your pandas workload to any size by running it distributed on one or more nodes. If you are already familiar with pandas and want to leverage Spark for big data

the pandas API on Spark makes you immediately productive and allows you to migrate your applications without modifying the code."

Source: Databricks Documentation > Docs > Pyspark > Pandas API on Spark > What is the pandas API on Spark?

2. Databricks Official Documentation

"From pandas to pandas API on Spark": "If you are a pandas user

you can get started with the pandas API on Spark with a single line of code change... The pandas API on Spark works with pandas objects such as Series

DataFrame

and Index. You can create them in a similar way as pandas."

Source: Databricks Documentation > Docs > Pyspark > Pandas API on Spark > User Guide > From pandas to pandas API on Spark

3. Apache Spark Official Documentation

"pandas API on Spark Overview": "It is useful for pandas users who would like to scale out their pandas workload to big data. It is also useful for PySpark users who would like to leverage pandas API... pandas API on Spark enables you to use pandas-like APIs to work with PySpark DataFrame."

Source: Apache Spark 3.4.1 Documentation > PySpark > pandas API on Spark > Overview section.

Question 11 of 15

Keep practicing

Take the free DATABRICKS-MACHINE-LEARNING-ASSOCIATE practice test

Ten exam-style questions with answers and explanations, plus the exam facts and study guides.

Start the free practice test