DATABRICKS-MACHINE-LEARNING-ASSOCIATE Practice Questions

Spark provides pyspark.ml.evaluation.RegressionEvaluator to compute standard regression metrics directly from a DataFrame that contains a prediction column and a label column.

Setting labelCol="actual", predictionCol="prediction", and metricName="rmse" returns the root-mean-squared-error in one call:

from pyspark.ml.evaluation import RegressionEvaluator

rmse = (RegressionEvaluator(labelCol="actual",

predictionCol="prediction",

metricName="rmse")

.evaluate(predsdf))

Option C shows exactly this pattern, therefore correctly assigns the RMSE to the variable rmse.

A. Uses labelCol="label" (column does not exist) → runtime error.

B. Requests metric "mse" (mean-squared-error), not "rmse".

D. Manually averages squared errors but forgets the final square-root, so it returns MSE, not RMSE.

1. Apache Spark™ 3.5.0 Documentation – ML > Evaluators > RegressionEvaluator

example code (Section “RegressionEvaluator”

first code block).

https://spark.apache.org/docs/3.5.0/api/python/

2. Databricks Academy – Machine Learning Fundamentals

Module “Model evaluation in Spark ML”

slide 18: “rmse = RegressionEvaluator(metricName='rmse'

…).evaluate(df)”

3. Zaharia et al.

“Apache Spark: A Unified Engine for Big Data Processing”

CACM 59(11)

2016

p.60 – discusses built-in ML evaluators (RMSE example). DOI:10.1145/2934664

For large-scale datasets, solving the linear regression problem with a direct analytical method like the normal equation (which involves matrix decomposition) is computationally infeasible. The X^T X matrix can become too large to fit in memory or too expensive to invert.

To overcome this, Spark ML's LinearRegression implementation utilizes iterative optimization algorithms. These methods, such as L-BFGS (Limited-memory Broyden–Fletcher–Goldfarb–Shanno), start with an initial guess for the model's weights and repeatedly update them in a direction that minimizes the loss function. The core computations, like calculating the gradient, can be efficiently distributed across the cluster, making this approach highly scalable for large data.

A. Logistic regression is a distinct algorithm used for classification tasks, not a method for training a linear regression model.

B. This is false. The primary purpose of Spark ML is to provide scalable, distributed implementations of machine learning algorithms, including linear regression.

D. The least-squares method describes the objective function (minimizing the sum of squared errors) that linear regression solves, not the computational strategy for distributed training.

E. Singular value decomposition is a matrix decomposition technique. The question's premise correctly states that such methods do not scale well for this problem.

1. Official Apache Spark Documentation: The documentation for LinearRegression in MLlib explicitly details the optimization algorithms used. It states

"The implementation is based on the MLlib LBFGS optimizer for L2-regularized linear regression... For unregularized linear regression

the implementation uses a wrapper for the NormalEquation and Cholesky solvers... The normal equation solver is limited to at most 4096 features." This confirms that for a large number of variables (beyond 4096)

the iterative L-BFGS optimizer is the method employed.

Source: Apache Spark 3.5.0 MLlib Guide

Classification and regression > Linear methods > Linear regression > Mathematical detail.

2. Academic Publication: The foundational paper on Spark's machine learning library highlights the design choice of using iterative methods for scalability. "For many ML algorithms

we can express the optimization problem as a sum of loss terms... and solve it with gradient descent. The gradient can be computed on a cluster by summing gradients computed on subsets of the data in parallel... MLlib has a general-purpose gradient descent optimizer." L-BFGS is a more advanced quasi-Newton iterative optimization method built on these principles.

Source: Meng

X.

Bradley

J.

Yavuz

B.

Sparks

E.

Venkataraman

S.

Liu

D.

... & Talwalkar

A. (2016). MLlib: Machine Learning in Apache Spark. Journal of Machine Learning Research

17(34)

1-7. (Section 3.1: Implementation).

3. Official Databricks Documentation: The Databricks documentation on Linear Regression also confirms the use of iterative optimization. "The training algorithm uses the L-BFGS optimizer." This directly points to an iterative optimization approach as the standard for their implementation.

Source: Databricks Documentation

Machine Learning Guide > MLflow > MLflow models > spark.mllib > Linear Regression.

Gradient Boosting is an ensemble learning technique that builds models in a sequential, stage-wise fashion. Each new model (typically a decision tree) is trained to correct the errors or residuals of the ensemble of all previously trained models. This means the construction of the second tree depends entirely on the outcome of the first tree, the third tree depends on the first two, and so on. This inherent sequential dependency, where each iteration requires the output from the previous one, is the fundamental reason why the overall tree-building process cannot be easily parallelized.

A. Parallelization is a general computing concept not restricted to linear algebra; many non-linear algorithms can be parallelized.

B. Many parallel algorithms work by distributing data across nodes; needing access to all data does not inherently prevent parallelization.

C. The calculation of gradients across data points can often be parallelized; this is not the primary bottleneck for the overall algorithm.

1. Official Vendor Documentation (Apache Spark

the foundation of Databricks ML): In the MLlib programming guide for Gradient-Boosted Trees (GBTs)

the documentation states: "GBTs train decision trees one by one

where each new tree helps to correct the errors of the previously trained ensemble of trees. The training of each tree is dependent on the previously trained trees." This highlights the sequential nature of the algorithm.

Source: Apache Spark 3.5.0 MLlib Guide

"Classification and regression - Gradient-boosted trees (GBTs)

" Algorithm section.

2. Academic Publication: The canonical textbook "The Elements of Statistical Learning" describes the gradient boosting algorithm as a forward stagewise procedure. The algorithm is presented as a loop from m=1 to M

where each step m explicitly uses the model f{m-1}(x) from the preceding step to compute the residuals and fit the next base learner hm(x). This iterative dependency is fundamental to the method.

Source: Hastie

T.

Tibshirani

R.

& Friedman

J. (2009). The Elements of Statistical Learning: Data Mining

Inference

and Prediction. Springer. Chapter 10

"Boosting and Additive Trees

" Algorithm 10.3

page 359.

3. Academic Publication: The paper introducing XGBoost

a highly optimized gradient boosting implementation

discusses this challenge. It explains that while the inter-tree creation is sequential

they achieve scalability by parallelizing the process within each tree's construction

specifically the split-finding part. This distinction confirms that the overall boosting process is iterative and not parallelizable across trees.

Source: Chen

T.

& Guestrin

C. (2016). XGBoost: A Scalable Tree Boosting System. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Section 2.1

"Regularized Learning Objective." DOI: https://doi.org/10.1145/2939672.2939785

The code snippet begins by grouping a Spark DataFrame using df.groupBy("groupid"), which creates a GroupedData object. The goal is to apply a function to each of these groups in parallel. The applyInPandas function is the correct method from the Pandas Function API for this "split-apply-combine" pattern. It takes a function that receives a pandas DataFrame for each group, performs computations (like loading and applying a model), and returns a new pandas DataFrame. This directly matches the requirement to apply the applymodel function to each group.

B. groupedApplyInPandas: This is not a valid function name in the PySpark API. It is a distractor that closely resembles the correct function name.

C. mapInPandas: This function operates on partitions of a DataFrame, not on groups created by groupBy. It is used for vectorizing operations on subsets of data without a grouping key.

D. predict: This is a generic method name for inference on a model object, not a method available on a PySpark GroupedData object to apply a custom function.

1. Databricks Documentation

"Pandas function APIs": Under the "Grouped map" section

the documentation states

"Grouped map Pandas UDFs are used with groupBy().applyInPandas() which implements the 'split-apply-combine' pattern." This directly confirms that applyInPandas is the correct function to use after a groupBy operation.

Source: Databricks Documentation > Apache Spark > Development > Pandas on Spark > Pandas function APIs.

2. Apache Spark Documentation

pyspark.sql.GroupedData.applyInPandas: The official API documentation specifies the method signature and usage: GroupedData.applyInPandas(func

schema). It explicitly states that this function "Applies a Python function func to each group of the DataFrame."

Source: Apache Spark 3.5.1 Documentation > API Docs > Python > pyspark.sql > pyspark.sql.GroupedData.applyInPandas.

3. Apache Spark Documentation

pyspark.sql.DataFrame.mapInPandas: To contrast with the correct answer

the documentation for mapInPandas shows it is a method on a DataFrame

not a GroupedData object. It "Maps an iterator of pandas DataFrames to an iterator of pandas DataFrames

" confirming it works on partitions

not groups.

Source: Apache Spark 3.5.1 Documentation > API Docs > Python > pyspark.sql > pyspark.sql.DataFrame.mapInPandas.

The DataFrame.randomSplit() method is the designated Spark operation for randomly partitioning a DataFrame into multiple, non-overlapping subsets. It accepts a list of weights that define the approximate proportions for each split and an optional seed for reproducibility. This function is the standard and most direct approach for creating training and test DataFrames from a single source DataFrame in a Spark-based machine learning workflow. The resulting splits are returned as a list of DataFrames.

A. TrainValidationSplit is a high-level hyperparameter tuning class that internally performs a single split, but it is not the fundamental DataFrame operation itself.

B. DataFrame.where is a filtering operation used to select rows based on a specific condition; it does not perform a random split.

C. CrossValidator is a hyperparameter tuning class that performs k-fold cross-validation, creating multiple splits, not a single train/test split operation.

D. TrainValidationSplitModel is the object returned after fitting a TrainValidationSplit estimator; it is a result, not an operation for splitting data.

1. Apache Spark Official Documentation. The PySpark API reference for pyspark.sql.DataFrame.randomSplit explicitly states its purpose: "Randomly splits this DataFrame with the provided weights." It provides a clear example of creating two splits.

Source: Apache Spark 3.5.0 Documentation

pyspark.sql.DataFrame.randomSplit.

2. Databricks Official Documentation. The documentation on preparing data for machine learning demonstrates this exact use case. It shows an example: traindf

testdf = originaldf.randomSplit([.8

.2]

seed=42)

directly confirming its use for creating training and test sets.

Source: Databricks Documentation > Machine Learning > Get started > Tutorial: ML end-to-end on Databricks > Step 2: Prepare the data.

3. University Courseware. Reputable data science courses that use Spark consistently teach randomSplit for this purpose. For example

materials from UC Berkeley's Data 100 course show randomSplit as the standard method for creating training and test sets in PySpark.

Source: University of California

Berkeley

Data 100: Principles & Techniques of Data Science

Course Materials (e.g.

Fall 2022

Lecture 20 on Parallel Processing).

The user's objective is to perform hyperparameter tuning by randomly selecting values from a specified search space. The GridSearchCV operation performs an exhaustive search, evaluating every possible combination of the provided hyperparameter values. This does not meet the requirement for random selection. The correct tool in the scikit-learn library for this task is RandomizedSearchCV, which samples a fixed number of parameter settings from the specified distributions. Replacing GridSearchCV with RandomizedSearchCV directly accomplishes the desired random tuning process.

B. crossvalidate is a function used to evaluate a model's performance with cross-validation but does not perform hyperparameter tuning over a search space.

C. ParameterGrid is a helper class that generates combinations of parameters but does not perform the actual model fitting or cross-validated search.

D. Changing the randomstate argument affects reproducibility but does not change the fundamental search algorithm from an exhaustive grid search to a random search.

E. GridSearchCV requires a list of discrete values for its parameters, not a statistical distribution. This change would cause an error and not alter the search method.

1. Official Databricks Documentation: In the "Hyperparameter tuning with scikit-learn and MLflow" guide

Databricks explains the two main approaches: "GridSearchCV...tries all possible combinations" and "RandomizedSearchCV...tries a fixed number of random combinations." This directly supports using RandomizedSearchCV for random selection.

Source: Databricks Documentation > Machine Learning > Model training > Hyperparameter tuning > scikit-learn and MLflow > Section: "Tune hyperparameters".

2. Official scikit-learn Documentation: The user guide explicitly contrasts the two methods. It states

"While GridSearchCV is a good method for finding parameters in a grid

RandomizedSearchCV can be preferred... It samples a given number of candidates from a parameter space with a specified distribution."

Source: scikit-learn User Guide > 3.2. Tuning the hyper-parameters of an estimator > Section 3.2.3. "Randomized search".

3. University Courseware: Stanford University's CS229 (Machine Learning) course materials often discuss hyperparameter tuning strategies

explaining that Grid Search can be computationally expensive and that Randomized Search is often a more efficient alternative for exploring a large parameter space.

Source: Stanford University

CS229: Machine Learning

Lecture Notes

"Advice for Applying Machine Learning

" Section on "Model Selection and Hyperparameter Tuning." (Illustrates the conceptual difference between grid and random search).

The primary performance benefit of vectorized pandas UDFs over standard, row-at-a-time PySpark UDFs is their execution model. Standard UDFs process data one row at a time, incurring significant overhead from data serialization and deserialization between the JVM and the Python process for every single row. In contrast, vectorized pandas UDFs operate on batches of data. They leverage Apache Arrow to efficiently transfer columnar data as a batch to the Python process, where it is represented as a pandas Series or DataFrame. This batch processing significantly reduces the serialization overhead and allows for the use of efficient, vectorized library functions, leading to substantial performance improvements.

A. While vectorized pandas UDFs use type hints for their signature, specifying return types is also a feature of standard UDFs, so this is not a unique benefit.

C. Using the pandas API is a feature enabled by batch processing, but the batch processing itself (Option B) is the more fundamental performance benefit.

D. Both standard and vectorized UDFs are designed to be applied to distributed Spark DataFrames; this is a common characteristic, not a distinguishing benefit.

E. Both UDF types process data in memory on worker nodes. Spilling to disk is a general Spark memory management mechanism, not a specific advantage of one UDF type.

1. Databricks Documentation

"Pandas UDFs": "Vectorized UDFs offer better performance than row-at-a-time Python UDFs... Vectorized UDFs execute by splitting a Spark DataFrame or column into batches

loading a batch of rows into memory as a pandas.Series or pandas.DataFrame

and applying the user-defined function to the pandas.Series or pandas.DataFrame." This directly contrasts the batch processing of vectorized UDFs with the row-at-a-time model.

2. Apache Spark Documentation

"Pandas UDFs (Arrow-optimized)": "Pandas UDFs allow vectorized execution that can increase performance up to 100x compared to row-at-a-time UDFs... Apache Arrow is used to transfer data and pandas is used to work with the data." This section explicitly states that the performance gain comes from vectorized (i.e.

batch) execution compared to the row-at-a-time alternative.

3. Karau

H.

Konwinski

A.

Wendell

P.

& Zaharia

M. (2015). Learning Spark: Lightning-Fast Big Data Analysis. O'Reilly Media. (Conceptual basis

though the book predates the modern Pandas UDF API

it explains the performance bottleneck of row-by-row processing in standard UDFs that Pandas UDFs were designed to solve). The principles discussed in Chapter 4 on RDDs and later on DataFrames highlight the costs of per-record serialization that batch operations mitigate.

Databricks AutoML is designed to automate the iterative and time-consuming tasks of the machine learning model development process. This includes data preprocessing, model training, hyperparameter tuning, and evaluation. The output of an AutoML experiment is a set of trained models ranked on a leaderboard, along with generated notebooks. While AutoML facilitates the next step by allowing one-click registration of the best model into the MLflow Model Registry, the actual deployment of that model—for example, creating a real-time serving endpoint or setting up a batch inference job—is a distinct, subsequent step in the MLOps lifecycle that must be performed by the user outside of the AutoML experiment itself.

A. Model tuning: This is incorrect. A core feature of Databricks AutoML is to automatically perform hyperparameter tuning across a range of algorithms to find the best-performing model.

B. Model evaluation: This is incorrect. AutoML automatically evaluates each model trial using relevant metrics (e.g., F1 score for classification, RMSE for regression) and presents the results in a leaderboard for comparison.

D. Exploratory data analysis: This is incorrect. As part of its output, Databricks AutoML generates a data exploration notebook which includes a statistical summary and visualizations of the training dataset, thus performing a form of automated EDA.

---

1. Databricks Official Documentation

"How Databricks AutoML works": This document outlines the automated steps performed by AutoML. The list includes: "Prepares the dataset for model training

" "Iterates to train and tune multiple models

" "Evaluates models

" and "Provides a Python notebook with the source code... including a data exploration notebook." Model deployment is not listed as an automated step.

Source: Databricks Machine Learning Guide > AutoML > How Databricks AutoML works.

2. Databricks Official Documentation

"Model serving with Databricks": This documentation describes model deployment as a separate process that occurs after a model has been trained and registered in the Model Registry. It details the steps to create and manage serving endpoints

which is distinct from the AutoML experiment workflow.

Source: Databricks Machine Learning Guide > MLflow > Model serving with Databricks.

3. The Big Book of MLOps (Databricks Ebook)

Chapter 2

"A Modern MLOps Architecture": This official resource presents a diagram of the MLOps lifecycle. The "Build & Train" stage

which includes AutoML

is shown as separate and preceding the "Model Deployment" and "Model Monitoring" stages. This clearly delineates deployment as an activity outside of the automated training experiment.

Source: Databricks Ebooks

"The Big Book of MLOps"

Page 15

Figure 2-1.

The decorator @pandasudf(resultschema, PandasUDFType.MAPITER) defines the predict function as a Pandas UDF that operates on an iterator of pandas DataFrames and returns an iterator of pandas DataFrames. According to the official Databricks and Apache Spark documentation, this specific type of UDF is applied to a Spark DataFrame using the mapInPandas transformation method. This method is designed for scenarios like model inference where an expensive state (e.g., a loaded model) can be initialized once per data partition and then used to process all records within that partition efficiently. The other options represent incorrect syntax for applying this type of UDF.

A. predict(sparkdf.columns): This is an incorrect invocation. It attempts to call the Python function directly with column names as strings, not the DataFrame data itself.

C. predict(Iterator(sparkdf)): This bypasses the Spark execution engine by trying to create a standard Python iterator from a Spark DataFrame, which is not the correct way to apply a UDF.

D. mapInPandas(predict(sparkdf.columns)): This syntax is invalid. It incorrectly calls the predict function with column objects and passes the result to mapInPandas.

E. predict(sparkdf.columns): This is incorrect as it passes a list of column objects to the UDF, which expects an iterator of pandas DataFrames supplied by the Spark engine.

1. Databricks Documentation

"Pandas UDFs (vectorized UDFs)": In the section "Iterator of DataFrames to Iterator of DataFrames UDF

" the documentation explicitly states

"This UDF type is supported only with DataFrame.mapInPandas." It provides an example with a function signature identical to the one in the question (def f(iterator: Iterator[pd.DataFrame]) -> Iterator[pd.DataFrame]:)

confirming that mapInPandas is the required method.

Source: Databricks Documentation > Docs > Apache Spark > PySpark > Pandas UDFs (vectorized UDFs).

2. Apache Spark Documentation

pyspark.sql.DataFrame.mapInPandas: The official Spark API reference describes mapInPandas as the method that "Maps an iterator of pandas DataFrames to another iterator of pandas DataFrames." This directly corresponds to the UDF type (MAPITER) and function signature provided in the question

establishing it as the correct transformation to use.

Source: Apache Spark 3.3.1 Documentation

pyspark.sql.DataFrame.mapInPandas.

3. Apache Spark Documentation

pyspark.sql.functions.pandasudf: In older versions of the documentation (e.g.

Spark 3.0)

the PandasUDFType.MAPITER is explicitly defined for use with DataFrame.mapInPandas to perform map operations on a DataFrame.

Source: Apache Spark 3.0.0 Documentation

pyspark.sql.functions.pandasudf.

Tree of Parzen Estimators (TPE) is a sequential, model-based (Bayesian) optimization algorithm. It iteratively builds a probabilistic model based on the history of evaluated hyperparameter sets and their corresponding performance. In each step, it uses this model to select the next set of hyperparameters that are most likely to yield an improvement according to an acquisition function (Expected Improvement). This "informed" selection process allows TPE to focus the search on more promising regions of the hyperparameter space, making it more efficient than methods that do not learn from past results.

A. Random Search samples hyperparameter configurations independently and randomly; it does not learn from past trials to inform future selections.

B. Halving Random Search is an early-stopping method that prunes poor configurations, but the initial selections are random and not informed by past results.

D. Grid Search exhaustively evaluates a predefined, static grid of values. The search path is fixed and does not adapt based on trial outcomes.

1. Official Databricks Documentation: Databricks Machine Learning Guide

"Hyperopt concepts". The documentation states

"Tree of Parzen Estimators (TPE) is a Bayesian approach that iteratively and adaptively chooses new hyperparameter settings to explore based on past results." This directly supports that TPE makes informed selections.

Source: Databricks Documentation > Machine Learning > Hyperparameter tuning > Hyperopt concepts.

2. Peer-Reviewed Academic Publication: Bergstra

J.

Bardenet

R.

Bengio

Y.

& Kégl

B. (2011). Algorithms for Hyper-Parameter Optimization. In Advances in Neural Information Processing Systems 24 (NIPS 2011).

Section: 2. Sequential Model-Based Optimization. The paper describes sequential model-based optimization (of which TPE is an example) as a formal process of using the history of (hyperparameter

loss) pairs to inform the choice of the next hyperparameter to evaluate.

3. Official Databricks Documentation: Databricks Machine Learning Guide

"Parallelize hyperparameter tuning with scikit-learn and MLflow". This guide contrasts TPE with Random Search

explaining

"Whereas random search is a parallelizable tuning algorithm

TPE makes decisions sequentially." This highlights the iterative

results-based nature of TPE.

Source: Databricks Documentation > Machine Learning > Hyperparameter tuning > Parallelize hyperparameter tuning with scikit-learn and MLflow.