A data scientist has created a linear regression model that uses log(price) as a label variable. Using this model, they have performed inference and…

The model was trained to predict the logarithm of the price, log(price). Consequently, its predictions are on a logarithmic scale. The Root Mean Squared Error (RMSE) is calculated based on the difference between true and predicted values. To obtain an RMSE that is interpretable in the original currency units (e.g., dollars), the error must be calculated on the original price scale. This requires applying the inverse of the logarithmic transformation, which is exponentiation, to both the predicted values and the true label values before they are passed to the evaluator. This converts both columns back to the price scale, allowing the RMSE to be computed in a directly comparable and meaningful way.

A. Exponentiating the final RMSE value is a mathematically incorrect operation that does not properly transform the error back to the original scale.

B. The model's predictions are already on the log scale, so taking the logarithm again would be an erroneous double transformation.

C. Evaluating the Mean Squared Error (MSE) instead of RMSE does not address the fundamental problem of the evaluation being performed on the incorrect (logarithmic) scale.

1. Databricks Documentation

PySpark API Reference for pyspark.ml.evaluation.RegressionEvaluator: The documentation specifies that the evaluator computes metrics based on an input DataFrame with predictionCol and labelCol. It operates directly on the data provided. This implies that if the data in these columns is on a transformed scale (like a log scale)

the user is responsible for applying the appropriate inverse transformation to the columns before evaluation to get a metric on the original scale.

Source: Apache Spark 3.5.0 documentation (as used by Databricks)

pyspark.ml.evaluation.RegressionEvaluator.

2. University Courseware on Statistical Modeling: In statistical modeling

when the response variable Y is transformed (e.g.

log(Y))

the model produces predictions on the transformed scale. To obtain a prediction for Y in its original units

one must back-transform the prediction. For a log transformation

the back-transformation is exponentiation. This principle is fundamental for both prediction and the evaluation of prediction error.

Source: PennState

STAT 501: Regression Methods

Lesson 9: Transformations. The course notes explicitly state: "To transform a predicted value for log Y back to a predicted value for Y

we need to take the antilog of the predicted value." This directly supports applying an exponential function before calculating error metrics.