An ML engineer needs to use an ML model to predict the price of apartments in a specific location. Which metric should the ML engineer use to evaluate the model's performance?

The task of predicting apartment prices is a regression problem, as the target variable (price) is a continuous numerical value. Mean Absolute Error (MAE) is a standard metric used to evaluate the performance of regression models. MAE calculates the average of the absolute differences between the predicted values and the actual values. This provides a straightforward, interpretable measure of the model's average prediction error in the same units as the target variable (e.g., dollars), making it highly suitable for this scenario.

A. Accuracy: This metric is used for classification tasks to measure the proportion of correct predictions. It is unsuitable for evaluating continuous value predictions like price.
B. Area Under the ROC Curve (AUC): AUC is a performance metric for binary classification models. It evaluates a model's ability to distinguish between positive and negative classes.
C. F1 score: The F1 score is the harmonic mean of precision and recall. It is a metric used to evaluate classification models, particularly in cases of class imbalance.

1. Amazon SageMaker Developer Guide: "For regression tasks

Amazon SageMaker automatically reports the mean squared error (MSE)

root mean square error (RMSE)

mean absolute error (MAE)

mean absolute percentage error (MAPE)

and the R-squared (R2) score." This source explicitly lists MAE as a primary metric for regression tasks within the AWS ecosystem.

Source: Amazon Web Services. (2024). Metrics for regression. Amazon SageMaker Developer Guide. Retrieved from https://docs.aws.amazon.com/sagemaker/latest/dg/notebooks-auto-ml-v2-metrics.html#notebooks-auto-ml-v2-regression

2. Stanford University Courseware (CS229): In the context of linear regression

evaluation involves minimizing a cost function. While Mean Squared Error (MSE) is often used

Mean Absolute Error (MAE) is a common alternative

particularly when robustness to outliers is desired. Both are fundamental regression loss functions and evaluation metrics. Classification metrics like Accuracy

AUC

and F1 are discussed separately for logistic regression and other classification algorithms.

Source: Ng

A. (2023). CS229: Machine Learning - Lecture Notes 1. Stanford University. pp. 4-5. Retrieved from https://cs229.stanford.edu/mainnotes.pdf