An IT services company is working on a project to develop an AI-based customer support system. During data preparation, the project manager needs to…

Removing records (or rows) with missing values, a technique known as listwise or casewise deletion, is a common and effective strategy for handling missing data, provided the amount of data being removed is minimal. This approach is straightforward and prevents the introduction of artificial data or bias that can arise from imputation methods. When the percentage of affected records is small (e.g., <5%), their removal is unlikely to significantly alter the overall data distribution, thus preserving the integrity of the dataset used for training the AI model.

A. Ignoring missing data can cause many machine learning algorithms to fail or produce errors, and it can introduce significant bias if the missingness is not random.

B. Duplicating existing data to fill gaps is a poor practice that creates artificial correlations and severely biases the model, leading to overfitting and unreliable performance.

C. Filling missing values with zeros without analysis is risky, as zero may be a meaningful value in the data's context, and this can skew the dataset's statistical properties.

1. Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer.

In discussions on data preprocessing, the authors describe the common approaches to missing data. They note that deleting records is a simple and often acceptable method when the number of cases with missing data is small. (See Chapter 1 for general data preprocessing concepts).

2. Stanford University. (n.d.). CS229 Machine Learning Course Notes: Data Preprocessing.

Course materials frequently cover data preprocessing steps. In the context of missing values, it is taught that if only a small number of samples have missing features, the simplest and often sufficient solution is to discard those samples from the dataset.

3. McKinney, W. (2017). Python for Data Analysis (2nd ed.). O'Reilly Media.

In Chapter 7: Data Cleaning and Preparation, Section 7.1 Handling Missing Data, the author explains that dropna (the function for removing missing values) is a common first step. It is presented as a valid, though potentially blunt, instrument for cleaning data, especially when the loss of data is not significant.

4. García-Laencina, P. J., Sancho-Gómez, J. L., & Figueiras-Vidal, A. R. (2010). Pattern classification with missing data: a review. Neural Computing and Applications, 19(2), 263–282. https://doi.org/10.1007/s00521-009-0295-6

This review paper discusses various methods for handling missing data. In Section 2.1, "Deleting cases or variables," the authors describe listwise deletion as the most frequent method used due to its simplicity, noting its validity when the missing data constitutes a small part of the complete dataset.