PMI-CPMAI practice question 23 of 25
During the configuration management of an AI/machine learning (ML) model, the team has observed inconsistent performance metrics across different test datasets. What will cause the inconsistency issue?
Full set: 102 questionsInconsistent performance metrics across different test datasets is a hallmark symptom of an overfit model. Overfitting occurs when a model learns the training data too well, including its noise and specific idiosyncrasies, rather than the underlying general pattern. This results in a model with high variance, meaning its performance is highly sensitive to the specific data it is evaluated on. Consequently, it performs well on data similar to the training set but poorly and inconsistently on new, unseen datasets with slightly different characteristics, leading to the observed performance fluctuations.
B. Low variance in the test results: This is the opposite of the problem. Low variance would imply consistent and stable performance metrics across different datasets.
C. Insufficient model complexity: This causes underfitting, where the model is too simple to capture the data's underlying structure. It would likely perform poorly, but consistently poorly, across all datasets.
D. Incorrect data preprocessing steps: While this can lead to poor model performance, overfitting is the specific modeling phenomenon that directly explains the high variance and inconsistent results across different test sets. ---
1. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press.
Reference: Chapter 5, "Machine Learning Basics," Section 5.2.2, "Capacity, Overfitting and Underfitting," pp. 110-111. The text explains that overfitting is characterized by a large gap between training error and test error. This sensitivity to the training data means the model has not generalized well, and its performance will vary significantly on different unseen test sets.
2. Stanford University. (n.d.). CS229: Machine Learning - Lecture Notes: Bias, Variance, and Error Analysis.
Reference: Part IV, "Bias, Variance, and Error Analysis," pp. 9-10. The notes define high variance (a key characteristic of overfitting) as a model's excessive sensitivity to the training data. It states that models with high variance can have vastly different performance on different test sets, directly explaining the inconsistency observed in the question.
3. Domingos, P. (2012). A few useful things to know about machine learning. Communications of the ACM, 55(10), 78–87.
Reference: Section 4, "Overfitting: The Bugbear of Machine Learning," p. 80. Domingos describes overfitting as the primary challenge in machine learning where the model fails to generalize. This failure to generalize manifests as performance that is not robust or consistent when applied to new data samples, such as different test datasets.
Keep practicing
Take the free PMI-CPMAI practice test
Ten exam-style questions with answers and explanations, plus the exam facts and study guides.
More questions
Other PMI-CPMAI practice questions
- Question 1A national health insurance company is embarking on a complex AI project to assist in coo…
- Question 2A project manager is tasked with overseeing the implementation of an AI model for financi…
- Question 3A financial services firm is operationalizing an AI-driven fraud detection system. The pr…
- Question 4A financial services firm is integrating AI to enhance fraud detection. To oversee data e…
- Question 5A project manager is considering different project management approaches for an AI soluti…
- Question 6A healthcare project manager is evaluating whether to implement an AI-powered diagnostic …
