“BioSearch” is creating an Al model used for predicting cancer occurrence via examining X-Ray images. The accuracy of the model in isolation has been…

The question asks which deficiency is most likely to be discovered by the test expert given the

scenario of poor real-world performance despite good isolated accuracy.

A lack of similarity between the training and testing data (A): This is a common issue in ML where the

model performs well on training data but poorly on real-world data due to a lack of

representativeness in the training data. This leads to poor generalization to new, unseen data.

The input data has not been tested for quality prior to use for testing (B): While data quality is

important, this option is less likely to be the primary reason for the described issue compared to the

representativeness of training data.

A lack of focus on choosing the right functional-performance metrics (C): Proper metrics are crucial,

but the issue described seems more related to the data mismatch rather than metric selection.

A lack of focus on non-functional requirements testing (D): Non-functional requirements are

important, but the scenario specifically mentions issues with detecting real cancer cases, pointing

more towards data issues.

Reference:

ISTQB CT-AI Syllabus Section 4.2 on Training, Validation, and Test Datasets emphasizes the

importance of using representative datasets to ensure the model generalizes well to real-world data.

Sample Exam Questions document, Question #40 addresses issues related to data

representativeness and model generalization.