CT-AI practice question 7 of 15

“BioSearch” is creating an Al model used for predicting cancer occurrence via examining X-Ray images. The accuracy of the model in isolation has been…

“BioSearch” is creating an Al model used for predicting cancer occurrence via examining X-Ray images. The accuracy of the model in isolation has been found to be good. However, the users of the model started complaining of the poor quality of results, especially inability to detect real cancer cases, when put to practice in the diagnosis lab, leading to stopping of the usage of the model. A testing expert was called in to find the deficiencies in the test planning which led to the above scenario. Which ONE of the following options would you expect to MOST likely be the reason to be discovered by the test expert? SELECT ONE OPTION

Answer options

A: A lack of similarity between the training and testing data. Correct answer

B: The input data has not been tested for quality prior to use for testing.

C: A lack of focus on choosing the right functional-performance metrics.

D: A lack of focus on non-functional requirements testing.

Correct Answer:

A. A lack of similarity between the training and testing data.

Explanation

The scenario describes a classic generalization problem, where a model performs well on the data it was trained and tested on but fails on new, real-world data. The most likely cause is that the dataset used for development and testing was not representative of the data encountered in the production environment (the diagnosis lab). This discrepancy, known as data drift or a lack of data similarity, means the model learned patterns specific to the initial dataset that do not apply to the real patient population, leading to a sharp decline in practical performance, especially its ability to detect true positives (low recall).

Why Incorrect

B. While poor quality test data is a problem, the core issue here is the mismatch between the test environment and production, not just the quality of the test data itself.
C. The problem is not the choice of metric but the model's actual poor performance ("inability to detect real cancer cases"), which indicates a functional failure like low recall.
D. The issue described—failing to detect cancer cases—is a critical functional failure, not a non-functional one like slow performance or high memory usage.

References

1. ISTQB® Certified Tester AI Testing (CT-AI) Syllabus Version 1.0

Section 2.3.1

"Data Preparation and Quality"

p. 19. This section states

"The training

validation

and test data should be representative of the data that the AI-based system will process in production."

2. ISTQB® Certified Tester AI Testing (CT-AI) Syllabus Version 1.0

Section 1.2.3

"Generalization"

p. 9. This section explains that a model's ability to generalize is its ability to perform well on new

unseen data

and failure to do so is a key challenge.

3. Goodfellow

Bengio

& Courville

A. (2016). Deep Learning. MIT Press. Chapter 5

"Machine Learning Basics

" Section 5.2

discusses the importance of the training and test data being drawn from the same distribution for performance to be meaningful. (p. 110).

Keep practicing

Take the free CT-AI practice test

Ten exam-style questions with answers and explanations, plus the exam facts and study guides.

Start the free practice test