CT-AI Practice Questions

A/B testing, also known as split testing, is a method used to compare two versions of a product or

system to determine which one performs better. It is widely used in web development, marketing,

and machine learning to optimize user experiences and model performance. Here’s why option C is

the least descriptive of an A/B testing scenario:

Understanding A/B Testing:

In A/B testing, two versions (A and B) of a system or feature are tested against each other. The

objective is to measure which version performs better based on predefined metrics such as user

engagement, conversion rates, or other performance indicators.

Application in Machine Learning:

In ML systems, A/B testing might involve comparing two different models, algorithms, or system

configurations on the same set of data to observe which yields better results.

Why Option C is the Least Descriptive:

Option C describes comparing the performance of an ML system on two different input datasets. This

scenario focuses on the input data variation rather than the comparison of system versions or

features, which is the essence of A/B testing. A/B testing typically involves a controlled experiment

with two versions being tested under the same conditions, not different datasets.

Clarifying the Other Options:

A . A comparison of two different websites for the same company to observe from a user acceptance

perspective: This is a classic example of A/B testing where two versions of a website are compared.

B . A comparison of two different offers in a recommendation system to decide on the more effective

offer for the same users: This is another example of A/B testing in a recommendation system.

D . A comparison of the performance of two different ML implementations on the same input data:

This fits the A/B testing model where two implementations are compared under the same

conditions.

Reference:

ISTQB CT-AI Syllabus, Section 9.4, A/B Testing, explains the methodology and application of A/B

testing in various contexts.

"Understanding A/B Testing" (ISTQB CT-AI Syllabus).

Data mislabeling occurs for several reasons, which can significantly impact the performance of

machine learning (ML) models, especially in supervised learning. According to the ISTQB Certified

Tester AI Testing (CT-AI) syllabus, mislabeling of data can be caused by the following factors:

Random errors by annotators – Mistakes made due to accidental misclassification.

Systemic errors – Errors introduced by incorrect labeling instructions or poor training of annotators.

Deliberate errors – Errors introduced intentionally by malicious data annotators.

Translation errors – Occur when correctly labeled data in one language is incorrectly translated into

another language.

Subjectivity in labeling – Some labeling tasks require subjective judgment, leading to inconsistencies

between different annotators.

Lack of domain knowledge – If annotators do not have sufficient expertise in the domain, they may

label data incorrectly due to misunderstanding the context.

Complex classification tasks – The more complex the task, the higher the probability of labeling

mistakes​.

Among the answer choices provided, "Lack of domain knowledge" (Option A) is the best answer

because expertise is essential to accurately labeling data in complex domains such as medical, legal,

or engineering fields.

Certified Tester AI Testing Study Guide Reference:

ISTQB CT-AI Syllabus v1.0, Section 4.5.2 (Mislabeled Data in Datasets)

ISTQB CT-AI Syllabus v1.0, Section 4.3 (Dataset Quality Issues)

A . Identifying suitable tests by looking at the complexity of the test cases.

While complexity analysis can help in selecting important test cases, it does not directly address the

issue of optimizing the entire regression suite effectively.

B . Using a random subset of tests.

Randomly selecting test cases may miss critical tests and does not ensure an optimized regression

suite. This approach lacks a systematic method for ensuring comprehensive coverage.

C . Automating test scripts using AI-based test automation tools.

Automation helps in running tests efficiently but does not inherently optimize the selection of tests

to prevent the suite from growing too large.

D . Using an AI-based tool to optimize the regression test suite by analyzing past test results.

This is the most effective approach as AI-based tools can analyze historical test data, identify

patterns, and prioritize tests that are more likely to catch defects based on past results. This method

ensures an optimized and manageable regression test suite by focusing on the most impactful test

cases.

Therefore, the correct answer is D because using an AI-based tool to analyze past test results is the

best option to optimize regression test selection and manage the size of the regression suite

effectively​​.

Defect prediction models aim to identify parts of the software that are likely to contain defects by

analyzing historical data and code quality metrics. The primary goal is to use this predictive

information to allocate testing and maintenance resources effectively. Let's break down why option

D is the correct choice:

Understanding Classification Models:

Classification models are a type of supervised learning algorithm used to categorize or classify data

into predefined classes or labels. In the context of defect prediction, the classification model would

classify parts of the code as either "defective" or "non-defective" based on the input features.

Input Data - Code Quality Metrics:

The input data for these classification models typically includes various code quality metrics such as

cyclomatic complexity, lines of code, number of methods, depth of inheritance, coupling between

objects, etc. These metrics help the model learn patterns associated with defects.

Historical Data:

Historical versions of the code along with their defect records provide the labeled data needed for

training the classification model. By analyzing this historical data, the model can learn which metrics

are indicative of defects.

Why Option D is Correct:

Option D specifies using a classification model to predict the presence of defects by using code

quality metrics as input data. This accurately describes the process of defect prediction using

historical bug data and quality metrics.

Eliminating Other Options:

A . Identifying the relationship between developers and the modules developed by them: This does

not directly involve predicting defects based on code quality metrics and historical data.

B . Search of similar code based on natural language processing: While useful for other purposes, this

method does not describe defect prediction using classification models and code metrics.

C . Clustering of similar code modules to predict based on similarity: Clustering is an unsupervised

learning technique and does not directly align with the supervised learning approach typically used

in defect prediction models.

Reference:

ISTQB CT-AI Syllabus, Section 9.5, Metamorphic Testing (MT), describes various testing techniques

including classification models for defect prediction.

"Using AI for Defect Prediction" (ISTQB CT-AI Syllabus, Section 11.5.1).

The process of developing a machine learning model typically involves the use of three types of

datasets:

Training Data: This is used to train the model, i.e., to learn the patterns and relationships in the data.

Validation Data: This is used to tune the model’s hyperparameters and to prevent overfitting during

the training process.

Test Data: This is used to evaluate the final model’s performance and to estimate how it will perform

on unseen data.

Let's analyze each option:

A . Training data - validation data - test data

This option correctly includes all three types of datasets used in the process of creating and

validating a model. The training data is used for learning, validation data for tuning, and test data for

final evaluation.

B . Training data - validation data

This option misses the test data, which is crucial for evaluating the model’s performance on unseen

data after the training and validation phases.

C . Training data - test data

This option misses the validation data, which is important for tuning the model and preventing

overfitting during training.

D . Validation data - test data

This option misses the training data, which is essential for the initial learning phase of the model.

Therefore, the correct answer is A because it includes all necessary datasets used during the process

of learning and creating the model: training, validation, and test data​​.

A . Regression

Regression is used to predict a continuous value and is not suitable for grouping people based on

similar tastes.

B . Association

Association is used to find relationships between variables in large datasets, often in the form of

rules (e.g., market basket analysis). It does not directly group individuals but identifies patterns of co-

occurrence.

C . Clustering

Clustering is an unsupervised learning method used to group similar data points based on their

features. It is ideal for identifying groups of people with similar tastes without prior knowledge of the

group labels. This technique will help the company segment its customer base effectively.

D . Classification

Classification is a supervised learning method used to categorize data points into predefined classes.

It requires labeled data for training, which is not the case here as we want to identify groups without

predefined labels.

Therefore, the correct answer is C because clustering is the most suitable method for grouping

people with similar tastes for targeted product marketing​​.

The question asks which deficiency is most likely to be discovered by the test expert given the

scenario of poor real-world performance despite good isolated accuracy.

A lack of similarity between the training and testing data (A): This is a common issue in ML where the

model performs well on training data but poorly on real-world data due to a lack of

representativeness in the training data. This leads to poor generalization to new, unseen data.

The input data has not been tested for quality prior to use for testing (B): While data quality is

important, this option is less likely to be the primary reason for the described issue compared to the

representativeness of training data.

A lack of focus on choosing the right functional-performance metrics (C): Proper metrics are crucial,

but the issue described seems more related to the data mismatch rather than metric selection.

A lack of focus on non-functional requirements testing (D): Non-functional requirements are

important, but the scenario specifically mentions issues with detecting real cancer cases, pointing

more towards data issues.

Reference:

ISTQB CT-AI Syllabus Section 4.2 on Training, Validation, and Test Datasets emphasizes the

importance of using representative datasets to ensure the model generalizes well to real-world data.

Sample Exam Questions document, Question #40 addresses issues related to data

representativeness and model generalization.

In a neural network, the activation value of a neuron is determined by a combination of inputs from

the previous layer, the weights of the connections, and the bias at the neuron level. Here’s a detailed

breakdown:

Inputs for Activation Value:

Activation Values of Neurons in the Previous Layer: These are the outputs from neurons in the

preceding layer that serve as inputs to the current neuron.

Weights Assigned to the Connections: Each connection between neurons has an associated weight,

which determines the strength and direction of the input signal.

Individual Bias at the Neuron Level: Each neuron has a bias value that adjusts the input sum, allowing

the activation function to be shifted.

Calculation:

The activation value is computed by summing the weighted inputs from the previous layer and

adding the bias.

Formula: z=∑(wi

⋅

⋅

ai)+bz = \sum (w_i \cdot a_i) + bz=∑(wi​

ai​)+b, where wiw_iwi​are the weights,

aia_iai​are the activation values from the previous layer, and bbb is the bias.

The activation function (e.g., sigmoid, ReLU) is then applied to this sum to get the final activation

value.

Why Option A is Correct:

Option A correctly identifies all components involved in computing the activation value: the

individual bias, the activation values of the previous layer, and the weights of the connections.

Eliminating Other Options:

B . Activation values of neurons in the previous layer, and weights assigned to the connections

between the neurons: This option misses the bias, which is crucial.

C . Individual bias at the neuron level, and weights assigned to the connections between the

neurons: This option misses the activation values from the previous layer.

D . Individual bias at the neuron level, and activation values of neurons in the previous layer: This

option misses the weights, which are essential.

Reference:

ISTQB CT-AI Syllabus, Section 6.1, Neural Networks, discusses the components and functioning of

neurons in a neural network.

"Neural Network Activation Functions" (ISTQB CT-AI Syllabus, Section 6.1.1).

The question refers to a problem where data used for an object detection ML system was labelled

incorrectly. This issue is most closely related to "accuracy issues." Here's a detailed explanation:

Accuracy Issues: The primary goal of labeling data in machine learning is to ensure that the model

can accurately learn and make predictions based on the given labels. Incorrectly labeled data directly

impacts the model's accuracy, leading to poor performance because the model learns incorrect

patterns.

Why Not Other Options:

Security Issues: This pertains to data breaches or unauthorized access, which is not relevant to the

problem of incorrect data labeling.

Privacy Issues: This concerns the protection of personal data and is not related to the accuracy of

data labeling.

Bias Issues: While bias in data can affect model performance, it specifically refers to systematic errors

or prejudices in the data rather than outright incorrect labeling.

Reference: This explanation is consistent with the concepts covered in the ISTQB CT-AI syllabus under

dataset quality issues and their impact on machine learning models​​.

AI-based systems can sometimes behave like black boxes, where the internal decision-making

process is unclear. Transparency refers to the ability to inspect and understand the training data,

algorithms, and decision-making process of the AI system.

Why is Option C Correct?

Transparency ensures that testers and stakeholders can review how an AI system was trained.

Access to training data is a key factor in transparency because it allows testers to analyze biases,

completeness, and representativeness of the dataset.

Transparency is an essential characteristic of explainable AI (XAI).

Having access to training data means that testers can investigate how data influences AI behavior.

Regulatory and ethical AI guidelines emphasize transparency.

Many AI ethics frameworks, such as GDPR and Trustworthy AI guidelines, recommend transparency

to ensure fair and explainable AI decision-making.

Why Other Options are Incorrect?

(A) Autonomy ❌

Autonomy refers to an AI system’s ability to make decisions independently without human

intervention. However, having access to training data does not relate to autonomy, which is more

about self-learning and decision-making without human control.

(B) Explorability ❌

Explorability refers to the ability to test AI systems interactively to understand their behavior, but it

does not directly relate to accessing training data.

(D) Accessibility ❌

Accessibility refers to the ease with which people can use the system, not the ability to inspect the

training data.

Reference from ISTQB Certified Tester AI Testing Study Guide

Transparency is the ease with which the training data and algorithm used to generate a model can be

understood.

"Transparency: This is considered to be the ease with which the algorithm and training data used to

generate the model can be determined."​

Thus, option C is the correct answer, as transparency involves access to training data, allowing testers

to understand AI decision-making processes.

Thank you for your visit.