To confirm the fairness of AI model decisions, the BEST way to collect reliable evidence during an AI audit is by:

The most reliable and objective method for an auditor to confirm the fairness of an AI model's decisions is by testing it with a curated sample dataset. This approach, a form of substantive testing, allows the auditor to use a controlled, representative, and specifically designed dataset to systematically evaluate the model's outputs for biases against protected attributes or subgroups. The results are quantifiable using established fairness metrics, providing direct, empirical, and repeatable evidence of the model's behavior. This is superior to indirect, subjective, or uncontrolled methods.

A. Analyzing system metadata: This provides contextual information about the model's lineage and training but does not directly test the fairness of its outputs on new data.

C. Interviewing developers: This is a form of inquiry that yields subjective information about intent and process, not objective, empirical evidence of the model's actual performance.

D. Observing the system’s interactions with end users: This method is uncontrolled and anecdotal; the observed data may not be sufficient or structured to systematically quantify fairness.

1. National Institute of Standards and Technology (NIST). (2023). AI Risk Management Framework (AI RMF 1.0). NIST AI 100-1. In the "Measure" function

section 4.4

the framework emphasizes the use of "technical tests and evaluations" with "methodologically sound metrics" to assess AI system trustworthiness

including fairness and bias. Testing with a curated dataset is a primary example of such a technical test.

2. The Institute of Internal Auditors (IIA). (2020). Global Technology Audit Guide (GTAG): Auditing Artificial Intelligence. This guide highlights that a key audit step is to "Test for biased outcomes

" which involves "running test scenarios with different data sets to determine whether the AI is producing biased outcomes" (p. 18).

3. Koene

A.

et al. (2021). A Governance Framework for Algorithm Accountability. In M. D. B. B. C. D. L. F. G. H. J. K. L. M. N. O. P. R. S. T. U. V. W. Z. D. Leslie (Ed.)

The Oxford Handbook of AI Ethics. Oxford University Press. Chapter 18 discusses the necessity of "testing and validation" using specific datasets to audit algorithms for fairness and other ethical concerns

distinguishing it from procedural checks.

4. Mittelstadt

B. (2019). Principles alone cannot guarantee ethical AI. Nature Machine Intelligence

1

501–507. https://doi.org/10.1038/s42256-019-0114-4. The paper argues for technical and institutional solutions beyond principles

where "auditing" through empirical testing is a key mechanism to verify claims of fairness and non-discrimination in practice.