Which of the following is a method in which simulation of various attack scenarios are applied to analyze the model's behavior under those conditions.

Adversarial testing is a security evaluation method specifically designed for AI/ML models. It involves systematically generating and applying "adversarial examples"—inputs intentionally crafted to deceive or disrupt the model. This process simulates various attack scenarios to analyze the model's behavior, identify vulnerabilities, and measure its robustness against malicious manipulations. The core purpose is to understand how the model performs under conditions it was not explicitly trained on, particularly those mimicking hostile actions, which directly aligns with the question's description.

A. input sanitation: This is a defensive measure to clean or validate data before it is processed by the model, not a method for simulating attacks to test behavior.

B. Model firewall: This is a security control that inspects and filters requests to a model, acting as a protective barrier rather than a testing methodology.

C. Prompt injections: This is a specific type of attack on large language models, representing one scenario within the broader practice of adversarial testing.

1. National Institute of Standards and Technology (NIST). (2023). Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (NIST AI 100-2). U.S. Department of Commerce. Section 2.1

"Adversarial Machine Learning

" describes the process of using crafted inputs to cause a model to misbehave

which is the foundation of adversarial testing.

2. Goodfellow

I. J.

Shlens

J.

& Szegedy

C. (2014). Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572. Section 1

Paragraph 1

introduces the concept of adversarial examples as a tool to analyze model properties and blind spots. https://doi.org/10.48550/arXiv.1412.6572

3. MIT OpenCourseWare. (2020). 6.S191: Introduction to Deep Learning. Lecture on Deep Learning Security and Privacy. The lecture materials discuss adversarial testing as the primary method for evaluating a model's resilience by simulating attacks with adversarial examples.