During a security audit, analysts discover that an attacker repeatedly queried a black-box ML model to infer if specific data points were in the training set. The attacker could determine if an individual’s data was used during training. What threat does this attack represent?

The scenario describes a membership inference attack. The goal of this attack is to determine whether a specific data record was included in the model's training dataset. The attacker achieves this by observing the model's output behavior (e.g., prediction confidence scores) in response to queries. Models often exhibit slightly different behavior for inputs they were trained on versus unseen inputs, and this difference can be exploited to "infer membership," which constitutes a significant privacy breach, violating the confidentiality of the training data.

A. Backdoor in the training set: This involves an attacker inserting a hidden trigger into the model during training, causing it to misbehave in a specific, predictable way only when the trigger is present in the input.
B. Data poisoning: This is a broader attack category where an attacker intentionally injects corrupted or malicious data into the training set to compromise the model's integrity, performance, or fairness. ---

1. Shokri, R., Stronati, M., Song, C., & Shmatikov, V. (2017). Membership Inference Attacks Against Machine Learning Models. 2017 IEEE Symposium on Security and Privacy (SP), pp. 3-18. Section I (Introduction) explicitly defines the attack: "The adversary’s goal is to determine, given a data record and black-box access to a model, whether the record was in the model’s training dataset." This directly matches the question's scenario. DOI: https://doi.org/10.1109/SP.2017.41

2. Papernot, N. (2022). Lecture 10: Privacy in Machine Learning. University of Toronto, Course CSC2547: Current Topics in Machine Learning. Slide 10 defines membership inference as: "Given a model and a data point, determine if the data point was used for training." This aligns perfectly with the described threat.

3. Goldwasser, S., et al. (2022). Planting Undetectable Backdoors in Machine Learning Models. arXiv preprint arXiv:2204.06974. Section 1.1 describes backdoor attacks as causing a model to "behave according to the adversary's objective on inputs with a specific trigger, but behave normally on other inputs." This is distinct from inferring data membership. DOI: https://doi.org/10.48550/arXiv.2204.06974

4. Jagielski, M., et al. (2018). Manipulating Machine Learning: Poisoning Attacks and Countermeasures for Regression Learning. 2018 IEEE Symposium on Security and Privacy (SP), pp. 19-35. Section I (Introduction) defines poisoning attacks as the adversary's ability to "inject a few malicious points into the training data to corrupt the learned model." This focuses on corrupting the model, not identifying its training data. DOI: https://doi.org/10.1109/SP.2018.00009