CASE STUDY Please use the following answer the next question: A local police department in the United States procured an Al system to monitor and…

The most appropriate human oversight mechanism is to use the AI system's output as one component of a broader, human-led criminal investigation. The police department's proposal to arrest individuals based solely on a 90% score constitutes a "human-out-of-the-loop" system for a high-stakes decision, which is inappropriate and dangerous. This approach risks violating due process and individual rights. By treating the AI's recommendation as a lead or a piece of evidence to be verified and contextualized by a human officer, the department maintains meaningful human control. This ensures that the final decision to arrest is based on a holistic investigation, mitigating risks of automation bias and potential errors from the AI system.

A. Explain to the accused how the Al system works. This relates to the principle of transparency, which is important, but it is a post-decision action, not an oversight mechanism for the decision-making process itself.
B. Confirm the Al recommendation prior to sentencing. This is far too late in the legal process. An arrest is a significant deprivation of liberty, and oversight must occur before this action is taken, not after a trial and conviction.
C. Ensure an accused is given notice that the Al system was used. This is a procedural notification requirement, similar to option A. It informs the individual but does not provide a check or balance on the police's decision to arrest.

1. NIST. (2023). AI Risk Management Framework (AI RMF 1.0). National Institute of Standards and Technology.

Reference: Page 25

Section 4.3.3

"Human in the loop

on the loop

and out of the loop." The document states

"For high-risk AI systems

the default should be human-in-the-loop... Sole reliance on AI systems for decision-making is not recommended." The police department's proposal to arrest directly based on an AI score is a "human-out-of-the-loop" approach

which NIST advises against for high-risk applications like law enforcement. Option D correctly implements a "human-in-the-loop" model.

2. OECD. (2019). Recommendation of the Council on Artificial Intelligence. OECD/LEGAL/0449.

Reference: Section 1.5

"Accountability." The principle states that "Actors should be accountable for the proper functioning of AI systems... and for respecting the above principles." Automating arrests based on an AI score removes direct human accountability for the arrest decision

violating this principle. Keeping an officer in charge of the investigation (Option D) maintains a clear line of accountability.

3. Elish

M. C. (2016). The Limits of Responsible Innovation. Daedalus

145(3)

53-65. (MIT Press Journals).

Reference: Page 59. The paper discusses the "moral crumple zone

" where human operators are held responsible for failures of automated systems they cannot fully control or understand. Option D avoids this by ensuring the human officer is not just a scapegoat for an AI decision but is an active

responsible agent in the investigation

using the AI as a tool rather than a directive.

DOI: https://doi.org/10.1162/DAEDa00402

4. Stanford University Human-Centered Artificial Intelligence (HAI). (2021). AI and the Rule of Law: A Call to Action.

Reference: Page 6

"Recommendations for Government Use of AI." The report recommends that "AI systems should be used to assist

not replace

human decision-makers in high-stakes domains." It explicitly warns against AI systems making final determinations in areas like criminal justice

advocating that they serve as inputs to a human-led process

which is precisely what Option D describes.