AIGP practice question 17 of 25

CASE STUDY Please use the following answer the next question: A local police department in the United States procured an Al system to monitor and…

Choose your answer, then check it against the explanation.

CASE STUDY Please use the following answer the next question: A local police department in the United States procured an Al system to monitor and analyze social media feeds, online marketplaces and other sources of public information to detect evidence of illegal activities (e.g., sale of drugs or stolen goods). The Al system works by surveilling the public sites in order to identify individuals that are likely to have committed a crime. It cross-references the individuals against data maintained by law enforcement and then assigns a percentage score of the likelihood of criminal activity based on certain factors like previous criminal history, location, time, race and gender. The police department retained a third-party consultant assist in the procurement process, specifically to evaluate two finalists. Each of the vendors provided information about their system's accuracy rates, the diversity of their training data and how their system works. The consultant determined that the first vendor’s system has a higher accuracy rate and based on this information, recommended this vendor to the police department. The police department chose the first vendor and implemented its Al system. As part of the implementation, the department and consultant created a usage policy for the system, which includes training police officers on how the system works and how to incorporate it into their investigation process. The police department has now been using the Al system for a year. An internal review has found that every time the system scored a likelihood of criminal activity at or above 90%, the police investigation subsequently confirmed that the individual had, in fact, committed a crime. Based on these results, the police department wants to forego investigations for cases where the Al system gives a score of at least 90% and proceed directly with an arrest. The best human oversight mechanism for the police department to implement is that a police officer should?

Answer options

Correct Answer:

Explanation

The most appropriate human oversight mechanism is to use the AI system's output as one component of a broader, human-led criminal investigation. The police department's proposal to arrest individuals based solely on a 90% score constitutes a "human-out-of-the-loop" system for a high-stakes decision, which is inappropriate and dangerous. This approach risks violating due process and individual rights. By treating the AI's recommendation as a lead or a piece of evidence to be verified and contextualized by a human officer, the department maintains meaningful human control. This ensures that the final decision to arrest is based on a holistic investigation, mitigating risks of automation bias and potential errors from the AI system.

Why Incorrect

A. Explain to the accused how the Al system works.

This relates to the principle of transparency, which is important, but it is a post-decision action, not an oversight mechanism for the decision-making process itself.

B. Confirm the Al recommendation prior to sentencing.

This is far too late in the legal process. An arrest is a significant deprivation of liberty, and oversight must occur before this action is taken, not after a trial and conviction.

C. Ensure an accused is given notice that the Al system was used.

This is a procedural notification requirement, similar to option A. It informs the individual but does not provide a check or balance on the police's decision to arrest.

References

1. NIST. (2023). AI Risk Management Framework (AI RMF 1.0). National Institute of Standards and Technology.

Reference: Page 25

Section 4.3.3

"Human in the loop

on the loop

and out of the loop." The document states

"For high-risk AI systems

the default should be human-in-the-loop... Sole reliance on AI systems for decision-making is not recommended." The police department's proposal to arrest directly based on an AI score is a "human-out-of-the-loop" approach

which NIST advises against for high-risk applications like law enforcement. Option D correctly implements a "human-in-the-loop" model.

2. OECD. (2019). Recommendation of the Council on Artificial Intelligence. OECD/LEGAL/0449.

Reference: Section 1.5

"Accountability." The principle states that "Actors should be accountable for the proper functioning of AI systems... and for respecting the above principles." Automating arrests based on an AI score removes direct human accountability for the arrest decision

violating this principle. Keeping an officer in charge of the investigation (Option D) maintains a clear line of accountability.

3. Elish

M. C. (2016). The Limits of Responsible Innovation. Daedalus

145(3)

53-65. (MIT Press Journals).

Reference: Page 59. The paper discusses the "moral crumple zone

" where human operators are held responsible for failures of automated systems they cannot fully control or understand. Option D avoids this by ensuring the human officer is not just a scapegoat for an AI decision but is an active

responsible agent in the investigation

using the AI as a tool rather than a directive.

DOI: https://doi.org/10.1162/DAEDa00402

4. Stanford University Human-Centered Artificial Intelligence (HAI). (2021). AI and the Rule of Law: A Call to Action.

Reference: Page 6

"Recommendations for Government Use of AI." The report recommends that "AI systems should be used to assist

not replace

human decision-makers in high-stakes domains." It explicitly warns against AI systems making final determinations in areas like criminal justice

advocating that they serve as inputs to a human-led process

which is precisely what Option D describes.

Question 17 of 25

Keep practicing

Take the free AIGP practice test

Ten exam-style questions with answers and explanations, plus the exam facts and study guides.

Start the free practice test