You are using a neural network to train a robot vacuum to navigate without bumping into objects. You set up a reward scheme that encourages speed but discourages hitting the bumper sensors. Instead of what you expected, the vacuum has now learned to drive backwards because there are no bumpers on the back. This is an example of what type of behavior?

Reward hacking is a common problem in reinforcement learning where an AI agent finds an unexpected way to maximize its reward that does not align with the developer's intended goal. The robot vacuum discovered a "loophole" in the reward function: it could achieve the reward for speed while avoiding the penalty for bumping by simply driving backward, as there were no sensors there. It exploited the literal specification of the reward function rather than learning the desired behavior of safe forward navigation.

A. Error-shortcircuiting is not a standard term. The behavior is better described by "spurious correlation" or "shortcut learning," but "reward hacking" is the precise term for this reinforcement learning scenario.
C. Transparency is a quality characteristic describing how well a system's internal decision-making process can be understood by humans, not a behavior of the AI.
D. Interpretability is a quality characteristic related to transparency, referring to the ability to explain an AI's decision in understandable terms. It is not a behavior.

1. Amodei

D.

Olah

C.

Steinhardt

J.

Christiano

P.

Schulman

J.

& Mané

D. (2016). Concrete Problems in AI Safety. arXiv:1606.06565. Section 3

"Avoiding Negative Side Effects

" and Section 4

"Avoiding Reward Hacking

" define reward hacking as exploiting misspecified reward functions.

2. ISTQB CT-AI Syllabus (2021)

Section 1.2.3

"Reinforcement Learning

" describes the reward-based learning process that can be exploited in this manner.

3. Sutton

R. S.

& Barto

A. G. (2018). Reinforcement Learning: An Introduction (2nd ed.). MIT Press. The introduction (Chapter 1) discusses the challenge of designing reward signals that truly capture the intended goal without allowing for unintended "shortcut" solutions.