CCAR-P practice question 2 of 10

You are an architect supporting the iteration phase of a deployed Claude-based system. Which activity most directly fits this phase?

Full set: 142 questions
Answer options
A: Rebuild the entire system architecture from scratch at the start of every iteration cycle regardless of what production telemetry, evaluations, and stakeholder feedback indicate is needed.
B: Stop measuring production outcomes once the deployment has successfully launched, treating the go-live milestone as the end of the evaluation and iteration cycle.
C: Discard the evaluation framework and reference set once the deployment is in production, accepting that future iterations will have no structured basis for measuring the impact of changes.
D: Review production telemetry, sampled output evaluations, and stakeholder feedback to identify the highest-impact change for the next cycle, then plan the change against the evaluation framework. Correct answer
Correct Answer:
D. Review production telemetry, sampled output evaluations, and stakeholder feedback to identify the highest-impact change for the next cycle, then plan the change against the evaluation framework.
Explanation

The iteration phase of an LLM deployment relies on continuous improvement driven by real-world data. Reviewing production telemetry, sampled evaluations, and stakeholder feedback allows architects to identify high-impact areas for refinement. Planning these changes against an established evaluation framework ensures that updates improve performance without introducing regressions. This data-driven approach is a core tenet of the LLM lifecycle, ensuring the system adapts to actual user behavior and edge cases discovered post-launch.

Why Incorrect

A. Rebuilding from scratch ignores valuable production data and is highly inefficient for iterative improvement.
B. Stopping measurement after launch prevents any data-driven iteration or monitoring of model drift.
C. Discarding the evaluation framework removes the ability to objectively measure the impact of future changes or catch regressions.

References

1. Anthropic API Documentation

"Evaluating and iterating on prompts" and "Productionizing your application" sections.

2. Stanford University CS324 (Large Language Models)

Section: "Deployment and Monitoring".

Scroll to Top