NCP-AAI practice question 16 of 25
A company wants to compare two prompting strategies for an internal writing assistant. Which method is BEST for measuring which version performs better?
Full set: 118 questionsA/B testing is the gold standard for empirically comparing two versions of a system component, such as a prompt. This method involves randomly assigning users or requests to either the "A" version (control prompt) or the "B" version (challenger prompt). By measuring performance against predefined, objective quality metrics (e.g., relevance, helpfulness score, task completion rate), the company can obtain statistically significant data to determine which prompt strategy is superior. This data-driven approach removes subjectivity and ensures the chosen prompt genuinely improves the user experience or assistant's performance.
B. Choosing the longer prompt automatically is flawed, as prompt length has no guaranteed correlation with performance or quality.
C. Using whichever prompt was written first is an arbitrary and unscientific method that ignores the actual performance of the prompts.
D. Avoiding all evaluation to save time is counterproductive, as it prevents any form of quality improvement or validation of the assistant's effectiveness.
1. Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press. Chapter 1: Introduction to Online Controlled Experiments. This book is a foundational text on A/B testing, outlining the principles that apply directly to comparing LLM prompts.
2. Stanford University. (2023). "CS224N: NLP with Deep Learning | Winter 2023." Lecture on Prompting and In-Context Learning. Course materials often discuss evaluation methodologies, where A/B testing is presented as a key method for production systems.
3. Google AI. (2023). "Testing and Evaluation." Google People + AI Guidebook. This guide emphasizes iterative testing with users and metrics, with A/B testing being a primary quantitative method. Section: Measure > Quantitative methods.
Keep practicing
Take the free NCP-AAI practice test
Ten exam-style questions with answers and explanations, plus the exam facts and study guides.
More questions
Other NCP-AAI practice questions
- Question 1Which two design goals are MOST important for human-in-the-loop agent systems in regulate…
- Question 2A company plans to launch a multi-agent system that must serve thousands of users simulta…
- Question 3Which two orchestration methods are MOST suitable for implementing complex agentic workfl…
- Question 4You are designing an AI agent for summarizing medical documents that include images and t…
- Question 5A company is deploying a multi-agent AI system to handle large-scale customer interaction…
- Question 6A multi-agent workflow is failing because one agent completes tasks before dependent stat…
