NCP-AAI practice question 13 of 25
You are evaluating your RAG pipeline. You notice that the LLM-as-a-Judge consistently assigns high similarity scores to responses that contain irrelevant information. What should you investigate as the most likely potential cause with the least development effort?
Full set: 118 questionsThe LLM-as-a-Judge's behavior is dictated entirely by its instructions. If it consistently gives high scores to responses with irrelevant information, the most likely cause is a flaw in the prompt used to guide its evaluation. The prompt may lack a clear, explicit instruction to penalize extraneous or non-contextual content. Modifying the prompt to be more specific about the evaluation criteria (e.g., "The response must only contain information directly supported by the provided context") is the most direct and lowest-effort way to correct this behavior.
A. The temperature setting controls the randomness of a model's generated output, not its behavior when performing a structured evaluation task like scoring.
B. The knowledge base size affects the RAG system's retrieval quality, but it does not directly influence how the separate judge module evaluates a given response.
C. While poor synthetic questions can lead to poor responses, the judge's failure is in its assessment criteria, not the quality of the item it is assessing.
1. NVIDIA NeMo Framework Documentation. (Accessed 2024). NeMo Evaluator. The documentation for evaluation tools emphasizes the importance of well-defined prompts and metrics to guide the LLM judge in assessing specific qualities like faithfulness, relevance, and conciseness. A failure in judging relevance points directly to the prompt's definition of that metric.
2. Zheng, L., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685 [cs.CL]. Section 3.1, "Prompting," discusses how the performance of an LLM-as-a-Judge is highly sensitive to the prompt format, including the scoring criteria and instructions provided. This academic paper validates that the prompt is the primary control mechanism.
Keep practicing
Take the free NCP-AAI practice test
Ten exam-style questions with answers and explanations, plus the exam facts and study guides.
More questions
Other NCP-AAI practice questions
- Question 1Which two design goals are MOST important for human-in-the-loop agent systems in regulate…
- Question 2A company plans to launch a multi-agent system that must serve thousands of users simulta…
- Question 3Which two orchestration methods are MOST suitable for implementing complex agentic workfl…
- Question 4You are designing an AI agent for summarizing medical documents that include images and t…
- Question 5A company is deploying a multi-agent AI system to handle large-scale customer interaction…
- Question 6A multi-agent workflow is failing because one agent completes tasks before dependent stat…
