NCP-AAI practice question 13 of 25

You are evaluating your RAG pipeline. You notice that the LLM-as-a-Judge consistently assigns high similarity scores to responses that contain irrelevant information. What should you investigate as the most likely potential cause with the least development effort?

Full set: 118 questions
Answer options
A: The temperature setting used by the LLM during response generation.
B: The size of the knowledge base used to power the RAG pipeline.
C: The quality of the synthetic questions used for evaluation.
D: The prompt used to instruct the LLM-as-a-Judge to assess the response. Correct answer
Correct Answer:
D. The prompt used to instruct the LLM-as-a-Judge to assess the response.
Explanation

The LLM-as-a-Judge's behavior is dictated entirely by its instructions. If it consistently gives high scores to responses with irrelevant information, the most likely cause is a flaw in the prompt used to guide its evaluation. The prompt may lack a clear, explicit instruction to penalize extraneous or non-contextual content. Modifying the prompt to be more specific about the evaluation criteria (e.g., "The response must only contain information directly supported by the provided context") is the most direct and lowest-effort way to correct this behavior.

Why Incorrect

A. The temperature setting controls the randomness of a model's generated output, not its behavior when performing a structured evaluation task like scoring.
B. The knowledge base size affects the RAG system's retrieval quality, but it does not directly influence how the separate judge module evaluates a given response.
C. While poor synthetic questions can lead to poor responses, the judge's failure is in its assessment criteria, not the quality of the item it is assessing.

References

1. NVIDIA NeMo Framework Documentation. (Accessed 2024). NeMo Evaluator. The documentation for evaluation tools emphasizes the importance of well-defined prompts and metrics to guide the LLM judge in assessing specific qualities like faithfulness, relevance, and conciseness. A failure in judging relevance points directly to the prompt's definition of that metric.

2. Zheng, L., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685 [cs.CL]. Section 3.1, "Prompting," discusses how the performance of an LLM-as-a-Judge is highly sensitive to the prompt format, including the scoring criteria and instructions provided. This academic paper validates that the prompt is the primary control mechanism.

Scroll to Top