NCP-AAI practice question 21 of 25

When analyzing an agent's failure to complete multi-step financial analysis tasks, which evaluation approach best identifies prompt engineering improvements needed for reliable task decomposition and execution?

Full set: 118 questions
Answer options
A: Implement systematic prompt testing with chain-of-thought reasoning templates, step-by- stepdecomposition analysis, and success rate tracking across tasks of varying complexity. Correct answer
B: Focus primarily on response speed optimization as a primary focus over reasoning quality, stepcompletion accuracy, and prompt clarity for complex analytical requirements.
C: Test only final output accuracy as this will automatically include intermediate reasoning steps,decomposition quality, and prompt structure effectiveness for complex workflows.
D: Rely on generic prompt templates which are by default already optimized for general use, instead oftailoring them to financial terminology, calculation needs, or specialized multi-step analysis patterns.
Correct Answer:
A. Implement systematic prompt testing with chain-of-thought reasoning templates, step-by- stepdecomposition analysis, and success rate tracking across tasks of varying complexity.
Explanation

For complex, multi-step tasks, failure often occurs in the intermediate reasoning or planning stages. A systematic approach is required to diagnose this. Testing with chain-of-thought (CoT) templates explicitly evaluates the model's reasoning process. Analyzing the step-by-step decomposition reveals where the agent's plan breaks down. Tracking success rates across different complexities helps identify the boundaries of the agent's capabilities. This structured, analytical approach directly targets the prompt's influence on task decomposition and execution, providing actionable insights for improvement, which is far superior to only checking the final answer or using generic prompts.

Why Incorrect

B. Optimizing for speed over reasoning quality is counterproductive for financial analysis, where accuracy and correctness are the primary requirements.
C. Testing only the final output is a black-box method that hides which intermediate step failed, making it impossible to debug the decomposition logic in the prompt.
D. Generic templates are unlikely to work for a specialized domain like finance, which requires specific terminology, formats, and reasoning patterns for reliable performance.

References

1. Wei, J., et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv preprint arXiv:2201.11903. This foundational paper demonstrates that prompting LLMs to generate intermediate reasoning steps (a chain of thought) improves performance on complex tasks, validating the focus on step-by-step decomposition analysis. (See Tables 3 and 4 for results).

2. Yao, S., et al. (2023). Tree of Thoughts: Deliberate Problem Solving with Large Language Models. arXiv preprint arXiv:2305.10601. This work extends CoT by exploring multiple reasoning paths, reinforcing the principle that evaluating and improving the intermediate thought process is key to solving complex tasks. (Section 2).

3. NVIDIA Developer Blog. (2023). A Practical Guide to Prompt Engineering with Llama 2. Discusses techniques like few-shot prompting and providing structured examples, which are core components of systematic prompt testing to improve task execution for specialized domains.

Scroll to Top