AIP-C01 practice question 9 of 15

A medical company uses Amazon Bedrock to power a clinical documentation summarization system. The system produces inconsistent summaries when…

A medical company uses Amazon Bedrock to power a clinical documentation summarization system. The system produces inconsistent summaries when handling complex clinical documents. The system performed well on simple clinical documents. The company needs a solution that diagnoses inconsistencies, compares prompt performance against established metrics, and maintains historical records of prompt versions. Which solution will meet these requirements?

Answer options

A: Create multiple prompt variants by using Prompt management in Amazon Bedrock. Manually test the prompts with simple clinical documents. Deploy the highest performing version by using the Amazon Bedrock console.

B: Implement version control for prompts in a code repository with a test suite that contains complex clinical documents and quantifiable evaluation metrics. Use an automated testing framework to compare prompt versions and document performance patterns. Correct answer

C: Deploy each new prompt version to separate Amazon Bedrock API endpoints. Split production traffic between the endpoints. Configure Amazon CloudWatch to capture response metrics and user feedback for automatic version selection.

D: Create a custom prompt evaluation flow in Amazon Bedrock Flows that applies the same clinical document inputs to different prompt variants. Use Amazon Comprehend Medical to analyze and score the factual accuracy of each version.

Correct Answer:

B. Implement version control for prompts in a code repository with a test suite that contains complex clinical documents and quantifiable evaluation metrics. Use an automated testing framework to compare prompt versions and document performance patterns.

Explanation

The most robust and systematic solution is to treat prompt engineering as a software development lifecycle process. Implementing version control (e.g., Git) provides a historical record. A dedicated test suite with complex clinical documents ensures the system is evaluated against the problematic cases, not just simple ones. Using quantifiable metrics (e.g., ROUGE, BLEU, or domain-specific accuracy scores) and an automated framework allows for objective, repeatable comparison between prompt versions. This directly addresses the need to diagnose inconsistencies, compare performance against metrics, and maintain version history.

Why Incorrect

A. Manually testing with simple documents will not solve the problem, which occurs with complex documents. This approach lacks rigor and automated, quantifiable metrics.
C. A/B testing in production is for validating user experience, not for systematically diagnosing and fixing underlying prompt performance issues against established, objective metrics.
D. While a possible approach, it is overly complex for the core task. A standard MLOps/PromptOps framework (as in B) is a more direct and standard way to manage, test, and version prompts.

References

1. AWS Machine Learning Blog. (2023, November 17). Prompt engineering for foundation models on Amazon Bedrock. AWS. Retrieved from https://aws.amazon.com/blogs/machine-learning/prompt-engineering-for-foundation-models-on-amazon-bedrock/ (Discusses the iterative nature of prompt engineering, implying a need for testing and versioning).

2. Cobbe, K., et al. (2021). Training Verifiers to Solve Math Word Problems. arXiv:2110.14168 [cs.LG]. https://doi.org/10.48550/arXiv.2110.14168 (Section 4, "Results," demonstrates the use of test sets and metrics to evaluate model/prompt performance, a standard academic practice).

3. Stanford University. (n.d.). CS224N: NLP with Deep Learning | Winter 2021. Lecture 15. Retrieved from https://web.stanford.edu/class/cs224n/ (Courseware often covers model evaluation, emphasizing the need for robust test sets and metrics to ensure performance).

Keep practicing

Take the free AIP-C01 practice test

Ten exam-style questions with answers and explanations, plus the exam facts and study guides.

Start the free practice test