CCAR-F practice question 6 of 10

You are building a structured data extraction system using Claude. The system extracts information from unstructured documents, validates the output…

Full set: 191 questions
You are building a structured data extraction system using Claude. The system extracts information from unstructured documents, validates the output using JavaScript Object Notation (JSON) schemas, and maintains high accuracy. It must handle edge cases gracefully and integrate with downstream systems. Your extraction pipeline processes contracts that frequently include amendments. When a contract contains both original terms and later amendments (e.g., original clause specifies “30-day payment terms” while Amendment 1 changes this to “45 days”), the model inconsistently extracts one value or the other with no indication of which applies. What’s the most effective approach to improve extraction accuracy for documents with amendments?
Answer options
A: Preprocess documents with a classifier that identifies and removes superseded sections before the main extraction step.
B: Redesign the schema so amended fields capture multiple values, each with source location and effective date. Correct answer
C: Add prompt instructions to always extract the most recent amendment value and ignore superseded original terms.
D: Implement post-extraction validation using pattern matching to detect amendments and flag those extractions for manual review.
Correct Answer:
B. Redesign the schema so amended fields capture multiple values, each with source location and effective date.
Explanation

The most robust and effective approach is to modify the data structure (the JSON schema) to accommodate the complexity of the source material. By designing the schema to capture multiple values for a single logical field—each with its context, such as source location and effective date—the system avoids forcing the model to make a complex legalistic judgment about which term is currently active. This method turns an ambiguous inference task into a more straightforward extraction task. It yields a richer, more auditable output that allows downstream systems to apply deterministic business logic to identify the operative term, thus ensuring high accuracy and gracefully handling the specified edge case.

Why Incorrect

A. Preprocessing to remove superseded sections is a complex, high-risk task that could inadvertently delete critical context or introduce errors before the LLM even sees the document.
C. Relying solely on prompt instructions to handle such a critical distinction is brittle. The model may still fail to consistently identify the "most recent" value in complex legal documents.
D. This is a reactive, not a proactive, solution. It adds a manual review step, which undermines the goal of automation, rather than improving the initial extraction accuracy.

References

1. Anthropic Developer Documentation, "JSON Mode": This documentation emphasizes the use of a JSON schema to constrain and guide Claude's output. It states, "By providing a JSON schema, you can constrain Claude's output to only valid JSON objects that conform to the schema..." This supports the principle that modifying the schema (Option B) is the primary and most reliable mechanism for controlling the structure and accuracy of the extracted data.

2. Anthropic Developer Documentation, "Prompting Strategies - Break down complex tasks": This guide recommends decomposing complex tasks into simpler, more manageable steps. Option B effectively does this by changing the task from a single complex step ("find and interpret the currently effective term") to a simpler one ("find all versions of a term and their dates"), leaving the interpretation to a more reliable downstream process.

3. Martin, J., & Odell, J. J. (1998). Object-Oriented Methods: A Foundation. Prentice-Hall. (Chapter 21: "Temporal and Versioned Objects"): While not specific to Claude, this foundational computer science text outlines the principle of temporal data modeling. When data changes over time (as with contract amendments), the best practice is to model the history of values with effective dates, rather than overwriting old values. This ensures auditability and correctness, a principle directly embodied in Option B.

Scroll to Top