A Generative Al Engineer is developing a RAG system for their company to perform internal document Q&A for structured HR policies, but the answers…

The core problem is that the retriever is not supplying sufficient context to the Large Language Model (LLM), resulting in incomplete answers. The documents are structured (HR policies), which is a key detail.

(B) Increase the document chunk size: A larger chunk size ensures that more related information and context are contained within a single retrieved block. This directly addresses the issue of "incomplete context" by providing the generation model with a more comprehensive passage to formulate its answer.

(A) Add the section header as a prefix to chunks: For structured documents like HR policies, section headers provide vital context. Prefixing chunks with their corresponding headers (e.g., "Leave Policies: Annual Leave Accrual") creates more semantically rich embeddings. This helps the retriever more accurately identify and fetch the most relevant sections of the document, even if the user's query doesn't use the exact phrasing from the policy text.

C. Split the document by sentence: This would create smaller, more fragmented chunks, likely exacerbating the problem of incomplete context and making it harder for the retriever to gather all necessary information.

D. Use a larger embedding model: The prompt explicitly states that experimenting with different embedding models did not resolve the issue, indicating the problem lies with the data preparation (chunking) strategy, not the model's capability.

E. Fine tune the response generation model: Fine-tuning the generator cannot fix the underlying problem. If the retriever provides incomplete context as input, the generator will still produce an incomplete output, as it cannot generate information it hasn't received.

1. Databricks Blog, "Building RAG Applications for Production with the Databricks Platform" (2024): In the section "Step 2: Data preparation and indexing," the document states, "Chunk size is a key parameter to tune. It has a trade-off: larger chunks can contain more context, but they also add more noise and cost." This supports increasing chunk size (B) to provide more context.

2. Gao, Y., et al. "Retrieval-Augmented Generation for Large Language Models: A Survey." (2023). arXiv:2312.10997: In Section 3.1, "Document Indexing," the paper discusses chunking strategies. It highlights the importance of chunk size and mentions advanced techniques like "semantic chunking," which often involves preserving document structure, such as headers and sections, to maintain context. This principle supports adding section headers as prefixes (A).

3. Databricks Documentation, "Advanced RAG: Chunking and Embedding": This section of the documentation often discusses strategies beyond simple fixed-size chunking. It notes that for structured documents (like PDFs with headers), partitioning strategies that respect the document's hierarchy (e.g., unstructured.io's partitionpdf with infertablestructure=True) lead to more contextually aware chunks. This aligns with adding headers (A) to preserve structural context.