A Generative AI Engineer has been asked to design an LLM-based application that accomplishes the following business objective: answer employee HR questions using HR PDF documentation. Which set of high level tasks should the Generative AI Engineer's system perform?

This option describes the standard and most effective architecture for this use case, known as Retrieval-Augmented Generation (RAG). Splitting large documents into smaller, semantically meaningful chunks preserves specific details that would be lost by summarizing or averaging. These chunks are then embedded and stored in a vector database for efficient similarity search. When a user asks a question, the system retrieves only the most relevant chunks, providing focused and accurate context to the LLM. This allows the LLM to generate a precise, grounded answer based on the source material.

A. Averaging embeddings for entire documents is ineffective as it dilutes specific details, making it difficult to match and retrieve precise information for specific queries.
B. Summarizing documents first is a flawed approach because the summarization process can omit the exact details required to answer a specific employee question.
C. This describes a collaborative filtering technique (ALS) used for recommender systems, which is entirely unsuitable for a content-based question-answering task.

1. Databricks Documentation, "What is Retrieval-Augmented Generation (RAG)?": This document outlines the standard RAG workflow. In the section "RAG workflow," it states, "The first step in a RAG application is to chunk the data from a source... Next, you use an embedding model to create vector embeddings... The embeddings are stored in a vector database... The relevant documents are then combined with the original user prompt and sent to an LLM." This directly validates the process described in option D.

2. Databricks Blog, "Building RAG Applications for Your Enterprise with Databricks": This article details the RAG pipeline. The section "The RAG Pipeline" explicitly lists the steps as: 1. Chunking, 2. Embedding, 3. Indexing (in a vector store), 4. Retrieval, and 5. Augmentation and Generation. This aligns perfectly with the correct answer.

3. Lewis, P., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020). Section 2, "Methods," describes the core RAG model, which combines a retriever (to find relevant document passages, i.e., chunks) with a generator (an LLM that uses the retrieved context to produce an answer). This foundational paper establishes the principles outlined in option D. (Available via arXiv:2005.11401)