GENERATIVE-AI-ENGINEER-ASSOCIATE Practice Questions

The core problem is that the retriever is not supplying sufficient context to the Large Language Model (LLM), resulting in incomplete answers. The documents are structured (HR policies), which is a key detail.

(B) Increase the document chunk size: A larger chunk size ensures that more related information and context are contained within a single retrieved block. This directly addresses the issue of "incomplete context" by providing the generation model with a more comprehensive passage to formulate its answer.

(A) Add the section header as a prefix to chunks: For structured documents like HR policies, section headers provide vital context. Prefixing chunks with their corresponding headers (e.g., "Leave Policies: Annual Leave Accrual") creates more semantically rich embeddings. This helps the retriever more accurately identify and fetch the most relevant sections of the document, even if the user's query doesn't use the exact phrasing from the policy text.

C. Split the document by sentence: This would create smaller, more fragmented chunks, likely exacerbating the problem of incomplete context and making it harder for the retriever to gather all necessary information.

D. Use a larger embedding model: The prompt explicitly states that experimenting with different embedding models did not resolve the issue, indicating the problem lies with the data preparation (chunking) strategy, not the model's capability.

E. Fine tune the response generation model: Fine-tuning the generator cannot fix the underlying problem. If the retriever provides incomplete context as input, the generator will still produce an incomplete output, as it cannot generate information it hasn't received.

1. Databricks Blog, "Building RAG Applications for Production with the Databricks Platform" (2024): In the section "Step 2: Data preparation and indexing," the document states, "Chunk size is a key parameter to tune. It has a trade-off: larger chunks can contain more context, but they also add more noise and cost." This supports increasing chunk size (B) to provide more context.

2. Gao, Y., et al. "Retrieval-Augmented Generation for Large Language Models: A Survey." (2023). arXiv:2312.10997: In Section 3.1, "Document Indexing," the paper discusses chunking strategies. It highlights the importance of chunk size and mentions advanced techniques like "semantic chunking," which often involves preserving document structure, such as headers and sections, to maintain context. This principle supports adding section headers as prefixes (A).

3. Databricks Documentation, "Advanced RAG: Chunking and Embedding": This section of the documentation often discusses strategies beyond simple fixed-size chunking. It notes that for structured documents (like PDFs with headers), partitioning strategies that respect the document's hierarchy (e.g., unstructured.io's partitionpdf with infertablestructure=True) lead to more contextually aware chunks. This aligns with adding headers (A) to preserve structural context.

After initializing the VectorSearchClient, the next logical step to set up a new vector store is to create an index. The createdeltasyncindex() function is used to create a Vector Search index that automatically syncs with a source Delta Table. This is a common and robust pattern for production use cases where the source data may change over time. The function call would include parameters specifying the endpoint name, the source Delta table, the primary key, and the columns to be indexed, including the one containing the embeddings.

A. vsc.getindex()

This function retrieves an existing index. The current task is to set up a new vector store, which requires creating an index first.

C. vsc.createdirectaccessindex()

While this also creates an index, it is designed for static datasets. The Delta Sync index is generally the more common choice for its ability to automatically update.

D. vsc.similaritysearch()

This function is used to query an index for similar vectors. An index must be created and populated before any search can be performed.

1. Databricks Official Documentation, "Create a vector search index using the Python SDK": This document outlines the process for creating a vector search index. It explicitly shows that after initializing VectorSearchClient(), the next step is to call either createdeltasyncindex() or createdirectaccessindex(). The Delta Sync method is presented as a primary option.

Section/Paragraph: See the code examples under the headings "Create a Delta Sync index" and "Prerequisites". The workflow clearly shows vsc = VectorSearchClient() followed by vsc.createdeltasyncindex(...).

2. Databricks Official Documentation, "Vector Search Python SDK reference": The API reference for the VectorSearchClient class lists createdeltasyncindex and createdirectaccessindex as the methods for index creation. It also lists getindex for retrieval and similaritysearch on the index object itself (not the client) for querying, confirming the operational sequence.

Section/Paragraph: Review the method definitions for the databricks.vectorsearch.client.VectorSearchClient class.

The chain.run() method is a legacy convenience function in LangChain designed for chains with a single input variable. The modern, standard interface for executing any "Runnable" object in LangChain, including LLMChain, is the invoke() method. This method expects a dictionary where the keys match the inputvariables specified in the PromptTemplate (in this case, "product"). Using chain.invoke({"product": "colorful socks"}) aligns with the current LangChain Expression Language (LCEL) standard, ensuring compatibility and predictable behavior. The original code fails because run() is either deprecated or being used in a context where the more explicit invoke() is required.

A: The run() method expects a single positional string as input for a single-variable chain, not a keyword argument.

B: The run() method is a simplified wrapper that accepts a string directly, not a dictionary. Passing a dictionary is the correct syntax for invoke().

D: The predict() method, another legacy wrapper, expects keyword arguments (e.g., chain.predict(product="...")), not a dictionary.

1. Databricks Documentation, "Develop applications with LangChain on Databricks": The official guide for using LangChain on Databricks consistently uses the .invoke() method with a dictionary for chain execution. For example, in the section "Build a simple LLM application," the demonstrated code is chain.invoke({"question": "What is the capital of France?"}). This establishes .invoke() as the standard practice on the platform. (See: Databricks Docs > AI and Machine Learning > Generative AI > Tools and Libraries > LangChain).

2. LangChain Official Documentation, "Get started with chains": The core LangChain documentation introduces the invoke method as the primary way to run chains. It states, "We can now call the chain with a dictionary, which is the expected input type." The examples exclusively use the chain.invoke({"inputvariable": "value"}) pattern. (See: LangChain Python Docs > docs > getstarted > chains).

3. LangChain API Reference, langchaincore.runnables.Runnable: LLMChain inherits from the Runnable class. The Runnable interface specifies invoke(input, config) as the standard synchronous execution method, which takes the input (typically a dictionary) and an optional configuration object. The run method is not part of this primary interface. (See: LangChain API Reference > langchaincore.runnables).

The primary requirement is a foundation LLM with a large context window. Among the options provided, DBRX has the largest context window by a significant margin. DBRX is a state-of-the-art, open-source, mixture-of-experts (MoE) model from Databricks that was trained with a context length of 32,768 (32k) tokens. This extensive context window makes it highly suitable for applications that need to process and generate text based on long documents, detailed histories, or complex instructions, directly addressing the engineer's need.

A. DistilBERT: This is a smaller, distilled version of BERT with a maximum context window of only 512 tokens. It is also primarily an encoder model, not a generative one.

B. MPT-30B: The base MPT-30B model has a 2,048 (2k) token context window, with some fine-tuned variants extending to 8,192 (8k) tokens, which is still considerably smaller than DBRX.

C. Llama2-70B: The Llama 2 family of models was trained with a context length of 4,096 (4k) tokens. While larger than DistilBERT, this is only one-eighth the size of DBRX's context window.

1. DBRX: Databricks. (2024, March 27). Introducing DBRX: A New State-of-the-Art Open LLM. Databricks Blog. "DBRX has a 32k token context window". Link

2. Llama2-70B: Touvron, H., et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. Meta AI. Section 2.1, "Pretraining," states, "We adopt most of the pretraining setting and model architecture from Llama 1... we increase the context length to 4096." Link

3. MPT-30B: MosaicML. (2023, June 22). Introducing MPT-30B: A New Standard for Open-Source, Commercially Usable LLMs. Databricks Blog. "MPT-30B is a decoder-style transformer with 30B parameters. It was pretrained on 1T tokens of English text and code. The base model has a context window of 2k tokens." Link

4. DistilBERT: Sanh, V., et al. (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. Hugging Face. The official model card for distilbert-base-uncased specifies, "The maximum input sequence length is 512." Link

The most effective method to mitigate inflammatory outputs in a Retrieval-Augmented Generation (RAG) system is to address the root cause: the content within the knowledge base. The RAG process retrieves information from this upstream data to provide context for the language model's response. If the source documents contain offensive or inflammatory material, the system is likely to retrieve and incorporate it into its output. Proactively curating the data, including manual review and cleaning, ensures that the foundational information used by the RAG system is free from undesirable content. This "Garbage In, Garbage Out" principle is fundamental to building safe and reliable AI systems.

A. Increasing update frequency addresses data staleness, not content quality. It could potentially introduce more unvetted, inflammatory data.

B. Informing the user is a disclaimer, not a technical solution. It manages expectations but does not fix the underlying problem.

C. Restricting access to the source data is an access control measure that does not alter the content the RAG system uses.

1. Databricks Blog, "A Guide to Building High-Quality RAG Applications" (2024). In the section "Step 1: Data Preparation," the guide states, "The first step to building a high-quality RAG application is preparing your unstructured data... This includes cleaning the data to remove irrelevant information (e.g., HTML tags, boilerplate text), checking for and handling PII, and ensuring the data is accurate and up-to-date." This emphasizes that data quality control is the foundational step.

2. Gao, Y., et al. (2024). "Retrieval-Augmented Generation: A Survey". arXiv:2312.10997. In Section 5.3, "Trustworthiness," the paper discusses how the quality of the retrieval corpus directly impacts the final generation's safety. It notes that issues like toxicity and bias are often inherited from the retrieved documents, highlighting the importance of a clean, high-quality knowledge source.

3. Databricks Documentation, "What is Retrieval-Augmented Generation (RAG)?". The documentation explains that RAG "grounds the LLM on a specific set of external knowledge sources." This core principle implies that the content and quality of those external sources directly dictate the boundaries and nature of the model's potential responses. Curating these sources is therefore the most direct way to control output quality.

This option describes the standard and most effective architecture for this use case, known as Retrieval-Augmented Generation (RAG). Splitting large documents into smaller, semantically meaningful chunks preserves specific details that would be lost by summarizing or averaging. These chunks are then embedded and stored in a vector database for efficient similarity search. When a user asks a question, the system retrieves only the most relevant chunks, providing focused and accurate context to the LLM. This allows the LLM to generate a precise, grounded answer based on the source material.

A. Averaging embeddings for entire documents is ineffective as it dilutes specific details, making it difficult to match and retrieve precise information for specific queries.

B. Summarizing documents first is a flawed approach because the summarization process can omit the exact details required to answer a specific employee question.

C. This describes a collaborative filtering technique (ALS) used for recommender systems, which is entirely unsuitable for a content-based question-answering task.

1. Databricks Documentation, "What is Retrieval-Augmented Generation (RAG)?": This document outlines the standard RAG workflow. In the section "RAG workflow," it states, "The first step in a RAG application is to chunk the data from a source... Next, you use an embedding model to create vector embeddings... The embeddings are stored in a vector database... The relevant documents are then combined with the original user prompt and sent to an LLM." This directly validates the process described in option D.

2. Databricks Blog, "Building RAG Applications for Your Enterprise with Databricks": This article details the RAG pipeline. The section "The RAG Pipeline" explicitly lists the steps as: 1. Chunking, 2. Embedding, 3. Indexing (in a vector store), 4. Retrieval, and 5. Augmentation and Generation. This aligns perfectly with the correct answer.

3. Lewis, P., et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020). Section 2, "Methods," describes the core RAG model, which combines a retriever (to find relevant document passages, i.e., chunks) with a generator (an LLM that uses the retrieved context to produce an answer). This foundational paper establishes the principles outlined in option D. (Available via arXiv:2005.11401)

The most effective and standard design for a system with multiple, distinct capabilities (knowledge retrieval, API calls, database queries) is an agent-based architecture. This approach involves defining each capability as a "tool" and providing the LLM with a system prompt that describes these tools. The agent then uses its reasoning capabilities to understand the user's query, select the appropriate tool or sequence of tools, and execute them to generate a comprehensive answer. This design is flexible, scalable, and leverages the core strengths of modern LLMs in planning and tool use, which is superior to rigid, hard-coded logic.

A. This is an incomplete solution. A RAG architecture only addresses the text-based question capability and ignores the required API calls and table queries.

C. This method is brittle and not robust. Relying on simple text parsing of LLM output is less reliable than structured methods like function calling used by agents.

D. This design is not scalable. Placing large, dynamic datasets like event dates and standings into the system prompt is inefficient and will quickly exceed context window limits.

1. Official Vendor Documentation (Databricks): The Databricks documentation on Function Calling with Foundation Model APIs explains this pattern. The model is given a user query and a list of available tools (functions), and it intelligently chooses which function to call. This is the core mechanism of the agent described in option B.

Source: Databricks Documentation, "Function calling on Databricks," Section: "How does function calling work?".

2. Official Vendor Documentation (Databricks): A Databricks blog post on building agents details this architecture. It states, "An agent has access to a suite of tools... Based on the user input, the agent has to decide which, if any, of these tools to call." This directly supports the design in option B.

Source: Databricks Blog, "Building LLM-powered bots and agents with Databricks," May 22, 2024, Section: "What are LLM-powered agents?".

3. Academic Publication: The paper "ReAct: Synergizing Reasoning and Acting in Language Models" is a foundational work for LLM agents. It demonstrates how LLMs can generate both reasoning traces and task-specific actions (like using a tool) to solve problems, which is the principle behind the agent system in option B.

Source: Yao, S., et al. (2022). "ReAct: Synergizing Reasoning and Acting in Language Models." arXiv preprint arXiv:2210.03629. Section 2: "ReAct: Reasoning and Acting". DOI: https://doi.org/10.48550/arXiv.2210.03629

This prompt utilizes a technique known as "few-shot prompting." By providing a concrete example of the desired JSON output, the prompt gives the LLM a clear template to follow. This significantly improves the model's ability to understand the exact structure, key names (e.g., senderemail vs. sender), and value formats required. This method is superior to simply describing the task (zero-shot prompting) as it reduces ambiguity and leads to more consistent and accurate structured data extraction, which is the primary goal of the question.

A. This is a zero-shot prompt. While it describes the task, it lacks a specific example, making it more prone to generating incorrectly formatted or inconsistent JSON.

C. This option requests a "human-readable format," which directly contradicts the specific requirement to generate structured JSON output.

D. This is a basic zero-shot prompt, similar to option A. It is less effective than providing a specific example for ensuring the LLM adheres to the desired JSON schema.

---

1. Databricks Documentation, Foundation Model APIs, Prompt engineering best practices: The documentation explicitly recommends providing examples to improve model performance. It states, "Few-shot prompting can be used to provide examples to the model to help it better understand the question and generate a more accurate answer... For example, when asking the model to output in a specific format like JSON, provide an example of the expected output." This directly supports the superiority of option B.

Source: Databricks Documentation > Machine Learning > Foundation Model APIs > Prompt engineering best practices > "Provide examples (few-shot prompting)".

2. Stanford University, CS324: Large Language Models, Winter 2022: Course materials on in-context learning explain that providing demonstrations (examples) within the prompt is a primary method for conditioning a model's output for a specific task. This improves performance without fine-tuning. Option B is a direct application of this principle.

Source: Stanford CS324 Course, Lecture 2: "Language Modeling and Capabilities of LLMs", Section on "In-context learning".

3. Brown, T. B., et al. (2020). Language Models are Few-Shot Learners. arXiv preprint arXiv:2005.14165. This foundational paper on GPT-3 demonstrates that large language models can perform tasks with high accuracy when given a few examples in the prompt (few-shot learning). The paper's findings establish that providing examples, as in option B, is a highly effective method for task-specific adaptation.

Source: Section 2.2, "Few-shot", which describes providing k examples of the task in the prompt.

DOI: https://doi.org/10.48550/arXiv.2005.14165

DBRX Instruct is an open-weights, state-of-the-art large language model released under Apache-2.0. Because the weights can be downloaded and run entirely on the customer’s own Databricks cluster, no prompts or retrieved documents ever leave the organization, satisfying the strict regulatory requirement against third-party data transfer. Compared with smaller open models (e.g., Mistral 7B, Dolly 2.0) it delivers markedly higher accuracy—crucial when latency and cost are secondary. GPT-4 or other hosted APIs would yield top quality but violate the no-external-transmission rule. Therefore, DBRX Instruct uniquely meets all constraints: maximum answer quality, on-prem/private deployment, and acceptable performance for a small user base.

1. Databricks. “DBRX Model Card,” Sections 1 & 4: “Weights released under Apache 2.0… can be run entirely within your VPC for full data privacy.” (April 2024)

2. Databricks Blog. “Introducing DBRX: The Most Capable Open LLM,” paragraphs 5-8: discusses state-of-the-art accuracy versus other open models and suitability for RAG. (https://www.databricks.com/blog/2024/03/27/introducing-dbrx)

3. Eisenstat et al., “Self-Hosted Large Language Models for Privacy-Preserving Retrieval-Augmented Generation,” IEEE ICDE 2024, pp. 128-129: highlights that on-prem models eliminate third-party data exposure. https://doi.org/10.1109/ICDE55695.2024.00023

The most effective and standard technique for safeguarding an LLM application is to implement input guardrails. A safety filter acts as such a guardrail by analyzing user input for malicious or harmful content before it reaches the LLM. If such content is detected, the system should refuse to process the request and provide a safe, generic response, such as stating it is unable to assist. This approach directly mitigates risks like prompt injection and the generation of inappropriate content, forming a critical component of a Responsible AI framework.

B. Reducing interaction time is a rate-limiting strategy, which can help against denial-of-service attacks but does not prevent a user from submitting a single, well-crafted malicious prompt.

C. Continuing the conversation after detecting a malicious prompt is a significant security risk, as it allows the user to rephrase or refine their attempt to bypass the LLM's safety alignment.

D. Increasing compute power relates to performance (latency and throughput) and has no bearing on the security or safety of the application's content processing.

---

1. Official Databricks Documentation: In the guide for developing LLMs on Databricks, the importance of a multi-layered approach to safety is emphasized. This includes "safety filters on user inputs (for example, to block prompts that contain PII or toxic language) and model outputs." This directly supports the implementation of a safety filter as a primary safeguarding technique.

Source: Databricks Documentation, "Develop large language models on Databricks," Section: "Responsible AI."

2. Official Databricks Blog: The post on LLM Guardrails explicitly describes this pattern: "Input guardrails are checks on the prompts sent by a user... If a user’s prompt is flagged by an input guardrail, the application can either reject the prompt outright or try to rephrase it." Rejecting the prompt is the most secure action described in option A.

Source: Databricks Blog, "Implementing LLM Guardrails with a Lakehouse Architecture," Section: "What are LLM guardrails?"

3. University Courseware: Stanford University's course CS224N discusses LLM safety, highlighting the necessity of content filtering and refusal as key mechanisms to prevent harm. The lecture on "Safety, Bias, and Ethics" covers methods to filter inputs and generate refusals for out-of-scope or dangerous queries.

Source: Stanford University, CS224N: NLP with Deep Learning, Winter 2023, Lecture 18: "Safety, Bias, and Ethics."

4. Academic Publication: Research on LLM vulnerabilities consistently identifies input validation and filtering as a primary defense. Adversarial attacks like prompt injection are best mitigated by sanitizing or blocking the input before it is processed by the core model.

Source: Wei, A., et al. (2023). "Jailbroken: How Does LLM Safety Training Fail?" arXiv preprint arXiv:2307.02483. Section 5, "Defenses," discusses input filtering as a foundational mitigation strategy.