A Generative Al Engineer has successfully ingested unstructured documents and chunked them by document sections. They would like to store the chunks…

Databricks Vector Search indexes data from a source Delta table. For the indexing process to be effective and performant, each row in this table must represent a single, discrete unit of text (a chunk) that will be embedded. The original dataframe structure, with an array of chunks per document, is unsuitable. The most performant and correct approach is to "flatten" or "explode" this array, creating a new row for each individual chunk. Furthermore, a Vector Search source table has a strict requirement for a primary key column to uniquely identify each row (i.e., each chunk).

A. Splitting data into train/test sets is a machine learning modeling practice and is not relevant for creating a retrieval index with Vector Search.

C. This approach fails to transform the array of chunks into individual rows, which is a fundamental requirement for the Vector Search indexing process.

D. Databricks Vector Search requires a Delta table as its data source, not a collection of individual JSON files stored in a Unity Catalog Volume.

---

1. Databricks Official Documentation, "Prepare data for vector search": This document explicitly details the necessary data preparation steps. It states, "The following code chunks the text column and explodes the chunks into new rows," directly supporting the "flatten" action described in the correct answer. The provided code examples use the explode() function to achieve this one-chunk-per-row structure.

Source: Databricks Documentation > Generative AI > Vector Search > Prepare data for vector search.

2. Databricks Official Documentation, "Create a vector search index": This guide lists the requirements for a source table. It specifies that the source "Must be a Delta Table" and "Must have a primary key." This confirms that the data must be stored in a Delta table and that a unique identifier for each row (chunk) is mandatory, not optional.

Source: Databricks Documentation > Generative AI > Vector Search > Create a vector search index > Source table requirements.

3. Databricks Official Documentation, "Tutorial: Create and query a vector search index": The tutorial's data preparation section demonstrates creating a Delta table where each row contains a unique id and a text column with a single chunk of text, reinforcing the pattern of flattening the data and adding a unique key before indexing.

Source: Databricks Documentation > Generative AI > Vector Search > Tutorial: Create and query a vector search index > Step 2: Create a source Delta table.