RAG Pipelines, Chunking and AI Search Configuration

What it is

A RAG pipeline turns a document corpus into chunks the agent can retrieve and place in its context. On Databricks the documents sit under Unity Catalog governance, the chunks live in a Delta table, and a Databricks AI Search index (the docs may still say Vector Search) serves similarity queries.

Most retrieval quality problems come from three places: how the documents were chunked, how the index is configured and synced, and how the query is sent. The exam describes a symptom and asks which of these to change.

How it works

The governed pipeline

1. Store source files in a Unity Catalog volume, so access is granted and audited in one place.
2. Parse and chunk them into a Delta table with a primary key, the chunk text and metadata columns such as source path, section, document type and effective date.
3. Create a DELTA_SYNC index on that table. On standard endpoints the source table needs Change Data Feed enabled. Either let AI Search compute embeddings from a text column with a model serving endpoint, or supply your own embedding column.
4. At run time, query the index and insert the returned chunks, with their source, into the prompt or the tool result.

Metadata columns let you filter by document type or date, cite sources and trace a bad answer back to a file.

Choosing a chunking strategy

The embedding model's context length is a hard ceiling. Text beyond the model's input limit is truncated, so it is never embedded and cannot be matched. With a 512 token embedding model, an 1,800 token section chunk is mostly invisible to search; split it, or use a model with a longer input limit.

A modest overlap keeps a sentence that crosses a boundary findable; too much fills the top results with near duplicates.

AI Search settings that affect signal quality

A DIRECT_ACCESS index is not synced from a table at all. Your code writes vectors with upserts, so staleness there means the ingestion code skipped an upsert.

Querying the index

from databricks.vector_search.client import VectorSearchClient
index = VectorSearchClient().get_index(
    endpoint_name="kb_endpoint", index_name="main.kb.policy_chunks_idx")
hits = index.similarity_search(
    query_text="EU refund window for damaged items",
    columns=["chunk_id", "text", "source_path"],
    num_results=5, filters={"region": "EU"}, query_type="HYBRID")

Return the source column with the text, so the agent can cite each fact.

A small worked budget

The model has 16,000 tokens for retrieved context after the system prompt and history. With 800 token chunks, num_results of 10 uses 8,000 tokens and leaves room; num_results of 25 would need 20,000 and overflow. Smaller chunks of 300 tokens let 10 results fit in 3,000 tokens, but each carries less surrounding context.

How the exam asks about this

• "The source table is updated hourly, the index is TRIGGERED, and answers lag by days." The sync is not being run; call sync after each update job, or use CONTINUOUS only if seconds of freshness are required.
• "Users search by SKU codes and the agent returns similar but wrong products." Switch query_type to HYBRID so keyword matching catches exact codes.
• "Long sections are chunked whole and the embedding model accepts 512 tokens; later parts of sections are never retrieved." Re chunk below the embedding limit, splitting by subsection.