Source Documents, Extraction and Delta Tables

What it is

A RAG application can only answer from what it retrieves, so data preparation decides its ceiling. You choose the source documents that hold the needed knowledge, extract their text with a parser that fits the format, filter out content that adds noise, and write the resulting chunks to a Delta table in Unity Catalog.

That table is the source for a Vector Search index.

How it works

Choosing source documents

Start from the questions users will ask, then find documents that answer them.

More documents are not always better. Loosely related or outdated material is retrieved in place of the right passage.

Raw files are usually stored in a Unity Catalog volume, so access is governed and the pipeline reads them by path.

Choosing a Python package for extraction

On Databricks, the ai_parse_document SQL function can also parse PDFs and other binary documents into structured content without managing parser libraries.

Pick the parser that keeps the structure you need; a plain text extractor flattens tables.

Filtering extraneous content

Remove content that matches many queries but answers none:

• Repeated page headers, footers, page numbers and copyright lines.
• Navigation menus, cookie banners and scripts in web pages.
• Tables of contents and indexes that repeat headings without content.
• Boilerplate disclaimers repeated in every document.
• Encoding errors, broken characters and very short fragments.

Filtering happens after extraction and before chunking. It improves retrieval because the embedding of each chunk then reflects its real subject, not the shared boilerplate.

Writing chunks to a Delta table

The usual order of operations:

1. Read the raw files from the volume.
2. Parse and filter each file.
3. Split into chunks and give each a stable unique ID, plus metadata such as source path, title and page.
4. Write to a Delta table in Unity Catalog with a three level name, catalog.schema.table.
5. Enable Change Data Feed on the table, which a Delta Sync index needs to pick up changes.
6. Create the Delta Sync index with the chunk ID as the primary key and the text column to embed.

CREATE TABLE IF NOT EXISTS main.rag.doc_chunks (
  chunk_id STRING, doc_path STRING, chunk_text STRING
) TBLPROPERTIES (delta.enableChangeDataFeed = true);

chunks_df.write.mode("append").saveAsTable("main.rag.doc_chunks")

A Delta Sync index then picks up changed rows in triggered mode (each time a sync runs) or continuous mode (within seconds).

How the exam asks about this

• "Answers quote last year's pricing although this year's sheet was added." Remove or replace outdated sources; both versions are retrieved and compete.
• "Scanned contracts return empty text when parsed." The files have no text layer, so use OCR before chunking.
• "The Delta Sync index cannot be created on the chunks table." Check that the table is in Unity Catalog, has a primary key column and has Change Data Feed enabled.