Application Development · Topic 2 of 4
Chunking Strategy and Embedding Context Length
Databricks GenAI Engineer: pick a chunking strategy by document type and retrieval results, and an embedding model whose context length fits the chunks.
- 5 min
- Chunk to the document's structure: headings for manuals, one record per item for FAQs and tickets.
- Overlap protects boundary sentences; too much creates duplicate hits.
- Choose chunk size by measuring retrieval and answer quality on an evaluation set.
What it is
Chunking splits source documents into pieces that are embedded and stored in a vector index. The chunk is the unit that retrieval returns, so its size and boundaries decide what context the LLM sees.
The embedding model's context length is the maximum number of tokens it can embed at once. Chunks must fit inside it, and the choice depends on the documents, the queries users ask and whether you optimize for precision, recall, cost or speed.
How it works
Chunking strategies
| Strategy | How it splits | Fits |
|---|---|---|
| Fixed size with overlap | Every N tokens, repeating some tokens at each boundary | Uniform plain text, a quick baseline |
| Recursive or sentence aware | On paragraphs, then sentences, then words, up to a size limit | General prose, articles, emails |
| Structure based | On headings, sections, pages or HTML and Markdown elements | Manuals, policies, documentation |
| Semantic | Where the meaning changes between sentences | Long text that mixes topics |
| Per record | One chunk per row, ticket, FAQ entry or product | Short, self contained items |
| Parent and child | Small chunks for search, a larger parent passage returned to the LLM | Precise search with enough context to answer |
Overlap keeps a sentence that crosses a boundary findable from both sides. Too much overlap stores near duplicate chunks, which costs storage and can fill the top results with the same text.
Size trade offs
| Smaller chunks | Larger chunks |
|---|---|
| More precise matches to narrow questions | More surrounding context per hit |
| Risk of losing context needed to answer | Risk of diluting the embedding with several topics |
| More chunks, so a larger index | Fewer chunks, but fewer fit in the LLM prompt |
The expected queries matter. Short factual questions ("what is the refund window?") favor small, precise chunks. Broad questions ("summarize the travel policy") favor larger chunks or a parent and child layout.
Choosing by evaluation, not by guess
Pick the strategy with measurements. Build an evaluation set of questions with the expected source passages or expected facts, try two or three chunking setups, and compare retrieval quality (did the right chunk appear in the top results) and final answer quality with MLflow evaluation.
import mlflow
from mlflow.genai.scorers import RetrievalGroundedness, RetrievalSufficiency
results = mlflow.genai.evaluate(
data=eval_set, predict_fn=rag_app,
scorers=[RetrievalSufficiency(), RetrievalGroundedness()])RetrievalSufficiency needs expected facts or an expected response in the evaluation set; groundedness does not. If retrieval scores are low, change the chunking or retrieval before rewriting prompts.
Embedding model context length
Text longer than the embedding model's limit is truncated, so the end of an oversized chunk is never embedded and cannot be found. Foundation Model APIs offer embedding models with different limits:
| Endpoint | Max context | Notes |
|---|---|---|
| databricks-gte-large-en | 8,192 tokens | English, 1024 dimensions |
| databricks-bge-large-en | 512 tokens | English, 1024 dimensions, normalized output |
Choose the context length from three inputs:
- Source documents: long sections that must stay together need a longer limit, or must be chunked smaller.
- Expected queries: queries are embedded with the same model, and short queries match best against focused chunks.
- Optimization strategy: a longer limit lets you embed bigger chunks with fewer vectors; a smaller model is often faster and cheaper per call.
Use the same embedding model for indexing and for queries. Vectors from different models live in different spaces and do not compare.
How the exam asks about this
- "Answers miss details that sit at the end of long sections; the embedding model has a 512 token limit." Chunks are being truncated; chunk smaller or use a model with a longer context length.
- "A product FAQ where each answer is two sentences." Use one chunk per FAQ entry.
- "Retrieval returns the right section but too little text to answer." Use larger chunks or parent and child retrieval.
What to remember for the exam
- Chunk to the document's structure: headings for manuals, one record per item for FAQs and tickets.
- Overlap protects boundary sentences; too much creates duplicate hits.
- Choose chunk size by measuring retrieval and answer quality on an evaluation set.
- Chunks longer than the embedding model's context length are truncated.
- databricks-gte-large-en takes 8,192 tokens; databricks-bge-large-en takes 512.
- Index and query with the same embedding model.
A longer embedding context length is not a quality fix by itself. A chunk that mixes several topics produces a blurred vector, so oversized chunks can lower precision even when they fit.
Common mistakes
Rewriting the prompt to fix poor retrieval. If the right passage never reaches the LLM, no prompt can recover it. Fix chunking or retrieval first and measure again.
Using one chunk size for every source. A policy manual, a chat log and a product table need different splits. Pick the strategy per document type.
Swapping the embedding model without re-indexing. Changing the model for queries but not for the stored chunks breaks similarity search. Re-embed the whole index with the new model.
Check yourself
Why can a fact at the end of a long chunk never be retrieved?
The chunk is longer than the embedding model's context length, so the text past the limit is truncated and never embedded.
Which chunking suits a set of support tickets?
One chunk per ticket, since each ticket is short and self contained and is the natural unit to return.
What tells you a new chunking strategy is better?
Higher retrieval and answer scores on the same evaluation set, compared in MLflow, rather than a few spot checks.
Topic 2 of 4 in Application Development
Test yourself on Application Development
The full Databricks Generative AI Engineer Associate question bank has 600 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Databricks. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.