Testing, Validation, and Troubleshooting · Topic 4 of 6
Retrieval Quality Testing and Troubleshooting
Test RAG retrieval with relevance scoring, context checks and latency, then fix embedding drift, chunking, preprocessing and slow vector search.
- 5 min
- Test retrieval alone with
Retrieveor a retrieve only evaluation job; measure relevance, coverage and latency. - Bad answer with good sources means a generation problem; bad sources means a retrieval problem.
- Chunking options: default, fixed-size, hierarchical, semantic, none, plus a custom Lambda transformation.
What it is
In a RAG application, the model can only be as good as the context it receives. If retrieval returns the wrong chunks, even the best model answers badly or hallucinates.
Retrieval quality testing measures the retriever on its own: are the returned chunks relevant, do they contain the needed facts, and how fast do they arrive. Troubleshooting works backward from a bad answer to the cause, which is usually in chunking, embeddings or the search configuration.
How it works
Testing retrieval
Build a test set of questions with the documents or passages that should be found for each one. Then measure:
| Measure | What it shows | How |
|---|---|---|
| Relevance scoring | Are the returned chunks about the question? | Bedrock RAG evaluation context relevance, or the score returned with each Retrieve result |
| Context matching | Do the chunks contain the facts needed to answer? | Bedrock context coverage against ground truth texts |
| Recall at k | Is the expected passage in the top k results? | Compare result IDs with the labeled passages |
| Retrieval latency | Time spent in search before generation | Time the Retrieve call separately from generation |
Use the Retrieve API to test the retriever alone. RetrieveAndGenerate mixes retrieval with generation, so a bad answer cannot be blamed on either step. A Bedrock retrieve only evaluation job scores retrieval at scale.
Test retrieval and generation separately. If the right chunk is not in the results, no prompt change will fix the answer.
Diagnosing a bad answer
Start from the response and walk back.
Model response relevance analysis compares the answer with the question and with the cited sources. If the citations are relevant but the answer ignores them, fix the prompt. If the citations are off topic, fix retrieval.
Chunking and preprocessing remediation
Amazon Bedrock Knowledge Bases offer default chunking (about 300 tokens, keeping sentence boundaries), fixed-size chunking with an overlap percentage, hierarchical chunking, semantic chunking and no chunking. A custom transformation Lambda function can apply your own chunking logic or add chunk-level metadata.
| Symptom | Likely cause | Fix |
|---|---|---|
| Answers miss facts that span paragraphs | Chunks too small, no overlap | Larger chunks, overlap, or hierarchical chunking (search child chunks, return parent) |
| Results relevant but noisy | Chunks too large, many topics per chunk | Smaller or semantic chunks |
| Tables and headers come back garbled | Poor parsing of complex documents | Use a foundation model or Bedrock Data Automation parser |
| Boilerplate matches every query | Headers, footers or navigation text indexed | Clean documents before ingestion |
The chunking strategy cannot be changed after the data source is connected. To change it, create a new data source and ingest again.
Embedding quality and drift
Queries and documents must be embedded with the same embedding model and settings. Changing the model, or the output dimensions of a model such as Amazon Titan Text Embeddings V2, makes old vectors incomparable with new query vectors, so the whole index must be re-embedded.
- Embedding quality diagnostics: check that known paraphrases land close together and unrelated texts land far apart; inspect nearest neighbors for sample queries.
- Drift monitoring: track average retrieval scores, the share of queries with no good match and user ratings over time. A falling trend means content or questions have moved away from the index.
- Vectorization issues: documents that failed ingestion, empty text after parsing, or a stale index after a source update. Check the ingestion job results and resync the data source.
Vector search performance
| Problem | Remedy |
|---|---|
| Exact keywords, product codes or names are missed | Hybrid search (overrideSearchType set to HYBRID), supported on Amazon OpenSearch Serverless, Amazon RDS and MongoDB vector stores with a filterable text field |
| Results from the wrong product, tenant or date | Metadata filters on the query |
| Right chunk is found but ranked too low | A reranker model on the results |
| Complex questions need several facts | Query decomposition into sub-queries |
| Too much or too little context | Tune numberOfResults (default 5) |
| Search is slow at scale | Scale the vector store, tune index settings, filter before search, cache frequent queries |
Common mistakes
Tuning the prompt for a retrieval failure. If the right passage is not in the results, prompt changes cannot help.
Changing the embedding model without re-indexing. Old document vectors and new query vectors no longer match, and relevance collapses.
Expecting to edit the chunking strategy in place. It is fixed per data source; plan a new data source and a full ingestion.
Using pure semantic search for codes and IDs. Embeddings blur exact tokens. Hybrid search adds keyword matching.
What to remember for the exam
- Test retrieval alone with
Retrieveor a retrieve only evaluation job; measure relevance, coverage and latency. - Bad answer with good sources means a generation problem; bad sources means a retrieval problem.
- Chunking options: default, fixed-size, hierarchical, semantic, none, plus a custom Lambda transformation.
- Chunking strategy is fixed once the data source is connected.
- Same embedding model for documents and queries; a model change means re-embedding everything.
- Hybrid search for exact terms, metadata filters for scope, reranking for ordering, query decomposition for multi part questions.
Check yourself
Users search for part numbers and the knowledge base returns similar but wrong parts. What should you change?
Use hybrid search so keyword matching on the exact part number complements semantic search.
Answers lack details that sit in the paragraph next to the matched text. Which chunking change helps?
Hierarchical chunking, which searches small child chunks and returns the larger parent chunk, or larger chunks with overlap.
After switching to a new embedding model, retrieval quality collapses. Why?
The stored document vectors came from the old model, so the whole corpus must be re-embedded with the new model.
Topic 4 of 6 in Testing, Validation, and Troubleshooting
Test yourself on Testing, Validation, and Troubleshooting
The full AWS AIP-C01 question bank has 827 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.