Testing, Validation, and Troubleshooting · Topic 4 of 6

Retrieval Quality Testing and Troubleshooting

Test RAG retrieval with relevance scoring, context checks and latency, then fix embedding drift, chunking, preprocessing and slow vector search.

  • 5 min
In a nutshell
  • Test retrieval alone with Retrieve or a retrieve only evaluation job; measure relevance, coverage and latency.
  • Bad answer with good sources means a generation problem; bad sources means a retrieval problem.
  • Chunking options: default, fixed-size, hierarchical, semantic, none, plus a custom Lambda transformation.

What it is

In a RAG application, the model can only be as good as the context it receives. If retrieval returns the wrong chunks, even the best model answers badly or hallucinates.

Retrieval quality testing measures the retriever on its own: are the returned chunks relevant, do they contain the needed facts, and how fast do they arrive. Troubleshooting works backward from a bad answer to the cause, which is usually in chunking, embeddings or the search configuration.

QuestionuserembedVector searchindextop resultsChunkscontextpromptFManswer

How it works

Testing retrieval

Build a test set of questions with the documents or passages that should be found for each one. Then measure:

MeasureWhat it showsHow
Relevance scoringAre the returned chunks about the question?Bedrock RAG evaluation context relevance, or the score returned with each Retrieve result
Context matchingDo the chunks contain the facts needed to answer?Bedrock context coverage against ground truth texts
Recall at kIs the expected passage in the top k results?Compare result IDs with the labeled passages
Retrieval latencyTime spent in search before generationTime the Retrieve call separately from generation

Use the Retrieve API to test the retriever alone. RetrieveAndGenerate mixes retrieval with generation, so a bad answer cannot be blamed on either step. A Bedrock retrieve only evaluation job scores retrieval at scale.

Exam tip

Test retrieval and generation separately. If the right chunk is not in the results, no prompt change will fix the answer.

Diagnosing a bad answer

Start from the response and walk back.

Bad answercheck sourcesRight chunks retrievedyesGeneration issueprompt or modelBad answercheck sourcesRight chunks retrievednoRetrieval issuechunks, embeddings, search

Model response relevance analysis compares the answer with the question and with the cited sources. If the citations are relevant but the answer ignores them, fix the prompt. If the citations are off topic, fix retrieval.

Chunking and preprocessing remediation

Amazon Bedrock Knowledge Bases offer default chunking (about 300 tokens, keeping sentence boundaries), fixed-size chunking with an overlap percentage, hierarchical chunking, semantic chunking and no chunking. A custom transformation Lambda function can apply your own chunking logic or add chunk-level metadata.

SymptomLikely causeFix
Answers miss facts that span paragraphsChunks too small, no overlapLarger chunks, overlap, or hierarchical chunking (search child chunks, return parent)
Results relevant but noisyChunks too large, many topics per chunkSmaller or semantic chunks
Tables and headers come back garbledPoor parsing of complex documentsUse a foundation model or Bedrock Data Automation parser
Boilerplate matches every queryHeaders, footers or navigation text indexedClean documents before ingestion

The chunking strategy cannot be changed after the data source is connected. To change it, create a new data source and ingest again.

Embedding quality and drift

Queries and documents must be embedded with the same embedding model and settings. Changing the model, or the output dimensions of a model such as Amazon Titan Text Embeddings V2, makes old vectors incomparable with new query vectors, so the whole index must be re-embedded.

  • Embedding quality diagnostics: check that known paraphrases land close together and unrelated texts land far apart; inspect nearest neighbors for sample queries.
  • Drift monitoring: track average retrieval scores, the share of queries with no good match and user ratings over time. A falling trend means content or questions have moved away from the index.
  • Vectorization issues: documents that failed ingestion, empty text after parsing, or a stale index after a source update. Check the ingestion job results and resync the data source.

Vector search performance

ProblemRemedy
Exact keywords, product codes or names are missedHybrid search (overrideSearchType set to HYBRID), supported on Amazon OpenSearch Serverless, Amazon RDS and MongoDB vector stores with a filterable text field
Results from the wrong product, tenant or dateMetadata filters on the query
Right chunk is found but ranked too lowA reranker model on the results
Complex questions need several factsQuery decomposition into sub-queries
Too much or too little contextTune numberOfResults (default 5)
Search is slow at scaleScale the vector store, tune index settings, filter before search, cache frequent queries

Common mistakes

Tuning the prompt for a retrieval failure. If the right passage is not in the results, prompt changes cannot help.

Changing the embedding model without re-indexing. Old document vectors and new query vectors no longer match, and relevance collapses.

Expecting to edit the chunking strategy in place. It is fixed per data source; plan a new data source and a full ingestion.

Using pure semantic search for codes and IDs. Embeddings blur exact tokens. Hybrid search adds keyword matching.

What to remember for the exam

  • Test retrieval alone with Retrieve or a retrieve only evaluation job; measure relevance, coverage and latency.
  • Bad answer with good sources means a generation problem; bad sources means a retrieval problem.
  • Chunking options: default, fixed-size, hierarchical, semantic, none, plus a custom Lambda transformation.
  • Chunking strategy is fixed once the data source is connected.
  • Same embedding model for documents and queries; a model change means re-embedding everything.
  • Hybrid search for exact terms, metadata filters for scope, reranking for ordering, query decomposition for multi part questions.

Check yourself

  • Users search for part numbers and the knowledge base returns similar but wrong parts. What should you change?

    Use hybrid search so keyword matching on the exact part number complements semantic search.

  • Answers lack details that sit in the paragraph next to the matched text. Which chunking change helps?

    Hierarchical chunking, which searches small child chunks and returns the larger parent chunk, or larger chunks with overlap.

  • After switching to a new embedding model, retrieval quality collapses. Why?

    The stored document vectors came from the old model, so the whole corpus must be re-embedded with the new model.

Topic 4 of 6 in Testing, Validation, and Troubleshooting

Test yourself on Testing, Validation, and Troubleshooting

The full AWS AIP-C01 question bank has 827 questions, with timed practice exams and an explanation for every answer.

This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.