Foundation Model Integration, Data Management, and Compliance · Topic 6 of 9
Data Integration, Synchronization, Chunking and Embeddings
Connect wikis and document systems to RAG, keep vector stores current with incremental sync, pick a chunking strategy and choose the right embedding model.
- 6 min
- Includes a diagram
- Bedrock Knowledge Bases connectors: S3, Confluence, SharePoint, Salesforce, web crawler, plus custom data sources.
- Sync is incremental; event driven sync for freshness, scheduled sync for sources without events.
- Chunking options: default, fixed size, hierarchical, semantic, none, plus custom Lambda transformation.
What it is
Before a vector store can answer anything, content must flow in from where it lives, such as S3, wikis and document management systems. It must then be split into chunks and turned into embeddings.
After that first load, the content keeps changing. A vector store that is not kept in sync returns outdated answers with full confidence, which is worse than no answer.
How it works
Integration components for data sources

| Source | Integration option |
|---|---|
| Files in Amazon S3 | Bedrock Knowledge Bases S3 data source |
| Confluence, SharePoint, Salesforce | Built in Bedrock Knowledge Bases connectors |
| Public or internal web pages | Bedrock Knowledge Bases web crawler data source |
| Any other system | Custom data source with direct ingestion APIs, fed by Lambda |
| Enterprise assistant over many apps | Amazon Q Business connectors |
| SaaS data into S3 | Amazon AppFlow, then an S3 data source |
When no connector exists, a Lambda function or AWS Glue job reads the source API, writes documents and metadata to S3 or calls the knowledge base ingestion API, and runs on a schedule or event. Keep credentials for those systems in AWS Secrets Manager.
Keeping the vector store current
Incremental updates. A Bedrock knowledge base sync (StartIngestionJob) processes only documents that were added, changed or deleted since the last sync, so you do not re-embed everything.
Real time change detection. S3 event notifications or Amazon EventBridge events signal a new or changed file. A Lambda function starts an ingestion job or, for a custom data source, pushes the single document with the direct ingestion API.
Automated synchronization workflows. AWS Step Functions can chain extract, validate, sync and verify steps, with retries and alerts.
Scheduled refresh pipelines. Amazon EventBridge Scheduler runs a sync nightly or hourly for sources that do not emit change events, such as a wiki.
| Approach | Freshness | Choose it when |
|---|---|---|
| Event driven | Near real time | Source emits change events and freshness matters |
| Scheduled | Up to one interval old | Source has no events, or changes in batches |
| Full rebuild | Fresh after a long job | Chunking or embedding model changes |
Deletions matter as much as additions. A sync that removes deleted documents keeps the model from quoting retired policies.
Document segmentation (chunking)
Chunk size controls what retrieval returns. Small chunks are precise but lose context; large chunks keep context but dilute relevance and use more of the context window.
| Strategy in Bedrock Knowledge Bases | How it splits | Best for |
|---|---|---|
| Default | Chunks of about 300 tokens that respect sentence boundaries | Getting started |
| Fixed size | Your maximum tokens per chunk plus an overlap percentage | Uniform text, predictable size |
| Hierarchical | Small child chunks for search, larger parent chunks returned for context | Long structured documents such as manuals |
| Semantic | Splits where meaning changes, using an embedding model | Text where topics shift unevenly |
| No chunking | Each file is one chunk | Content you pre-split yourself |
For full control, a custom transformation Lambda function runs during ingestion and can apply your own logic, such as splitting by headings, keeping tables whole or adding metadata per chunk. A standalone Lambda function can also implement fixed size chunking before writing to your own vector store.
You cannot change the chunking strategy of an existing data source. Changing it means a new data source and re-ingestion, so test strategies on a sample first.
Choosing embedding models
| Model on Amazon Bedrock | Notes |
|---|---|
| Amazon Titan Text Embeddings V2 | Text; output size of 1024, 512 or 256 dimensions; optional normalization |
| Amazon Titan Multimodal Embeddings G1 | Text and images in one vector space |
| Cohere Embed models | Text embeddings, including multilingual options |
Dimensionality is a trade off. More dimensions can capture more nuance, while fewer dimensions cut storage and memory and speed up search. Test whether the smaller size keeps acceptable recall on your queries.
Domain fit matters more than generic scores. Evaluate candidates by embedding your own documents and measuring whether known relevant passages come back for real questions.
To embed large volumes, use Amazon Bedrock batch inference, which reads input records from S3 and writes results back asynchronously. For smaller jobs, a Lambda function can group texts and call the model in batches with backoff for throttling.
How the exam asks about this
- "Confluence pages must feed a RAG assistant with minimal custom code" -> Bedrock Knowledge Bases Confluence connector with scheduled sync.
- "Answers must reflect S3 documents within minutes of upload" -> S3 events to EventBridge to Lambda starting an incremental ingestion job.
- "Long technical manuals return fragments that lack context" -> Hierarchical chunking.
- "Cut vector storage cost with little accuracy loss" -> Titan Text Embeddings V2 at a lower dimension, validated on test queries.
Common mistakes
Full re-embedding on every change. Knowledge base sync is incremental; a full rebuild wastes time and money.
Mixing embedding models. Query and document vectors must come from the same model. Changing the model means re-embedding the whole store.
One chunk size for all content. Contracts, chat logs and tables need different strategies.
Forgetting deletes. Removed source files must also leave the vector store.
Chunks with no overlap can split a key sentence in half so neither chunk answers the question. Use some overlap with fixed size chunking.
What to remember for the exam
- Bedrock Knowledge Bases connectors: S3, Confluence, SharePoint, Salesforce, web crawler, plus custom data sources.
- Sync is incremental; event driven sync for freshness, scheduled sync for sources without events.
- Chunking options: default, fixed size, hierarchical, semantic, none, plus custom Lambda transformation.
- Chunking strategy is fixed per data source; changing it means re-ingestion.
- Titan Text Embeddings V2 supports 1024, 512 or 256 dimensions.
- Same embedding model for documents and queries; batch inference for bulk embedding.
Check yourself
How do you keep a knowledge base current within minutes of S3 uploads?
Send S3 events through EventBridge to a Lambda function that starts an incremental ingestion job.
What does hierarchical chunking return to the model?
It searches small child chunks for precision and returns the larger parent chunk for context.
What must happen if you switch to a different embedding model?
All documents must be re-embedded, because queries and documents must use vectors from the same model.
Topic 6 of 9 in Foundation Model Integration, Data Management, and Compliance
Test yourself on Foundation Model Integration, Data Management, and Compliance
The full AWS AIP-C01 question bank has 827 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.