Implementation and Integration · Topic 6 of 8
FM APIs, Streaming and Resilient Model Calls
Call Bedrock synchronously or via SQS, stream tokens with ConverseStream and API Gateway, manage token limits, and add backoff, rate limits and fallbacks.
- 6 min
InvokeModelandConverseare synchronous;InvokeModelWithResponseStreamandConverseStreamstream.- SQS decouples bursty work from the FM quota.
- API Gateway validates requests, throttles clients and can stream REST responses.
What it is
Every GenAI application ends in a model call. How that call is made decides latency, user experience and what happens when the model is slow or throttled.
This page covers the interaction styles (synchronous, asynchronous, streaming), the API layer that fronts the model, and the resilience patterns that keep the application up when the FM is not.
How it works
Interaction styles
| Style | Bedrock API | Pick it when |
|---|---|---|
| Synchronous | InvokeModel or Converse | Short answers needed right away, from Lambda, containers, EC2 or on premises code with an AWS SDK |
| Streaming | InvokeModelWithResponseStream or ConverseStream | Chat and long answers where the user should see text as it is generated |
| Asynchronous queue | Amazon SQS in front of workers that call Bedrock | Bursty or long work where the caller does not wait, such as document processing |
| Batch | Bedrock batch inference job | Large offline datasets where a result in hours is fine |
The Converse API gives one request shape across models that support it, so switching models needs less code change. InvokeModel takes the native body format of each model provider.
Language specific AWS SDKs (Python with Boto3, JavaScript, Java and others) call the same Bedrock APIs from any compute environment, and they bring built in retries and credential handling.
The queue absorbs bursts, so workers call the FM at a steady rate under the quota. A dead letter queue catches requests that keep failing.
Request validation with API Gateway
API Gateway REST APIs can validate requests before they reach the backend. A request validator with a JSON schema model rejects malformed bodies, missing parameters or over long prompts with a 400 error, so bad requests never cost a model call.
Custom API clients get one stable contract, while the backend can change models freely. Usage plans with API keys give each client its own throttling and quota.
Real time streaming
Streaming lowers the time to first token the user sees. The total generation time is the same, but the answer starts appearing at once.
| Delivery method | How it works | Notes |
|---|---|---|
| WebSocket (API Gateway WebSocket API) | Two way connection; the backend pushes chunks to the connection ID | Good for chat where the client also sends messages mid session |
| Server sent events | One way HTTP stream of events from server to browser | Simple for one way token streaming |
| API Gateway REST response streaming | Proxy integration with response transfer mode STREAM sends chunks as they arrive | Works with Lambda proxy and HTTP proxy integrations |
| Lambda response streaming | Function URL with streaming invoke mode | Direct streaming from a function without API Gateway |
REST API response streaming does not support endpoint caching or VTL response transformation, because API Gateway no longer buffers the whole response. A buffered response is limited by the integration timeout, 29 seconds by default.
Token limits and timeouts
FM APIs have needs ordinary APIs do not.
- Token limits: the prompt plus the output must fit the model's context window. Count or estimate input tokens, trim history or retrieved text, and set
maxTokensin the inference configuration to cap output length and cost. - Truncation: if the response ends because of the token cap, the stop reason says so (
max_tokensin Converse). The application can ask for a continuation or raise the limit. - Model timeouts: long generations can exceed client or gateway timeouts. Stream the response, move long work to an asynchronous queue, or set client timeouts above the expected generation time. Retry a timeout only with a limit, because each retry costs tokens.
Resilience patterns
| Pattern | AWS mechanism | Protects against |
|---|---|---|
| Exponential backoff with jitter | AWS SDK retry modes (standard or adaptive) | ThrottlingException and transient errors |
| Rate limiting | API Gateway throttling and usage plans | One client exhausting the shared model quota |
| Fallback and graceful degradation | Secondary model, cross Region inference profile, cached answer, or a "try again later" message | Model or Region unavailability |
| Circuit breaker | Failure counter that stops calls for a cool down | Hammering a failing dependency |
| Observability | AWS X-Ray traces across API Gateway, Lambda and the Bedrock call | Not knowing which hop is slow or failing |
Jitter adds randomness to each wait, so many clients that failed together do not retry at the same moment and cause another spike.
X-Ray shows each segment of a request. If latency jumps, the trace shows whether the time went to the model, the retrieval step or the application code.
Retrying without backoff during throttling makes it worse. Every immediate retry is another request against the same exhausted quota.
How the exam asks about this
- "Users wait 20 seconds before any text appears in the chat" ->
ConverseStreamdelivered through WebSockets, server sent events or API Gateway response streaming. - "A nightly upload of 50,000 documents causes throttling errors" -> put the work in SQS and process at a controlled rate, with SDK backoff.
- "The app must keep answering when the primary model is unavailable" -> a fallback model or cached answer for graceful degradation.
Common mistakes
Synchronous calls for long jobs. They hit Lambda or gateway timeouts. Use a queue or streaming.
No output cap. Without maxTokens, a runaway answer costs more and can exceed timeouts.
Ignoring the stop reason. A truncated answer looks complete to the user unless the app checks why generation stopped.
One shared quota with no per client limits. A single noisy client can throttle everyone. Use usage plans.
What to remember for the exam
InvokeModelandConverseare synchronous;InvokeModelWithResponseStreamandConverseStreamstream.- SQS decouples bursty work from the FM quota.
- API Gateway validates requests, throttles clients and can stream REST responses.
- Manage tokens with input trimming,
maxTokensand the stop reason. - AWS SDK retries use exponential backoff; add fallbacks for graceful degradation and X-Ray for traces.
Check yourself
Which Bedrock API streams a response through the unified message format?
ConverseStream, the streaming form of the Converse API.Why add jitter to exponential backoff?
So clients that failed together retry at different times instead of creating a new spike.
How can API Gateway stop malformed prompts from reaching the model?
A REST API request validator with a JSON schema model rejects invalid requests with a 400 error.
Topic 6 of 8 in Implementation and Integration
Test yourself on Implementation and Integration
The full AWS AIP-C01 question bank has 827 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.