Implementation and Integration · Topic 6 of 8

FM APIs, Streaming and Resilient Model Calls

Call Bedrock synchronously or via SQS, stream tokens with ConverseStream and API Gateway, manage token limits, and add backoff, rate limits and fallbacks.

  • 6 min
In a nutshell
  • InvokeModel and Converse are synchronous; InvokeModelWithResponseStream and ConverseStream stream.
  • SQS decouples bursty work from the FM quota.
  • API Gateway validates requests, throttles clients and can stream REST responses.

What it is

Every GenAI application ends in a model call. How that call is made decides latency, user experience and what happens when the model is slow or throttled.

This page covers the interaction styles (synchronous, asynchronous, streaming), the API layer that fronts the model, and the resilience patterns that keep the application up when the FM is not.

ClientrequestAPI Gatewayvalidation, throttlinginvokeLambdaSDK with retriesConverse or ConverseStreamAmazon Bedrock

How it works

Interaction styles

StyleBedrock APIPick it when
SynchronousInvokeModel or ConverseShort answers needed right away, from Lambda, containers, EC2 or on premises code with an AWS SDK
StreamingInvokeModelWithResponseStream or ConverseStreamChat and long answers where the user should see text as it is generated
Asynchronous queueAmazon SQS in front of workers that call BedrockBursty or long work where the caller does not wait, such as document processing
BatchBedrock batch inference jobLarge offline datasets where a result in hours is fine

The Converse API gives one request shape across models that support it, so switching models needs less code change. InvokeModel takes the native body format of each model provider.

Language specific AWS SDKs (Python with Boto3, JavaScript, Java and others) call the same Bedrock APIs from any compute environment, and they bring built in retries and credential handling.

Producerupload or APISendMessageAmazon SQSbufferpollWorkerLambda or ECSConverseAmazon Bedrockstore resultDynamoDB or S3

The queue absorbs bursts, so workers call the FM at a steady rate under the quota. A dead letter queue catches requests that keep failing.

Request validation with API Gateway

API Gateway REST APIs can validate requests before they reach the backend. A request validator with a JSON schema model rejects malformed bodies, missing parameters or over long prompts with a 400 error, so bad requests never cost a model call.

Custom API clients get one stable contract, while the backend can change models freely. Usage plans with API keys give each client its own throttling and quota.

Real time streaming

Streaming lowers the time to first token the user sees. The total generation time is the same, but the answer starts appearing at once.

Delivery methodHow it worksNotes
WebSocket (API Gateway WebSocket API)Two way connection; the backend pushes chunks to the connection IDGood for chat where the client also sends messages mid session
Server sent eventsOne way HTTP stream of events from server to browserSimple for one way token streaming
API Gateway REST response streamingProxy integration with response transfer mode STREAM sends chunks as they arriveWorks with Lambda proxy and HTTP proxy integrations
Lambda response streamingFunction URL with streaming invoke modeDirect streaming from a function without API Gateway
BrowserHTTPSAPI Gatewaytransfer mode STREAMproxyLambdaConverseStreamchunksAPI Gatewaychunked responseBrowser
Exam tip

REST API response streaming does not support endpoint caching or VTL response transformation, because API Gateway no longer buffers the whole response. A buffered response is limited by the integration timeout, 29 seconds by default.

Token limits and timeouts

FM APIs have needs ordinary APIs do not.

  • Token limits: the prompt plus the output must fit the model's context window. Count or estimate input tokens, trim history or retrieved text, and set maxTokens in the inference configuration to cap output length and cost.
  • Truncation: if the response ends because of the token cap, the stop reason says so (max_tokens in Converse). The application can ask for a continuation or raise the limit.
  • Model timeouts: long generations can exceed client or gateway timeouts. Stream the response, move long work to an asynchronous queue, or set client timeouts above the expected generation time. Retry a timeout only with a limit, because each retry costs tokens.

Resilience patterns

PatternAWS mechanismProtects against
Exponential backoff with jitterAWS SDK retry modes (standard or adaptive)ThrottlingException and transient errors
Rate limitingAPI Gateway throttling and usage plansOne client exhausting the shared model quota
Fallback and graceful degradationSecondary model, cross Region inference profile, cached answer, or a "try again later" messageModel or Region unavailability
Circuit breakerFailure counter that stops calls for a cool downHammering a failing dependency
ObservabilityAWS X-Ray traces across API Gateway, Lambda and the Bedrock callNot knowing which hop is slow or failing
RequestcallPrimary modelthrottled, retry with backoffPrimary modelPrimary modelstill failingFallbacksmaller model or cached answerdegraded but workingClient

Jitter adds randomness to each wait, so many clients that failed together do not retry at the same moment and cause another spike.

X-Ray shows each segment of a request. If latency jumps, the trace shows whether the time went to the model, the retrieval step or the application code.

Watch out

Retrying without backoff during throttling makes it worse. Every immediate retry is another request against the same exhausted quota.

How the exam asks about this

  • "Users wait 20 seconds before any text appears in the chat" -> ConverseStream delivered through WebSockets, server sent events or API Gateway response streaming.
  • "A nightly upload of 50,000 documents causes throttling errors" -> put the work in SQS and process at a controlled rate, with SDK backoff.
  • "The app must keep answering when the primary model is unavailable" -> a fallback model or cached answer for graceful degradation.

Common mistakes

Synchronous calls for long jobs. They hit Lambda or gateway timeouts. Use a queue or streaming.

No output cap. Without maxTokens, a runaway answer costs more and can exceed timeouts.

Ignoring the stop reason. A truncated answer looks complete to the user unless the app checks why generation stopped.

One shared quota with no per client limits. A single noisy client can throttle everyone. Use usage plans.

What to remember for the exam

  • InvokeModel and Converse are synchronous; InvokeModelWithResponseStream and ConverseStream stream.
  • SQS decouples bursty work from the FM quota.
  • API Gateway validates requests, throttles clients and can stream REST responses.
  • Manage tokens with input trimming, maxTokens and the stop reason.
  • AWS SDK retries use exponential backoff; add fallbacks for graceful degradation and X-Ray for traces.

Check yourself

  • Which Bedrock API streams a response through the unified message format?

    ConverseStream, the streaming form of the Converse API.

  • Why add jitter to exponential backoff?

    So clients that failed together retry at different times instead of creating a new spike.

  • How can API Gateway stop malformed prompts from reaching the model?

    A REST API request validator with a JSON schema model rejects invalid requests with a 400 error.

Topic 6 of 8 in Implementation and Integration

Test yourself on Implementation and Integration

The full AWS AIP-C01 question bank has 827 questions, with timed practice exams and an explanation for every answer.

This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.