Implementation and Integration · Topic 3 of 8

Model Coordination, Routing and Cascading

Combine specialized FMs, ensembles and model selection, route requests by content or metrics, and cascade from small models to large ones to save cost.

  • 6 min
  • Includes a diagram
In a nutshell
  • Specialized FMs handle sub tasks they suit; ensembles combine outputs with custom aggregation such as voting or a judge model.
  • Model selection is measured on your own evaluation data for quality, latency and cost.
  • Smaller pre trained models cut cost and latency for narrow tasks.

What it is

No single FM is best at everything. A large model writes the best analysis but is slow and expensive; a small model classifies or extracts fields quickly and cheaply.

Model coordination uses several models together. The exam tests three ways to do it: giving each task to a specialized model, combining several model outputs, and routing each request to the cheapest model that can handle it.

RequestinspectRouterrules or classifiersimpleSmall modelfast, cheapRoutercomplexLarge modelbest qualityRouterimageMultimodal model

How it works

Specialized models and ensembles

Model ensemble: the same request goes to several models, custom aggregation logic (majority vote, weighted vote or a judge model) combines the results, the answer is accepted, or escalated when the models disagree under a confidence threshold. A complex task often splits into sub tasks that suit different models. A claims workflow might use a multimodal model to read photos, a small model to extract fields, and a large model to write the decision letter.

Ensembles send the same request to several models and combine the results with custom aggregation logic:

AggregationHow it worksFits
Majority votePick the answer most models agree onClassification, yes or no decisions
Weighted voteWeight each model by measured accuracyModels with known different strengths
Judge modelA further model compares the candidates and picks or mergesFree text answers
Confidence thresholdAccept only if models agree, otherwise escalateHigh risk outputs

Ensembles raise quality and robustness but multiply cost and latency. Use them only where an error costs more than the extra calls.

Model selection frameworks

A model selection framework is a repeatable way to pick the model for each task. It compares candidates on quality for the task, latency, cost per request, context window and modality, using an evaluation set.

Amazon Bedrock model evaluation can score candidate models on your own prompt dataset. Store the chosen model ID per task in configuration, such as AWS AppConfig or Parameter Store, so you can change it without a code deploy.

Exam tip

The exam favors measured choices. "Choose the model with the best benchmark score" loses to "evaluate candidate models on a representative dataset and pick the smallest model that meets the quality bar".

Smaller models and cascading

The cheapest way to optimize a GenAI workload is often to use a smaller pre trained model for narrow tasks such as classification, extraction, summarization of short text or routing. Smaller models answer faster and cost less per token.

Model cascading tries the cheap model first and escalates only when needed:

Queryfirst trySmall modelconfidentAnswerSmall modellow confidence or failed checkLarge modelescalated answerAnswer
  • Routine queries stay on the small model, so most traffic is cheap.
  • A check decides when to escalate: a confidence signal, a validation rule, or a classifier that predicts difficulty.
  • Behind an API, the cascade is invisible to the client; this is API based cascading.

Routing options

Routing typeImplemented withPick it when
Static routingApplication code or configuration maps a task type to a model IDTask types are known and stable
Content based routingStep Functions Choice state or a classifier reads the request and sends it to a specialized FMRequests differ by topic, language or modality
Metric based routingRouter uses latency, error rate, cost or quality metrics from CloudWatchModel choice should adapt to live conditions
Gateway routingAPI Gateway with request transformations or a routing Lambda sets the target modelClients call one stable API while routing changes behind it
Amazon Bedrock Intelligent Prompt RoutingA prompt router predicts response quality and sends each prompt to a model within one model familyYou want managed quality and cost routing with no custom code
Clientone APIAPI Gatewayrequest transformationrouteStep FunctionsChoice statelegal textSpecialized FM AStep FunctionsChoice statecode questionSpecialized FM B

Static routing is simplest and fully predictable. Content based routing in Step Functions adds visible branches and retries. Metric based routing shifts traffic when a model slows down or starts throwing errors.

Inference profiles and Regions

Cross Region inference profiles in Amazon Bedrock spread on demand requests across several Regions in a geography, which raises available throughput. Routing logic should call the inference profile ID rather than a single Region model ID when the scenario mentions throttling at peak load.

How the exam asks about this

  • "Most customer questions are simple FAQs, but some need deep analysis; reduce cost without hurting quality" -> cascade from a small model to a large model, escalating on low confidence.
  • "Send contract questions to a legal tuned model and code questions to a code model, with visible branching and retries" -> Step Functions with a Choice state for content based routing.
  • "Clients must keep one endpoint while the team changes which model serves each request" -> API Gateway in front, with routing in a Lambda function or request transformation.

Common mistakes

Using the largest model for every call. It is slower and costlier, and for classification or extraction it often adds nothing.

Hard coding model IDs across services. Changing a model then needs many deploys. Keep the mapping in configuration or a central router.

Ensembling everything. Several models per request multiply cost. Reserve ensembles for high risk decisions.

Cascading without a reliable escalation check. If the small model is wrong but confident, the cascade never escalates. Validate the output, not only the confidence.

What to remember for the exam

  • Specialized FMs handle sub tasks they suit; ensembles combine outputs with custom aggregation such as voting or a judge model.
  • Model selection is measured on your own evaluation data for quality, latency and cost.
  • Smaller pre trained models cut cost and latency for narrow tasks.
  • Cascading sends routine queries to a small model and escalates hard ones.
  • Static routing lives in code or configuration; dynamic content based routing uses Step Functions; metric based routing reacts to live CloudWatch data.
  • Bedrock Intelligent Prompt Routing is the managed option for routing within a model family.
Watch out

Intelligent Prompt Routing chooses among models in the same family. If the scenario needs routing across providers or to a custom specialized model, build the router yourself.

Check yourself

  • What is model cascading?

    Sending each request to a cheap small model first and escalating to a larger model only when the answer fails a confidence or validation check.

  • Which service fits dynamic content based routing with auditable branches?

    AWS Step Functions, using a Choice state that sends each request to a specialized FM.

  • How do you let clients keep one API while model routing changes?

    Put API Gateway in front and do routing behind it with request transformations or a routing Lambda function.

Topic 3 of 8 in Implementation and Integration

Test yourself on Implementation and Integration

The full AWS AIP-C01 question bank has 827 questions, with timed practice exams and an explanation for every answer.

This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.