Operational Efficiency and Optimization for GenAI Applications · Topic 2 of 7

Model Selection and Inference Parameter Tuning

Choose the cheapest FM that meets quality with tiered routing and Bedrock intelligent prompt routing, then tune temperature, top-p and top-k and A/B test.

  • 6 min
In a nutshell
  • The best model is the cheapest one that meets the quality bar on your own evaluation set.
  • Bedrock Evaluations supports programmatic, LLM-as-a-judge and human evaluation for comparing models.
  • Intelligent prompt routing: same family, fallback model, response quality difference, English-optimized.

What it is

Model choice is the biggest cost lever in a GenAI application. Prices per token differ widely between small and large models, and a small model is often good enough for simple requests.

A cost-effective framework picks the cheapest model that meets the quality bar for each kind of request. Parameter tuning and A/B testing then get the best result out of the chosen model and prove that a change really helped.

Requestclassify complexityRouterrules or prompt routersimpleSmall modellow cost per tokenRoutercomplexLarge modelhigher qualityanswerUser

How it works

Cost-capability tradeoff and price-to-performance

Compare models on the same task data, not on general benchmarks.

  1. Build an evaluation set from real prompts with reference answers.
  2. Run each candidate model through Amazon Bedrock Evaluations (programmatic metrics, LLM-as-a-judge or human review).
  3. Record quality score, cost per request (tokens times price) and latency.
  4. Choose the model with the best quality per dollar that still meets the minimum score.

Price-to-performance is a ratio: quality or task success divided by cost per request. A model that is slightly better but several times more expensive usually loses unless the scenario says quality is the top priority.

Tiered FM usage

ApproachHow it routesPick it when
Rule-based tieringCode routes by intent, length or customer tierRequest types are known and stable
Classifier tieringA small model labels complexity first, then routesMixed traffic with no simple rule
Amazon Bedrock intelligent prompt routingA prompt router predicts response quality per request and picks a modelYou want managed routing between two models of the same family
CascadeTry the small model, escalate if a check failsA cheap validator can detect a bad answer

Intelligent prompt routing gives one serverless endpoint for a prompt router. You choose two models in the same family, a fallback model, and a response quality difference: the router switches away from the fallback only when it predicts the other model is better by more than that margin. The response tells you which model served the request.

Know its limits: routing stays within one model family, it is optimized for English prompts, and it does not learn from your application's own performance data.

Other efficient inference patterns

  • Model distillation: Amazon Bedrock Model Distillation trains a smaller student model on responses from a larger teacher, so you get near-teacher quality for a narrow task at lower cost.
  • Flex service tier: non-urgent work, such as evaluations or summarization, can run on the Flex tier at a discount in exchange for longer processing times.
  • Batch inference: large offline jobs run asynchronously at a lower price than on-demand for supported models.
  • Shorter outputs: structured output and maxTokens limits cut the most expensive tokens.

Inference parameters

ParameterEffectTypical setting
temperatureRandomness of token choiceLow (near 0) for extraction, classification, code, factual Q and A; higher for creative writing
topPSamples only from the smallest set of tokens whose probability adds up to PLower to keep answers focused
topKSamples only from the K most likely tokensLower for predictable output
maxTokensMaximum output lengthJust above the expected answer size
stopSequencesStrings that end generationFormat markers

In the Converse API, inferenceConfig holds maxTokens, temperature, topP and stopSequences. Model-specific parameters such as top_k for some models go in additionalModelRequestFields. Parameter names and ranges differ by model, so check the model's request format.

Change temperature or topP, not both at once, so you can see which one caused a difference.

Exam tip

"Responses must be consistent and repeatable" points to low temperature. "Responses feel repetitive and bland" points to raising temperature or topP for that use case only.

A/B testing improvements

A change to a model, prompt or parameter is only an improvement if it wins on measured results.

Userssplit trafficVariant Acurrent prompt and modellog outcomeMetricsquality, cost, latencyUserssplit trafficVariant Bcandidatelog outcomeMetricscompareDecisionpromote or roll back
  • For Bedrock, split traffic in the application or with feature flags in AWS AppConfig, and tag each request with its variant.
  • Amazon Bedrock Prompt management keeps versioned prompts, so each variant points to a fixed prompt version.
  • On SageMaker AI endpoints, production variants with traffic weights split requests between models on one endpoint.
  • Measure task success (thumbs up, resolution rate), cost per request and latency, and run offline evaluation on the same set before going live.

How the exam asks about this

  • "Most queries are simple FAQs but some need deep reasoning; reduce cost with minimal custom code" -> Bedrock intelligent prompt routing or tiered routing to a smaller model for simple queries.
  • "Extraction output changes between identical runs" -> lower temperature (and topP or topK).
  • "Prove a new prompt improves resolution rate without risking all users" -> A/B test a prompt version on a small share of traffic and compare metrics.

Common mistakes

Picking the largest model by default. It raises cost and latency for every request, including the many that a small model answers well.

Choosing a model from public leaderboards. Leaderboards do not reflect your data. Evaluate on your own prompts.

Expecting the prompt router to route across providers. A prompt router chooses between models of one family. Cross-family routing needs your own logic.

Declaring a winner from a handful of examples. Small samples mislead; use enough traffic and an offline evaluation set.

What to remember for the exam

  • The best model is the cheapest one that meets the quality bar on your own evaluation set.
  • Bedrock Evaluations supports programmatic, LLM-as-a-judge and human evaluation for comparing models.
  • Intelligent prompt routing: same family, fallback model, response quality difference, English-optimized.
  • Distillation, Flex tier and batch inference are efficient inference patterns for narrow or non-urgent work.
  • Low temperature, topP and topK give focused, repeatable output; higher values give variety.
  • A/B testing needs versioned variants, a traffic split and measured outcomes.

Check yourself

  • What does the response quality difference control in a Bedrock prompt router?

    How much better the other model must be predicted to be before the router moves a request away from the fallback model.

  • Which parameter change best fixes inconsistent answers to the same extraction prompt?

    Lower the temperature, and if needed topP or topK, so token sampling becomes more deterministic.

  • How do you measure price-to-performance for two candidate models?

    Run both on the same evaluation set, score quality, compute cost per request from token counts, and compare quality per dollar.

Topic 2 of 7 in Operational Efficiency and Optimization for GenAI Applications

Test yourself on Operational Efficiency and Optimization for GenAI Applications

The full AWS AIP-C01 question bank has 827 questions, with timed practice exams and an explanation for every answer.

This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.