Fundamentals of GenAI · Topic 5 of 7
Model Choice and Business Metrics
Choose a GenAI model by performance, cost, speed and compliance needs, and measure business value with conversion, efficiency and satisfaction.
- 4 min
- Choose a model by modality, performance, size, latency, cost, context length, languages, customisation and compliance.
- Bigger is not always better: a smaller model can be cheaper, faster and good enough.
- Amazon Bedrock lets you try models from several providers through one service.
What it is
Once you know GenAI fits a problem, you must pick a suitable model and decide how to judge success. The best model is the one that meets the business need at an acceptable cost, not the biggest or newest one.
Success is measured with business metrics, which show value to the organisation. Technical scores alone do not prove that a solution works.
How it works
Factors in choosing a model
| Factor | Question to ask |
|---|---|
| Model type and modality | Does it handle the data you have, such as text, image or both |
| Performance | Is the quality good enough for the task |
| Capability | Can it do the specific task, such as summarising or writing code |
| Size and complexity | Larger models are often more capable, but slower and costlier |
| Latency | Is the response fast enough for the user |
| Cost | What are the charges per request or per token |
| Input and output length | Does the context window fit your documents |
| Language support | Does it support the languages your users need |
| Customisation | Can you adapt it to your data if needed |
| Compliance and safety | Does it meet your legal, security and data rules |
| Availability | Is it offered in the Region you need |
A smaller, cheaper model is often the right choice for simple tasks such as classifying short messages. Save the larger model for complex reasoning or long documents.
Trying models side by side
Amazon Bedrock lets you call models from several providers through one service, so you can test different models on the same prompts. Comparing outputs and costs on your own examples is a better guide than general rankings.
Business metrics
Pick metrics that match the goal of the project.
| Metric | What it shows |
|---|---|
| User satisfaction | How happy people are with the answers |
| Conversion rate | How many users take a wanted action, such as a purchase |
| Efficiency | Time saved on a task or fewer steps per case |
| Average revenue per user (ARPU) | Revenue earned per user, which may rise with better personalisation |
| Cross domain performance | How well the solution works across different topics or business areas |
| Cost per interaction | What each answer costs to produce |
| Resolution rate | Share of requests solved without a human |
| Error or escalation rate | How often a person must step in |
Running a pilot
Start with a small pilot that uses real tasks. Measure the chosen metrics before and after, and gather user feedback. If the gains do not justify the cost, change the model, improve the prompts or stop.
Balancing the trade offs
The factors pull against each other.
| Trade off | Effect |
|---|---|
| Higher quality | Often higher cost and slower responses |
| Faster response | May mean a smaller model with simpler answers |
| More customisation | More effort and cost, but a better fit |
| Strict compliance | May limit the models or Regions you can use |
State the priority first. If speed matters most, choose for latency. If accuracy on hard tasks matters most, accept a larger model.
Repeating the review
Models and prices change. Review the choice from time to time, since a newer or cheaper model might do the job better.
What to remember for the exam
- Choose a model by modality, performance, size, latency, cost, context length, languages, customisation and compliance.
- Bigger is not always better: a smaller model can be cheaper, faster and good enough.
- Amazon Bedrock lets you try models from several providers through one service.
- Business metrics include user satisfaction, conversion rate, efficiency, ARPU and cross domain performance.
- Evaluate on your own data and tasks, not only on public rankings.
- Pilot first, and measure the metrics before and after.
- Trade offs exist between quality, cost, speed and customisation.
- Compliance needs can limit which models and Regions you can use.
If a question mentions revenue per customer going up after personalised content, the metric is ARPU. If it mentions users completing a purchase, it is conversion rate.
Common mistakes
Picking the largest model by default. It costs more and responds more slowly, and it may not be needed.
Judging success only by technical scores. The business needs a metric tied to its goal, such as time saved.
Ignoring context length. A model that cannot fit your documents will not work for the task.
Skipping a pilot. Without real tests you cannot tell if the model fits your use case.
Forgetting compliance. A model that breaks data rules is not usable, however good the output.
Check yourself
Name three factors to weigh when choosing a model.
Examples are performance, cost, latency, context length, language support and compliance.
Which metric shows how many users complete a wanted action such as buying?
Conversion rate.
Why start with a pilot?
It tests the model on real tasks and measures business results before a full rollout.
Topic 5 of 7 in Fundamentals of GenAI
Test yourself on Fundamentals of GenAI
The full AWS AIF-C01 question bank has 849 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.