Techniques to improve gen AI model output · Topic 2 of 5
Monitoring and Evaluating Gen AI
Google-recommended practices for monitoring and evaluating gen AI: model upgrades, KPIs, security patches, versioning, performance tracking, drift and Feature Store.
- 4 min
- Practices: automatic model upgrades, KPIs, security patches, versioning, performance tracking, drift monitoring, Feature Store.
- KPIs cover quality, user experience, operations, cost, business value and safety.
- Drift means the world changed; respond by updating grounding, prompts or the model.
What it is
A gen AI solution is never finished at launch. Models change, data changes and user needs change. The exam lists Google's recommended practices for continuous monitoring and evaluation: automatic model upgrades, key performance indicators (KPIs), security patches and updates, versioning, performance tracking, drift monitoring and Agent Platform Feature Store.
How it works
The practices
| Practice | What it means | Why it matters |
|---|---|---|
| Automatic model upgrades | Moving to newer model versions as the provider releases them, often through managed aliases or update policies | Gain quality, speed and cost improvements; but evaluate before switching critical workloads |
| Key performance indicators (KPIs) | Measurable targets tied to business and quality goals | Shows whether the solution delivers value |
| Security patches and updates | Keeping models, frameworks and dependencies up to date | Closes vulnerabilities attackers could exploit |
| Versioning | Recording which model, prompt and data version produced each result | Reproduce results, compare versions, roll back safely |
| Performance tracking | Measuring latency, error rates, cost and output quality over time | Catches degradation before users complain |
| Drift monitoring | Detecting when incoming data or behavior moves away from what the model was built for | Signals when to retrain, re-ground or re-tune |
| Agent Platform Feature Store | A managed store of ML features served consistently for training and prediction | Keeps features consistent and fresh, reducing training-serving mismatch |
Versioning is the safety net for upgrades. If a new model version performs worse, you need the old version, prompt and settings recorded to roll back.
KPIs for gen AI
| Area | Example KPIs |
|---|---|
| Quality | Accuracy of answers, groundedness (answers supported by sources), rate of hallucinations found in review |
| User experience | User satisfaction, task completion rate, adoption |
| Operations | Latency, availability, error rate |
| Cost | Cost per request or per resolved case |
| Business | Time saved, cases resolved without a human, revenue influenced |
| Safety | Rate of blocked unsafe content, policy violations |
Drift in plain words
Drift happens when the world changes after a model was built. Customers start asking about a new product; regulations change; the language customers use shifts. The model's inputs no longer match what it was designed for, and quality slips. Monitoring watches for these shifts so the team can update grounding data, prompts or the model.
Evaluation
Evaluation compares outputs against expectations, before launch and continuously after. Teams build a set of test prompts with good answers, score new model versions or prompt changes against it, and use human reviewers for nuanced quality. Agent Platform provides evaluation tools to compare models and prompts.
A worked scenario
A retailer's shopping assistant launches with KPIs for groundedness, task completion and cost per conversation. Three months later, performance tracking shows completion falling. Drift monitoring reveals many questions about a new loyalty program missing from the grounding data. The team adds the program documents, tests a newer model version against its evaluation set, records the new version of model and prompt, and keeps security patches current on its serving stack.
How the exam asks about this
- "Inputs have changed since launch and quality is falling." Drift monitoring.
- "Be able to roll back to the previous model and prompt." Versioning.
- "Measure whether the assistant delivers business value." KPIs.
- "Serve the same features for training and prediction." Agent Platform Feature Store.
What to remember for the exam
- Practices: automatic model upgrades, KPIs, security patches, versioning, performance tracking, drift monitoring, Feature Store.
- KPIs cover quality, user experience, operations, cost, business value and safety.
- Drift means the world changed; respond by updating grounding, prompts or the model.
- Evaluate new versions before relying on them.
An automatic upgrade can change behavior. For critical workloads, test the new version against your evaluation set first.
Common mistakes
Measuring only technical metrics. Leaders need business KPIs too.
Skipping versioning of prompts. A prompt change can alter results as much as a model change.
Assuming quality stays constant after launch. Drift makes monitoring necessary.
Check yourself
What is drift monitoring for?
Detecting when incoming data or behavior changes from what the model was built for, signaling the need to update it.
Why is versioning important?
It records which model, prompt and data produced results so teams can compare versions and roll back safely.
Name two categories of KPIs for a gen AI solution.
Any two of quality, user experience, operations, cost, business value and safety.
Topic 2 of 5 in Techniques to improve gen AI model output
Test yourself on Techniques to improve gen AI model output
The full Google Cloud Generative AI Leader question bank has 512 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by Google Cloud. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.