Chapter 5 of 5 · Microsoft AI-300

Optimize Generative AI Systems and Model Performance

Tuning RAG retrieval, embeddings, hybrid search, A/B testing, and fine-tuning with synthetic data through to production.

  • 3topics
  • 16 minfast track
  • 1diagrams and exhibits
  • 10–15%of the exam
Start with Retrieval Tuning and Embedding Models

Topics in this chapter

Take them in order, or jump to the one you need.

  1. 1Retrieval Tuning and Embedding ModelsAI-300 optimization: tune chunk size, overlap, similarity thresholds and retrieval strategy for RAG, and select or fine-tune…5 min
  2. 2Hybrid Search and RAG Evaluation with A/B TestingAI-300 optimization: combine keyword and vector search with semantic ranking in Azure AI Search, then evaluate RAG with relevance…Diagram5 min
  3. 3Fine-Tuning Methods, Synthetic Data and Production DeploymentAI-300 optimization: choose fine-tuning methods such as SFT and DPO, build synthetic data, monitor tuned models and deploy them…6 min

About this chapter

What this chapter covers

Optimize Generative AI Systems and Model Performance is 10 to 15% of the Microsoft AI-300 exam. It has two skill groups: optimizing retrieval-augmented generation (RAG) performance, and advanced fine-tuning and model customization.

Once a system works, you make it better. For RAG you tune chunking, similarity thresholds, embeddings and hybrid search, and prove improvements with metrics and A/B tests. For fine-tuning you choose a method, prepare real and synthetic data, and carry the model into production.

Questions describe a quality problem and ask which lever to pull first, for example wrong chunks retrieved or a model that ignores a required output format.

How the topics fit together

Tune retrievalchunks, thresholdsEmbeddingsselect or fine tuneHybrid searchkeyword plus vectorMeasuremetrics, A/B testsFine tuneSFT, DPO, synthetic dataProduction

Always measure before and after a change. Prefer the cheapest fix first (prompt, retrieval) and fine-tune only when the behaviour you need cannot be reached another way.

Exam tip

Change one variable at a time and compare on the same test set. Questions often hide a trap where two things were changed at once.

How to use these pages

Each topic page has the same shape, so you can skim for what you need.

  • What it is: the idea in plain words.
  • How it works: the mechanism, with tables and examples.
  • What to remember for the exam: the facts to memorise and how questions are phrased.
  • Common mistakes: the classic traps.
  • Check yourself: three questions to test your understanding.

Ready for more practice?

The full Microsoft AI-300 question bank has 650 questions, with timed practice exams and an explanation for every answer.

SubscribeTry Sample Exam

This study guide is independent and is not affiliated with, endorsed by, or sponsored by Microsoft. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.