Applications of Foundation Models · Topic 6 of 7
FM Training and Fine Tuning
Understand pre training, fine tuning, continued pre training, distillation and RLHF for foundation models, plus how to prepare data for fine tuning.
- 4 min
- Includes a diagram
- Pre training builds the base model from huge broad data and is done by the provider.
- Fine tuning adapts a pre trained model with a smaller, focused dataset.
- Instruction tuning uses labeled prompt and response pairs.
What it is
A foundation model passes through several kinds of training. Each has a different goal, different data and a different cost.
Knowing the vocabulary lets you pick the right method for a scenario, and explain what is happening when you customise a model.
How it works
Pre training
Pre training builds the base model from very large collections of data. For an LLM, the model learns to predict the next token over huge amounts of text. This requires large computing resources and time, so providers do it once, and customers reuse the result.
Fine tuning
Fine tuning continues training a pre trained model on a smaller, focused dataset, so it performs better for a task or domain. It is a form of transfer learning: knowledge from the broad training is carried over and adapted.
| Method | Data | Purpose |
|---|---|---|
| Instruction tuning | Prompt and response pairs (labeled) | Teach the model to follow instructions or a task format |
| Domain adaptation | Domain specific examples | Fit a field such as law or medicine |
| Continued pre training | Large amounts of unlabeled domain text | Extend general knowledge into a specialised area |
On Amazon Bedrock, you can fine tune supported models with labeled data stored in Amazon S3, and you can run continued pre training with unlabeled data. The result is a private custom model for your account. Amazon SageMaker AI offers more control if you need it.
Reinforcement learning from human feedback
Reinforcement learning from human feedback (RLHF) improves a model using human judgement. People rank or rate model outputs, a reward signal is built from those ratings, and the model is adjusted to produce answers people prefer. RLHF helps make models more helpful and safer.
Distillation
Distillation trains a smaller student model to copy the behaviour of a larger teacher model. The goal is a model that is cheaper and faster to run while keeping much of the quality for a given task.
Choosing a method
| Goal | Method |
|---|---|
| Follow a specific task format | Instruction tuning with labeled pairs |
| Learn the language of a specialised field | Continued pre training on unlabeled text |
| Match human preferences | RLHF |
| Cut cost and delay of a large model | Distillation |
| Add fresh facts | RAG rather than training |
Preparing data for fine tuning
Data quality decides the result. Good preparation covers these points.
- Curation: Collect accurate, relevant examples and remove errors and duplicates.
- Labeling: Examples must have correct labels or ideal responses. Amazon SageMaker Ground Truth can help with labeling.
- Representativeness: The data should cover the range of cases and groups the model will meet, to limit bias.
- Size: Enough examples are needed, but quality matters more than raw quantity.
- Governance: Check ownership, licences and privacy rules, and remove sensitive data you should not use.
- Format: Put the data in the format the training tool expects.
Risks of fine tuning
Training on poor or biased data teaches the model those flaws. The model can also lose some of its general skill if it is over tuned on a narrow set, and it may memorise private data. Evaluate the tuned model before use.
What to remember for the exam
- Pre training builds the base model from huge broad data and is done by the provider.
- Fine tuning adapts a pre trained model with a smaller, focused dataset.
- Instruction tuning uses labeled prompt and response pairs.
- Continued pre training uses large amounts of unlabeled domain data.
- RLHF uses human ratings to align the model with preferences.
- Distillation trains a smaller student to imitate a larger teacher.
- Transfer learning means reusing knowledge from one training stage in another.
- Data for fine tuning must be curated, correctly labeled, representative and governed.
Labeled prompt and response pairs point to fine tuning. Large unlabeled domain text points to continued pre training.
Common mistakes
Using fine tuning to add changing facts. It would need retraining each time. RAG is better.
Believing more data is always better. Poor quality data harms the model.
Mixing up distillation and fine tuning. Distillation shrinks a model by copying a larger one. Fine tuning adapts a model to a task.
Forgetting data governance. Training data can bring legal and privacy problems.
Skipping evaluation. A tuned model must be tested before it replaces the base model.
Check yourself
What data does instruction tuning use?
Labeled prompt and response pairs.
What does distillation produce?
A smaller student model that imitates a larger teacher model at lower cost.
Why must fine tuning data be representative?
So the model works well for all the cases and groups it will meet and does not learn bias.
Topic 6 of 7 in Applications of Foundation Models
Test yourself on Applications of Foundation Models
The full AWS AIF-C01 question bank has 849 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.