Guidelines for Responsible AI · Topic 2 of 6
Datasets, Bias and Variance
See how dataset quality, diversity and balance shape fairness, and how bias and variance in a model lead to underfitting, overfitting and unequal results.
- 4 min
- Good datasets are inclusive, diverse, curated and balanced.
- Biased or unbalanced data produces biased models.
- High bias in the technical sense leads to underfitting, and high variance leads to overfitting.
What it is
A model learns from its data, so unfair or poor data produces unfair or poor results. Responsible AI therefore starts with the dataset.
Bias and variance describe two kinds of model error. Both can harm accuracy, and both can affect some groups of people more than others.
How it works
Characteristics of a good dataset
| Characteristic | Meaning |
|---|---|
| Inclusivity | The data covers all the people and cases the system will serve |
| Diversity | The data includes variety in groups, settings, languages and conditions |
| Curated sources | Data comes from known, reviewed and trusted sources |
| Balance | No group or class is badly over or under represented |
| Accuracy | Labels and values are correct |
| Relevance | The data matches the task and the real world conditions |
| Legal use | The data can be used for this purpose and respects privacy |
A dataset can be large and still be a poor one. If a face recognition training set holds mostly one skin tone, the model will probably work worse for people with other skin tones.
Common sources of data bias
- Sampling bias: the data does not reflect the real population.
- Historical bias: the data records past unfair decisions, and the model repeats them.
- Label bias: human labelers apply labels in an unequal way.
- Measurement bias: the way data is collected differs between groups.
- Selection bias: only some cases make it into the dataset.
Bias and variance in the technical sense
In model terms, bias is error from a model that is too simple, and variance is error from a model that is too sensitive to the training data.
| Error type | Result | Effect |
|---|---|---|
| High bias | Underfitting | The model is inaccurate even on training data |
| High variance | Overfitting | The model works on training data but fails on new data |
You want a balance, so the model generalises to new data.
Effects on demographic groups
When training data is unbalanced, the effects are often uneven. A model may be accurate for the majority group and make more mistakes for a minority group. Overall accuracy can hide this, because the majority group dominates the average.
An overfit model may also memorise rare cases and perform badly on groups it saw little of. An underfit model may ignore patterns that matter for smaller groups.
Reducing bias in datasets
| Action | Effect |
|---|---|
| Collect more data for under represented groups | Improves balance |
| Review and clean labels | Reduces label bias |
| Use diverse labelers and reviewers | Reduces one sided judgement |
| Remove or limit sensitive features | Reduces direct use of protected traits, but check for proxies |
| Re balance or re sample the data | Improves group balance |
| Test performance by group | Reveals gaps early |
Removing a sensitive field does not always remove bias, because other fields can act as proxies. A postal code, for example, may stand in for an ethnic group.
Keeping data fresh
The world changes, so a dataset that was balanced last year may not be balanced now. Review the data and retrain when needed.
What to remember for the exam
- Good datasets are inclusive, diverse, curated and balanced.
- Biased or unbalanced data produces biased models.
- High bias in the technical sense leads to underfitting, and high variance leads to overfitting.
- Unbalanced data can lead to unequal accuracy across demographic groups.
- Overall accuracy can hide poor results for a small group.
- Sources of data bias include sampling, historical, label, measurement and selection bias.
- Removing a sensitive attribute may not remove bias because of proxy features.
- Balance the data, clean labels, and test per group to reduce bias.
When a question says a model works well overall but poorly for one group, think of unbalanced or non representative data and group level testing.
Common mistakes
Assuming more data fixes bias. More of the same skewed data repeats the same problem.
Trusting the overall accuracy figure. It can mask low accuracy for a minority group.
Confusing the two meanings of bias. Social bias is unfairness in outputs. Technical bias is oversimplified modelling that causes underfitting.
Dropping a sensitive column and calling it fixed. Proxy features can bring the same bias back.
Using old data unchanged. Populations and behaviour change.
Check yourself
What are two characteristics of a responsible dataset?
Examples are inclusivity, diversity, balance and curated, accurate sources.
Which problem does high variance cause?
Overfitting, where the model does well on training data but poorly on new data.
Why can overall accuracy be misleading for fairness?
A large majority group can hide poor results for a smaller group.
Topic 2 of 6 in Guidelines for Responsible AI
Test yourself on Guidelines for Responsible AI
The full AWS AIF-C01 question bank has 849 questions, with timed practice exams and an explanation for every answer.
This study guide is independent and is not affiliated with, endorsed by, or sponsored by AWS. Vendor names, certification names and exam codes are trademarks of their respective owners and are used for identification only.