Every supervised model makes two kinds of mistakes: it can be too simple to capture the pattern, or so flexible that it learns the noise. The bias-variance trade-off is the name for how those two mistakes pull against each other.
What bias means
Bias is the error that comes from a model being too simple for the problem. A straight line fit to data that curves will be wrong in a consistent way, no matter how much training data you give it. High bias shows up as underfitting: poor scores on both the training data and new data.
What variance means
Variance is the error that comes from a model being too sensitive to the specific training sample. A very flexible model fits the quirks and noise of the data it saw, so a different training sample would give a noticeably different model. High variance shows up as overfitting: strong training scores and weaker scores on new data, the gap described in how to detect overfitting.
Why it is a trade-off
Making a model more flexible usually lowers its bias and raises its variance. Making it simpler does the opposite. Total prediction error combines the two with a third piece, irreducible noise in the data that no model can remove. The goal is not zero bias or zero variance, but the flexibility level where their combined error on unseen data is lowest.
Seeing it in familiar models
The pattern shows up everywhere. In k-nearest neighbors, a small k means low bias and high variance, while a large k means the reverse. A single deep decision tree has high variance, which is the weakness that random forests address by averaging many trees. Regularization, such as L2 regularization, deliberately accepts a little extra bias to reduce variance.
How to find the right balance
Compare training and validation scores as you change model complexity. If both are poor, suspect high bias and try a more flexible model, better features, or less regularization. If training is strong and validation is weak, suspect high variance and try more data, a simpler model, stronger regularization, or an ensemble. Judge all of this with a held-out score from cross-validation, using the approaches in cross-validation for small datasets, since training scores alone will mislead you.
A caution about the textbook picture
The classic U-shaped curve is a useful mental model, but it is not a law. Very large modern models, such as deep neural networks, can fit training data almost perfectly and still generalize well, so the simple rule that more flexibility always means more variance does not always hold. For the tabular models most projects use, though, the trade-off remains a reliable guide, and ensemble methods are one of the practical tools for managing it.
A quick checklist
- Are both your training and validation scores poor, or only the validation score?
- Have you plotted scores against model complexity, not just checked one setting?
- Did you try more data or stronger regularization before adding complexity?
- Is your validation score from cross-validation instead of a single split?
- Did you compare against a baseline model?
FAQ
What is the bias-variance trade-off in simple terms?
Simple models miss real patterns (bias), while flexible models chase noise (variance). The trade-off is finding the complexity where the combined error on new data is lowest.
How do I know if my model has high bias or high variance?
If it scores poorly on both training and validation data, bias is the likely problem. If it scores well on training data but poorly on validation data, variance is the likely problem.
Does more data fix high bias?
Not usually. More data mainly reduces variance. A model that is too simple stays too simple, so high bias needs a more flexible model or better features.
