A random forest is a collection of decision trees whose predictions are combined. It fixes the biggest weakness of a single tree, instability, by averaging out the mistakes that individual trees make.
How a random forest is built
The algorithm trains many trees, each on a different random sample of the training data, drawn with replacement. This is called bootstrap sampling, and combining models trained this way is known as bagging. On top of that, each split inside a tree considers only a random subset of the features, not all of them. For classification, the forest returns the majority vote of its trees. For regression, it returns their average.
Why many trees beat one
A single decision tree can change a lot when the training data changes slightly. Each tree in a forest sees different data and different features, so their errors are less correlated, and averaging them cancels much of the noise. The randomness over features matters as much as the randomness over rows: without it, strong features would dominate every tree and the trees would look alike. This is one of the cases covered in ensemble methods.
Out-of-bag error
Because each tree is trained on a bootstrap sample, some training rows are left out of it. A forest can predict those rows using only the trees that never saw them, which gives an error estimate without a separate validation set. It is a useful quick check, though a proper cross-validated evaluation is still the more reliable test.
The settings that matter
Four settings do most of the work: the number of trees, the maximum depth of each tree, the number of features considered at each split, and the minimum number of samples in a leaf. More trees usually help until the gains flatten, at a cost in training and prediction time. Depth and leaf size control how closely each tree fits the training data. Tune them with the approaches in hyperparameter tuning methods.
Feature importance: use it carefully
Forests report how much each feature contributed to the splits, which is a handy first look at your data. The default impurity-based measure can favor features with many distinct values, so treat it as a hint, not a conclusion. Permutation importance, which measures how much the score drops when a feature is shuffled, is usually a more reliable check.
Where random forests fall short
A forest is harder to explain than a single tree, since the answer comes from hundreds of trees. For regression, it cannot predict beyond the range of target values it saw in training, so it extrapolates poorly. It also does not remove overfitting entirely, especially with noisy data and very deep trees. Boosting methods such as gradient boosting often score higher on tabular data, at the cost of more careful tuning.
When to use it
A random forest is a strong default for tabular data. It needs little preparation: unlike the models discussed in normalization vs standardization, it does not need feature scaling. It also handles nonlinear relationships and interactions without extra work, and it makes a solid baseline model before you try something more complex.
A quick checklist
- Did you set the number of trees high enough that results stop changing?
- Did you limit depth or leaf size, or let trees grow until every leaf is pure?
- Are you treating impurity-based feature importance as a hint, not proof?
- Do you need a model you can explain rule by rule?
- Did you compare it against gradient boosting and a simple baseline?
FAQ
How is a random forest different from a decision tree?
A decision tree is a single model. A random forest trains many trees on random samples of rows and features, then combines their predictions, which makes it far more stable.
Does a random forest need feature scaling?
No. Trees split on the ordering of values, so scaling does not change the result.
Can random forests overfit?
Yes, though less easily than a single tree. Noisy data and very deep trees can still lead to overfitting, so check the gap between training and validation scores.
