A decision tree is the one model you can explain to someone who has never seen machine learning, which is also why it overfits so readily when you let it grow unchecked.
How a decision tree makes a prediction
A tree asks a sequence of yes/no questions about your features, such as “is this value above a threshold?”, and each answer sends a record down one branch. The path ends at a leaf, and the leaf holds the prediction: the majority class for classification, or the average target value for regression. Reading the tree from the root to a leaf gives you a plain-language rule for that prediction.
How the splits are chosen
At every node, the algorithm tries candidate questions and keeps the one that makes the resulting groups as pure as possible. For classification, purity is usually measured with Gini impurity or entropy. For regression, it is typically the reduction in squared error. The process is greedy: each split is the best available at that moment, with no look-ahead to whether a different early split would have produced a better tree overall.
Why trees overfit
Left alone, a tree keeps splitting until every leaf is pure, which can mean close to one leaf per training record. That tree has memorized the training data, noise included. Its training score looks perfect while its validation score drops, the same gap described in how to detect overfitting.
Controlling tree size
There are two standard ways to keep a tree from memorizing. Pre-pruning stops growth early with limits on maximum depth, minimum samples per leaf, or minimum samples required to split a node. Post-pruning grows the full tree first, then cuts back branches that don’t improve validation performance. Either way, choose the limits with cross-validation, using the approaches in hyperparameter tuning methods, rather than picking numbers by eye.
What trees do well
Trees need no feature scaling, because a split depends on the ordering of values, not their magnitude. That is a contrast with principal component analysis, where scaling decides the result. Trees also capture nonlinear relationships and feature interactions without you specifying them, and a small tree is easy to explain to a non-technical reader.
Where a single tree falls short
A single tree is unstable: a small change in the training data can produce a noticeably different tree. Its splits are also axis-aligned, so a diagonal class boundary gets approximated with a staircase of cuts. That instability is the main reason ensemble methods exist. Techniques like gradient boosting combine many shallow trees so that no single tree’s quirks decide the answer.
A quick checklist
- Have you limited depth or leaf size, or let the tree grow until every leaf is pure?
- Did you choose those limits with cross-validation instead of the test set?
- Is the gap between training and validation scores small enough to trust?
- Do you need a model you can explain rule by rule, or is an ensemble acceptable?
- Have you compared it against a baseline model?
FAQ
Do decision trees need feature scaling?
No. Splits depend on how values are ordered, not how large they are, so standardizing features doesn’t change the tree.
What’s the difference between Gini impurity and entropy?
Both measure how mixed the classes are in a node. They usually produce very similar trees, and the choice rarely matters as much as tree depth and leaf size.
Can decision trees handle missing values?
It depends on the implementation. Some libraries handle missing values natively, while others require you to impute them first, so check the documentation for the one you use.
