The k-nearest neighbors algorithm makes a prediction by looking at the training points closest to a new one and copying what they say. There is no model to fit in the usual sense, which makes it easy to understand and easy to misuse.
How it makes a prediction
To predict for a new record, the algorithm measures its distance to every training record and takes the k closest. For classification, it returns the majority class among them. For regression, it returns their average value. Nothing is learned in advance, so the method is often called lazy: the work happens at prediction time.
Choosing k
A small k, such as 1, follows the training data very closely and is sensitive to noise. A large k smooths the prediction but can blur real structure, and with a very large k the result drifts toward the overall majority. This is the bias-variance trade-off in its plainest form. Choose k with cross-validation, using the approaches in hyperparameter tuning methods, rather than a fixed rule of thumb.
Scaling is not optional
Because the method relies on distance, a feature with large numbers dominates the result. Income in dollars will swamp age in years. Standardize or normalize features first, as explained in normalization vs standardization, and fit the scaler on training data only to avoid data leakage.
Many features weaken distance
As the number of features grows, points tend to become almost equally far from each other, so “nearest” carries less meaning. Reducing the feature count first, for example with principal component analysis, often helps.
Speed and memory
The model stores the whole training set and compares every prediction against it, so prediction gets slower as the data grows. Indexing structures can speed up the search, but k-nearest neighbors is usually a poor fit for very large datasets or low-latency serving.
When it fits
It suits small to medium datasets with few, well-scaled features and local structure, and it makes a useful baseline model. It is a weak choice when features are many, scales differ widely, or predictions must be fast.
A quick checklist
- Did you scale the features, fitting the scaler on training data only?
- Did you choose k with cross-validation?
- Do you have so many features that distance stops being meaningful?
- Is prediction speed acceptable at your data size?
- Did you compare it against a baseline?
FAQ
Does k-nearest neighbors need feature scaling?
Yes. It relies on distance, so features on larger scales dominate unless you standardize or normalize them.
What is a good value for k?
There is no universal value. Pick it by cross-validation, and prefer an odd k for two-class problems to avoid ties.
Is k-nearest neighbors used for regression too?
Yes. Instead of a majority vote, it averages the target values of the k nearest points.
