Feature selection in machine learning means deliberately choosing which input variables a model actually uses, instead of feeding it every column available. More features isn’t automatically better — irrelevant or redundant ones add noise, slow training, and can hurt generalization.
Feature selection in machine learning: why more isn’t better
Every added feature is a chance for the model to find a spurious pattern that doesn’t hold up outside the training data. Feature selection in machine learning exists specifically to filter down to the inputs that carry real signal, which tends to produce a simpler, more generalizable, and more interpretable model than throwing everything at it.
Three main approaches
- Filter methods. Rank features by a statistical measure (correlation with the target, mutual information) independent of any model, then keep the top ones. Fast, but ignores feature interactions.
- Wrapper methods. Test subsets of features by actually training and evaluating a model on each subset (e.g. recursive feature elimination). More accurate, but computationally expensive.
- Embedded methods. Let the model itself select features during training — L1 regularization (Lasso) zeroing out coefficients, or tree-based models exposing feature importance scores directly.
A real example: explicit, chosen features over raw pixels
The visual quality inspection project deliberately extracts specific visual features — edges, texture, color distribution — rather than feeding raw pixel arrays into an opaque network. That’s feature selection in machine learning applied at the input-design stage: choosing interpretable features on purpose, not just accepting whatever the raw data provides.
A real example: explaining which features actually mattered
The bike-share demand forecasting project uses SHAP to show which factors — weather, time of day, season — actually drive each prediction. That’s a useful complement to feature selection: even after choosing which features to include, SHAP reveals which of them are pulling real weight, which can inform a second, more targeted round of feature selection in machine learning workflows.
The risk of doing it wrong: leakage through selection
Selecting features using the full dataset — test set included — before splitting is a specific form of data leakage. Feature selection has to be fit on training data only, exactly like any other preprocessing step, or the resulting evaluation score is inflated.
A quick checklist
- Is every included feature something you can justify, or did it just happen to be in the dataset?
- Was feature selection performed using training data only, not the full dataset?
- Have you checked for redundant features that are highly correlated with each other?
- Would a simpler feature set perform nearly as well — and if so, is the extra complexity worth keeping?
FAQ
Is feature selection still necessary with tree-based models that handle irrelevant features well?
Less critical than with linear models, but still valuable for interpretability, training speed, and removing features that could introduce spurious splits.
What’s the difference between feature selection and feature engineering?
Feature selection chooses among existing inputs. Feature engineering creates new inputs from existing ones (ratios, aggregates, encodings). They’re complementary steps, usually done in that order.
Can feature selection hurt model performance?
Yes, if done too aggressively or with leakage — dropping a feature that genuinely carries signal, or selecting based on the full dataset, both distort the result.
