Principal component analysis gets taught as a dimensionality reduction trick, which undersells what it does and hides the two ways it most often goes wrong: unscaled features, and fitting it before the train/test split.
What PCA actually computes
PCA finds new axes through your data. The first axis points in the direction where the data varies the most. The second points in the direction of the next most variation, with the constraint that it sits at a right angle to the first, and the pattern continues from there. Each axis is a principal component, and each one is a weighted blend of your original features. Keeping the first few components and dropping the rest is what gives you a smaller dataset.
Why variance is the thing being kept
PCA treats variance as a stand-in for information. Directions where the data barely changes are assumed to carry little signal, so dropping them should cost little. That assumption holds often enough to be useful, but it is still an assumption. A high-variance feature with no relationship to your target gets kept, and a low-variance feature that cleanly separates your classes can get thrown away.
Scale your features first
Because PCA chases variance, a feature measured in large units dominates the result just by having bigger numbers. Income in dollars will swamp age in years before the method learns anything. Standardizing each feature to zero mean and unit variance before running PCA is the usual fix. Skipping it is the most common way to get a result that looks plausible and means nothing.
How many components to keep
Each component accounts for a share of the total variance. Two common ways to choose are keeping enough components to cover a threshold you set, or plotting the variance per component and stopping where adding another stops adding much, the “elbow” in a scree plot. Neither gives a universally correct answer. The better test is to check downstream model performance at a few different component counts, since the right number depends on what the output is used for.
What you give up
Components are blends of the original features, so a model trained on them is harder to explain. “Component 2 went up” doesn’t map to anything a stakeholder can act on, while a statement about one named column does. If interpretability matters, that cost is real, and it is one reason feature selection, which keeps the original columns, is sometimes the better choice.
Fit it inside cross-validation
PCA learns from data, so it can leak. If you fit it on the full dataset before splitting, the test set has already shaped the components. Fit PCA on the training fold only, then apply that fitted transformation to the validation data. It is the same discipline behind catching data leakage everywhere else in a pipeline.
When PCA helps and when it doesn’t
PCA works best when features are strongly correlated, which is the situation behind multicollinearity in regression. It also suits plotting high-dimensional data in two or three dimensions and speeding up models on wide datasets. It is a weaker fit when the structure is nonlinear, because PCA only finds linear combinations, and when you have few features that are already mostly independent.
A quick checklist
- Did you standardize the features before fitting PCA?
- Is the PCA fitted on training data only, with the same transformation applied to validation and test data?
- Did you pick the number of components by checking model performance, not only a variance threshold?
- Can you live with components you can’t explain feature by feature?
- Have you compared the result against a baseline model trained on the original features?
FAQ
Does PCA make a model more accurate?
Not automatically. It can reduce noise and shorten training time, but it can also discard information the model needed. Compare against a baseline on the original features before assuming it helped.
Is PCA supervised or unsupervised?
Unsupervised. It never looks at the target variable, which is why it can drop directions that matter for prediction while keeping directions that don’t.
Can PCA be used on categorical features?
Standard PCA is built for continuous numeric features. Categorical columns need other techniques or careful encoding, and the resulting components are often hard to interpret.
