Regularization is a way of stopping a model from fitting the noise in its training data. It adds a penalty to the training objective that discourages large or numerous weights, trading a little accuracy on the training set for better behavior on new data.
The idea in one line
A model is normally trained to minimize its error on the training data. Regularization changes the objective to the error plus a penalty on the size of the weights, scaled by a strength setting, usually written as lambda. A larger lambda pulls the weights harder toward zero and gives a simpler model. This is a direct way to manage the overfitting side of the bias-variance trade-off, and you can spot the need for it using the signs in how to detect overfitting.
L2 regularization
L2 regularization, used in ridge regression, adds the sum of the squared weights to the objective. It shrinks all weights smoothly toward zero but rarely makes any of them exactly zero. When several features are correlated, it tends to share the weight among them, which makes it a useful remedy for the instability described in multicollinearity in regression. The details and an example are in the L2 regularization formula explained.
L1 regularization
L1 regularization, used in lasso regression, adds the sum of the absolute values of the weights. Its key property is that it can push some weights to exactly zero, which removes those features from the model and acts as built-in feature selection. When features are strongly correlated, it tends to keep one and drop the others, and which one it keeps can change with small changes in the data. The trade-offs between the two are compared in L1 vs L2 regularization.
L0 regularization
The L0 norm counts the number of non-zero weights, so an L0 penalty directly charges the model for each feature it uses. That matches the goal of using as few features as possible, but the resulting problem is not smooth or convex and is hard to solve exactly, so in practice it is approximated by L1 or by step-by-step feature selection. L0 norm and L0 regularization covers the definition and why it is rarely used directly.
Elastic net: using both
Elastic net combines the L1 and L2 penalties, with a setting that controls the mix. It keeps some of L1’s ability to zero out weights while borrowing L2’s stability with correlated features. It is a reasonable choice when you want feature selection but have many related features.
Comparing the three penalties
| Penalty | What it charges | Typical effect |
|---|---|---|
| L2 | Sum of squared weights | Shrinks weights smoothly, rarely to zero |
| L1 | Sum of absolute weights | Can set weights exactly to zero |
| L0 | Count of non-zero weights | Directly limits the number of features, hard to optimize |
How to use it in practice
Standardize your features first, because the penalty depends on the size of the weights and features on large scales would otherwise be penalized unevenly, as explained in normalization vs standardization. Then choose the strength with cross-validation, using the methods in hyperparameter tuning methods. Libraries differ in how they express the strength: scikit-learn’s logistic regression, for example, uses a setting called C that is the inverse of the regularization strength, and it applies L2 regularization by default, as noted in logistic regression explained.
A quick checklist
- Did you standardize the features before applying a penalty?
- Did you choose the strength by cross-validation, not by eye?
- Do you want shrinkage (L2) or feature selection (L1)?
- Are your features strongly correlated, which favors L2 or elastic net?
- Did you compare the regularized model against an unregularized baseline?
FAQ
What is the difference between L1 and L2 regularization?
L1 penalizes the sum of absolute weights and can set some weights exactly to zero. L2 penalizes the sum of squared weights and shrinks them toward zero without usually eliminating them.
Which is better, L1 or L2?
Neither is better in every case. Use L1 when you want a sparse model with fewer features, and L2 when you want stable weights, especially with correlated features. Compare them with cross-validation.
What does lambda do in regularization?
Lambda sets how strongly the penalty is applied. Larger values give simpler models with smaller weights, and smaller values let the model fit the training data more closely.
