What is ridge regression?

A ridge regression is a least-squares regression that uses L2-regularization.

The regularized loss function on some dataset $\sets$ is thus:

$$\l_\text{ridge}(\sets, \vw) = \lmse(\sets, \vw) + \lambda\normtwo{\vw}^{2}$$

where $\lambda$ is an hyper-parameter that controls the importance of the regularization.

For a complete discussion about the effect of L2-regularization on the parameters of the model, check out our dedicated article: L2-regularization.

Common mistake: it is important to normalize the features before using regularization. Failure to do so will yield incoherent regularization behavior.

Analytical solution

Let $\trainset$ be the training-set and note $\mx$ and $\vy$ the corresponding design matrix and output vector.

We can compute the value of the parameter vector that minimizes the regularized loss using differentiation.

	$\grad\l(\trainset, \vw) = - \frac{1}{\card{\trainset}}\mx^{\top}(\vy - \mx\vw)$
	$\grad\lambda\normtwo{\vw} = 2\lambda\vw$

Setting all directional derivatives to $0$, we get:

$$\hat{\vw}_{\trainset} = [\mx^{\top}\mx + (2\lambda \card{\trainset}) \mi]^{-1}\mx^{\top}\vy$$