The bias-variance-noise decomposition

The MSE loss is attractive because the expected error in prediction can be explained by the bias-variance of the model and the variance of the noise. This is called the bias-variance-noise decomposition. In this article, we will introduce this decomposition using the tools of probability theory.

In short, when $\ry = \ff(\rvx) + \epsilon$, the bias-variance-noise decomposition is:

$$\expectation_{\sets, \epsilon}[(\,\hat{\ry}_{\sets} - \ry\,)^2] = \var(\hat{\ff}(\rvx)) + \bias(\hat{\ff}(\rvx))^2 + var(\epsilon)$$

Notations

Let $(\rvx, \ry)$ be a pair of random variables on $\realvset{\sd} \times \realset$.

Assume there exists a $0$-mean random noise $\epsilon$ and a function $\ff$ such that:

$$\sy = \ff(\rvx) + \epsilon$$

The goal of a regression is to use a sample $\trainset$ to estimate this function:

$$\hat{\ff}_{\trainset} \approx \ff$$

For instance, in a linear regression the function $\ff$ is a linear function with parameter $\vw$:

$$\expectation[\ry \mid \rvx] = \vw\cdot\rvx$$

And the regression aims at estimating $\vw$ from the training-set:

$$\hat{\vw}_{\trainset} \approx \vw$$

Once the function $\ff_{\trainset}$ is estimated, we can measure the error between a predictions $\hat{\sy}_{\trainset} = \ff_{\trainset}(\vx)$ and the true value $\sy$:

$$\lmse(\hat{\sy}_{\trainset}, \sy) = (\hat{\sy}_{\trainset} - \sy)^2$$

The expected error in prediction is:

$$\begin{align*} & \expectation_{\sets, \epsilon}[(\,\hat{\sy}_{\sets} - \sy\,)^2] \\ = & \expectation_{\sets, \epsilon}[(\,\hat{\ff}_{\sets}(\rvx) - (\ff(\rvx) + \epsilon)\,)^2] \\ = & \expectation_{\sets, \epsilon}[(\,\underbrace{\hat{\ff}_{\sets}(\rvx) - \ff(\rvx)}_{A} - \epsilon\,)^2] \end{align*}$$

Define $A$ as a shorthand.

$$\begin{align*} & \expectation_{\sets, \epsilon}[(\,A - \epsilon\,)^2] \\ = & \expectation_{\sets, \epsilon}[A^2] - 2\expectation_{\sets, \epsilon}[A\epsilon] + \expectation_{\sets, \epsilon}[\epsilon^2] \\ \end{align*}$$

$A$ does not depend on $\epsilon$ and $\epsilon$ does not depend on $\sets$, so:

$$\begin{align*} & \expectation_{\sets, \epsilon}[A^2] - 2\expectation_{\sets, \epsilon}[A\epsilon] + \expectation_{\sets, \epsilon}[\epsilon^2] \\ = & \expectation_{\sets}[A^2] - 2\expectation_{\epsilon}[\epsilon]\expectation_{\sets}[A] + \expectation_{\epsilon}[\epsilon^2] \\ \end{align*}$$

Recall that $\expectation[\epsilon] = 0$:

$$\begin{align*} & \expectation_{\sets}[A^2] - 2\expectation_{\epsilon}[\epsilon]\expectation_{\sets}[A] + \expectation_{\epsilon}[\epsilon^2] \\ = & \expectation_{\sets}[A^2] + \expectation_{\epsilon}[\epsilon^2] \\ \end{align*}$$

Since $\epsilon$ is a $0$-mean noise we have:

$$\var(\epsilon) = \expectation[\epsilon^2] - \expectation[\epsilon]^2 = \expectation[\epsilon^2]$$

Hence:

$$\begin{align*} & \expectation_{\sets}[A^2] + \expectation_{\epsilon}[\epsilon^2] \\ = & \expectation_{\sets}[A^2] + \var(\epsilon) \\ \end{align*}$$

Finally, the term $\expectation_{\sets}[A^2]$ is exactly the error in estimation between $\ff$ and $\hat{\ff}$. We can exprees it using the bias-variance decomposition:

$$\begin{align*} & \expectation_{\sets}[A^2] + \var(\epsilon) \\ = & \bias(\hat{\ff})^2 + \var(\hat{\ff}) + \var(\epsilon) \\ \end{align*}$$

Finally, the bias-variance-noise decomposition is:

$$\expectation_{\sets, \epsilon}[(\,\hat{\sy}_{\sets} - \sy\,)^2] = \bias(\hat{\ff})^2 + \var(\hat{\ff}) + \var(\epsilon)$$