The bias-variance decomposition

The MSE loss is attractive because the expected error in estimation can be explained by the bias and the variance of the model. This is called the bias-variance decomposition. In this article, we will introduce this decomposition using the tools of probability theory.

In short, the bias-variance decomposition is:

In machine-learning, the error in estimation is not the same as the error in prediction. The error in prediction can be explained in terms of the bias-variance-noise decomposition.

Notations

Let $(\rvx, \ry)$ be a pair of random variables on $\realvset{\sd} \times \realset$.

Assume there exists a function $\ff$ such that:

The goal of a regression is to use a sample $\trainset$ to estimate this function:

For instance, in a linear regression the function $\ff$ is a linear function with parameter $\vw$:

And the regression aims at estimating $\vw$ from the training-set:

The expected output

The output $\hat{\ff}_{\trainset}$ of the regression depends on the sample $\sets = \trainset$ that was used during training. In expectation, the estimated function is:

Bias

The bias measures how wrong the model is on average. It is the difference between this expected function and the true function:

Variance

The variance measures how instable the model is. The more the estimated function $\hat{\ff}_{\sets}$ depends on the specific details of the training-set $\sets$, the higher the variance. It is equal to:

The bias-variance decomposition

What error in estimation can we expect when $\trainset$ variates?

The bias-variance decomposition is:

It is important to distinguish error in estimation (between $\ff$ and $\hat{\ff}$) and error in prediction (between $\sy$ and $\hat{\sy}$). Let’s tackle the error in prediction now.

The bias-variance-noise decomposition

A frequent use case for regression is when $\sy$ is a signal of $\rvx$ distorted by some random $0$-mean noise $\epsilon$:

In such cases, the error in prediction between $\ry$ and $\hat{\ry}$ can be expressed using the bias-variance-noise decomposition:

See our dedicated article for more info.

Illustration

This can be illustrated using polynomial regressions. A polynomial regression of degree $1$ is a line regression, which have high bias but very low variance. As the polynomial degree increases, the bias reduces but the variance increases.

On the plot below:

• the red curve is the deterministic relationship $\truemodel(\inputvec)$;
• the red dots are observations $(\inputvec, \outputval)$ polluted by the noise $\epsilon$ (which explains why they are not on the line);
• the blue curve is the regression line of one model $\trainedmodel$.

To vizualize the bias and variance tradeoff, we need to train several models. Let’s generate several trainset $\trainset \sim \mathcal{S}$ using the source. Recall that the relationship $\truemodel$ to be learned is the same accross datasets, but the noise observations are random.

On the plot below:

• the red curve is the deterministic relationship $\truemodel$;
• we didn’t plot the red dots to avoid cluter;
• the blue curves are the regression lines for each of the trained model generated thus.

We can see that as the degree increases the blue curves are more and more appart from each other. This is a manifestation of the high variance.

Taking the average of the blue lines, we can visualize the bias. On the picture below, we graphed the mean of the regression curves in blue. The gray shape around it is the spread of all the regression curves.

• We can see that for low degree polynomials, the blue curve does not fit the red curve: they have high bias. But the gray shape has small width: they have low variance.
• On the other hand, for high degree polynomials, the blue curve perfectly match the red curve: they have low bias. But the gray shape has large width: they have high variance.

Other articles you might like: