# Underfitting and overfitting illustrated

In this article, we define underfitting and overfitting and show some nice ways to vizualize them on polynomial regressions.

## In short

Underfitting and overfitting describe the ability of a machine-learning model to make good predictions on datasets it wasn’t trained on.

Underfitting happens when:

• The model is too rigid to learn the true relationship in the data.
• Both test error and train error are large.
• The error is dominated by the bias error.

Overfitting happens when:

• The model is not rigid enough and mistakes noise for signal.
• The train error is low but the test error is large.
• The error is dominated by the variance error.

• The left hand side graph shows the underfitting scenario. We can see that the model is so rigid that is doesn’t fit the general shape of the data. This is called the bias of the model.
• The graph in the middle displays a good fit. The model fits the general shape of the data but does not wiggle too much between data points. This is the good bias-variance equilibrium.
• The right hand side graph shows the overfitting scenario. The model wiggles too much. And the general shape of the data is obfuscated by this high variance.

## The signal and the noise

Let $\mathcal{X} \in \realvset{\inputdim}$ be a random vector. Given a deterministic function $\truemodel:\inputvec \mapsto \outputval \in \realset$, we say that the random variable $\model\paren{\mathcal{X}}$ is a signal.

Let $\epsilon \sim \gaussian(0, \sigma)$ a gaussian noise with $0$ mean. The random variable $\mathcal{Y} = \model\paren{\mathcal{X}} + \epsilon$ is our signal polluted by the noise:

The train dataset, $\trainset$ is made of $\ndataset$ independent observations of $(\mathcal{X}, \mathcal{Y})$:

This means that for each observation $\idataset \leq \ndataset$, we have:

Where:

• $\ninputvec{\idataset}$ is a realization of $\mathcal{X}$, whose value is known, and:
• $\epsilon_n$ is a realization of $\epsilon$ whose value is unknown.

## Learning the signal

The goal of machine-learning is to learn the signal function $\model$ from the inputs generated by $\mathcal{X}$ and the outputs generated by $\mathcal{Y}$. This task is made complex because of the unknown noise observations $\epsilon_n$.

To do so, we suppose a model $\model_{\text{ML}}$ for the function $\model$ and minimize the error between the observed outputs $\ioutputval{\idataset}$ and the predictions $\model_{\text{ML}}(\ninputvec{\idataset})$.

While our end goal is to approximate $\model\paren{\inputvec}$, we can only do so by comparing our predictions with $\model\paren{\inputvec} + \epsilon$. Overfitting happens when our model “learns” the specifics of the noise realizations in the train dataset.

To illustrate overfitting, we will use polynomial regression. As the degree of the fitted polynomial increases, the model has more freedom to fit complex signals. But also more freedom to fit the unwanted noise.

This is illustrated on the picture below, where:

• the red curve is the real signal $\model\paren{\mathcal{X}}$;
• the red points are observed values for this signal, polluted by the random noise $\epsilon$;
• the blue curve is the regression line learned by a polynomial regression.

## High variance

To better visualize the implications of overfitting on the regression curve, we can generate multiple train datasets, each with the same signal curve (in red), but with different random values for the noise $\epsilon$. For instance:

and:

where $\epsilon_{n, 1}$ and $\epsilon_{n, 2}$ are different values for the noise, drawn from the same $\gaussian(0, 1)$ distribution.

Let’s generate a lot of train datasets like those.

If we fit a polynomial regression to each train dataset thus generated and we graph all the regression lines (in blue) on the same plot, we can visualize the high variance induced by overfitting. See picture below.

Since we used a gaussian noise with $0$ mean, taking the mean of the regression curves should yield the signal. Indeed, as the number $\ndataset_s$ of train sets used increases, the mean of the noise observations converges to $0$: