The geometry of the normal equations

In this article, I show that the normal equations define the orthogonal projection of a vector onto a linear subspace.

Setup

Let $\mathcal{S}$ be a vector space and $\mathcal{X} \subseteq \mathcal{S}$ a linear subspace. Let $X$ be a matrix whose column vectors are a basis for $\mathcal{X}$, i.e. $\mathcal{X} = \mathrm{span}(X)$. Each vector in $\mathcal{X}$ can be expressed as a linear combination of the column vectors in $X$:

$$\forall \vec{v} \in \mathcal{X}, \quad \exists \vec{w} : \quad \vec{v} = X\vec{w}$$

The orthogonal projection

Given a vector $\vec{y} \in \mathcal{S}$, the orthogonal projection of $\vec{y}$ onto $\mathcal{X}$ is the vector $\vec{y}_{\parallel}$ in $\mathcal{X}$ such that the residual $\vec{y}_{\perp} = \vec{y} - \vec{y}_{//}$ is orthogonal to every vector in $\mathcal{X}$:

1. $(\vec{y}_{\parallel}) \in \mathcal{X}$,
2. $\vec{y} = (\vec{y}_{\parallel}) + (\vec{y}_{\perp})$,
3. $\forall w \in \mathcal{S}, \quad (\vec{y}_{\perp}) \perp (X\vec{w})$.

Solving the equations

A vector $\vec{v}$ is orthogonal to every vector in $\mathcal{X}$ if and only if $X^{\top}\,\vec{v} = 0$, hence we are looking to solve the following equation:

$$\begin{align*} &X^{\top}\,(\vec{y}_{\perp}) = 0 \\ \iff &X^{\top}\,(\vec{y} - \vec{y}_{\parallel}) = 0 \end{align*}$$

Since $(\vec{y}_{\parallel}) \in \mathcal{X}$, we can find a vector $\vec{w} \in \mathcal{S}$ such that:

$$\vec{y}_{\parallel} = X\vec{w}$$

So the equation to be solved is:

$$X^{\top}\,(\vec{y} - X\vec{w}) = 0$$

Which is the matrix form of the normal equations.