# The geometry of the normal equations

In this article, I show that the normal equations define the orthogonal projection of a vector onto a linear subspace.

## Setup

Let $\mathcal{S}$ be a vector space and $\mathcal{X} \subseteq \mathcal{S}$ a linear subspace. Let $X$ be a matrix whose column vectors are a basis for $\mathcal{X}$, i.e. $\mathcal{X} = \mathrm{span}(X)$. Each vector in $\mathcal{X}$ can be expressed as a linear combination of the column vectors in $X$:

## The orthogonal projection

Given a vector $\vec{y} \in \mathcal{S}$, the orthogonal projection of $\vec{y}$ onto $\mathcal{X}$ is the vector $\vec{y}_{\parallel}$ in $\mathcal{X}$ such that the residual $\vec{y}_{\perp} = \vec{y} - \vec{y}_{//}$ is orthogonal to every vector in $\mathcal{X}$:

1. $(\vec{y}_{\parallel}) \in \mathcal{X}$,
2. $\vec{y} = (\vec{y}_{\parallel}) + (\vec{y}_{\perp})$,
3. $\forall w \in \mathcal{S}, \quad (\vec{y}_{\perp}) \perp (X\vec{w})$.

## Solving the equations

A vector $\vec{v}$ is orthogonal to every vector in $\mathcal{X}$ if and only if $X^{\top}\,\vec{v} = 0$, hence we are looking to solve the following equation:

Since $(\vec{y}_{\parallel}) \in \mathcal{X}$, we can find a vector $\vec{w} \in \mathcal{S}$ such that:

So the equation to be solved is:

Which is the matrix form of the normal equations.

Other articles you might like: