The geometry of the normal equations

In this article, I show that the normal equations define the orthogonal projection of a vector onto a linear subspace.


Let be a vector space and a linear subspace. Let be a matrix whose column vectors are a basis for , i.e. . Each vector in can be expressed as a linear combination of the column vectors in :

The orthogonal projection

Given a vector , the orthogonal projection of onto is the vector in such that the residual is orthogonal to every vector in :

  1. ,
  2. ,
  3. .

Solving the equations

A vector is orthogonal to every vector in if and only if , hence we are looking to solve the following equation:

Since , we can find a vector such that:

So the equation to be solved is:

Which is the matrix form of the normal equations.