It is a fact that for any overdetermined matrix A the nullsp

It is a fact that for any overdetermined matrix A, the nullspace of A and the nullspace of A^TA are the same. You do not have to explain this fact.

Based on the equality of the nullspaces of A and A^TA , explain why an overdetermined system Ax=b has a unique least squares solution if A is full rank.

Solution

Let A be a m x n matrix over field F where m>n that is it represents an overdetermined matrix and Ax=b represents the overdetermined system.

to show it has a unique least square solution if A is full rank.

Let y belong to F^m .Define W={Ax : x belong to Fⁿ}.

There is a result that states that if W is a finite dimensional subspace of an inner product space V and let y belong to V. Then there exists unique vectors u in W and z in S(orthogonal complement of W) such that y=u+z. Also vector u is the unique vector in W that is closest to y that is for any s in W ,|| y-s||>=||y-u||.

By the above result there exists a unique vector in W that is closest to y. call this vector Ax_o where x_o is in Fⁿ. Then ||Ax_o -y||<=||Ax-y|| for all x in F^n. So x_o has the property that ||Ax_o -y|| is minimal.

Now we can see from the above result that Ax_o -y belongs to S, which is the orthogonal complement of W, so, <Ax, Ax_o -y>=0 for all x in Fⁿ which implies <x, A*(Ax_o -y)>=0 for all x in Fⁿ .This implies A*(Ax_o - y)=0. So, we need only find a solution x_o  to A*Ax=A* y.

if it is given that A is full rank then even A*A is full rank( Since nullspace of A and null space of A*A are same.)

So, x_o =(A*A)^-1A* y which is the unique least square solution.