Next  |  Prev  |  Up  |  Top  |  JOS Index  |  JOS Pubs  |  JOS Home  |  Search

Geometrical Interpretation of Least Squares

Typically, $ L\ll N$ , i.e., the number of frequency constraints is much greater than the number of design variables (filter taps). In these cases, we have an overdetermined system of equations (more equations than unknowns). Therefore, we cannot generally satisfy all the equations, and we are left with minimizing some error criterion to find the ``optimal compromise'' solution.

In the case of least-squares approximation, we are minimizing the Euclidean distance, which suggests the following geometrical interpretation:
\begin{psfrags}\psfrag{Ax}{\normalsize $A\hat{x}$}\psfrag{column}{\normalsize column-space of $A$}\psfrag{space}{}\begin{center}
\epsfig{file=eps/lsq.eps,width=3in} \\
\end{center} % was epsfbox
\end{psfrags}

$\displaystyle b = A \hat{x} + e
$

$\displaystyle \hbox{Minimize}_x \Vert e\Vert _2 = \Vert b-Ax\Vert _2
$


\begin{psfrags}\psfrag{Ax}{\normalsize $A\hat{x}$}\psfrag{column}{\normalsize column-space of $A$}\psfrag{space}{}\begin{center}
\epsfig{file=eps/lsq.eps,width=3in} \\
\end{center} % was epsfbox
\end{psfrags}

This diagram suggests that the error vector $ b-A\hat{x}$ is orthogonal to the column space of the matrix $ A$ , hence it must be orthogonal to each column in $ A$ .

$\displaystyle A^T(b-A\hat{x})=0 \Rightarrow A^TA\hat{x}=A^Tb
$

This is how the orthogonality principle can be used to derive the fact that the best least squares solution is given by

$\displaystyle \hat{x} = (A^TA)^{-1}A^T b = A^\dagger b
$

Note that the pseudo-inverse $ A^\dagger$ projects the vector $ b$ onto the column space of $ A$ .

In Matlab:


Next  |  Prev  |  Up  |  Top  |  JOS Index  |  JOS Pubs  |  JOS Home  |  Search

[Comment on this page via email]

``The Window Method for FIR Digital Filter Design}'', by Julius O. Smith III, (From Lecture Overheads, Music 421).
Copyright © 2020-06-27 by Julius O. Smith III
Center for Computer Research in Music and Acoustics (CCRMA),   Stanford University
CCRMA  [Automatic-links disclaimer]