Likelihood interpretation

• The use of a deterministic least squares criterion has an important statistical interpretation when the observations are Gauss-Markov. A ``Gauss-Markov'' process results from a finite-length all-pole filter driven by white Gaussian noise, as in (1) where $A_{1} \ldots A_{p}$ are ``filter coefficients'' and $e_{n}$ is the ``noise''. The ``Markov'' property refers to the finite memory of the filter, and means that given the pth-order past $y_{n-p:n-1}$, $y_{n}$ is independent of the further past $y_{1:n-p-1}$.

• Asymptotic efficiency of maximum likelihood From the statistical interpretation, (1) gives a parametric form for the observations' joint density. We wish to estimate the unknown parameters, $A_{1} \ldots A_{p}$, which can be thought of as a block vector, say $\theta$. A good, or ``efficient'' estimate $\hat{\theta}$ is a function of observations with the following properties:
  • Unbiased: $E(\hat{\theta}) = \theta$
  • Minimum variance: $E\left[(\hat{\theta}\!-\!\theta) (\hat{\theta}\!-\!\theta)^{*}\right]$ is PSD-minimal

  By the Cramer-Rao inequality, the maximum likelihood (ML) estimate:
  

  $$\hat{\theta} = \mathrm{arg}\ \mathrm{max} \left\{\theta: f_{\theta}(y_{1:N})\right\}$$

  
  

  ( $f_{\theta}$ gives the joint density) is asymptotically efficient as $N \rightarrow \infty$.

• Computing the ML estimate in the Gauss-Markov case, we have:

  
  

  $$f_{\left\{A_{k}\right\}}(y_{1:N}) = \prod_{n=1}^{N} f_{\left\{A_{k}\right\}}(y_{n} \vert y_{1:n})$$

  $$\approx \prod_{n=1}^{N} f_{\left\{A_{k}\right\}}(y_{n} \vert y_{n-p:n-1})$$

  $$= \prod_{n=1}^{N} f_{\left\{A_{k}\right\}}(e_{n} \vert y_{n-p:n-1})$$

  $$= \prod_{n=1}^{N} f_{\left\{A_{k}\right\}}(e_{n})$$ (10)

  
  

  The second step is justified by the Markov property: conditional independence allows us to drop conditioning on the further past $y_{1:n-p-1}$. The $\approx$ is due to the absence of data for $n<p$. These ``edge effects'' wash out for large $N$. The third step comes by the fact that conditional on $y_{n-p:n-1}$, $e_{n}$ and $y_{n}$ differ by a constant; thus the Jacobian for the change-of-variables $y_{n} \rightarrow e_{n}$ is identity. Finally, the last step results from independence of the $e_{n}$ and the fact $y_{n}$ depends only on present and past $e_{n}$.

• Now to maximize the likelihood, it is equivalent to minimize the negative log likelihood, $-L(\left\{A_{k}\right\}) = - \log f_{\left\{A_{k}\right\}}(y_{1:N})$. From (10) and the form of the Gaussian likelihood:
  

  $$-L(\left\{A_{k}\right\}) = \sum_{n=1}^{N} - \log \left[ \left(2 \pi \vert R_{e}\vert\right)^{1/2} \mathrm{exp}\left(-e^{*}_{n}R^{-1}_{e}e_{n}/2\right) \right]$$

  $$= \frac{1}{2} \log \left(2 \pi \vert R_{e}\vert\right) + \frac{1}{2}\sum_{n=1}^{N}e^{*}_{n}R^{-1}_{e}e_{n}$$ (11)

  
  

  The first term is constant w.r.t. $\left\{A_{k}\right\}$ and may be dropped. The second term is equivalent to the weighted-norm criterion (4) with $W = R^{-1}_{e}$. Hence in the Gauss-Markov case, the deterministic least-squares criterion approaches the ML criterion for large $N$, giving an asymptotically efficient estimate of the $\left\{A_{k}\right\}$.