Least-squares solution and data windowing

• Observations come in a finite interval, say $n=1 \ldots N$. So far we haven't considered the range of the error-summation (2). Writing the model equation (1) for every $n=1 \ldots N$ and collecting the equations in a single matrix-vector equation gives:
  

  $$A T = H$$ (5)

  
  where
  

  $$T = \left[\begin{array}{cccc} y_{0} & y_{1} & \ldots & y_{N-1} \\ y_... ... & \ddots & \vdots \\ y_{1-p} & y_{2-p} & \ldots & y_{N-p} \end{array} \right]$$

  
  
  

  $$A = \left[A_{1} \ldots A_{p}\right]$$

  $$H = \left[y_{1} \ldots y_{N}\right]$$

  
  
• Assuming $T$ has full row rank, there exists a unique solution for the PSD-minimal least-squares criterion 2, namely the standard one:
  

  $$A = H T^{*} (R)^{-1},$$

  $$R = (T T^{*})$$ (6)

  
  Let $R \stackrel{\Delta}{=}TT^{*}$ for the remainder of these notes. The proof is deferred to an Appendix. $R$ is singular iff $T$ is rank deficient, in which case optimal $A$ exist, but are no longer unique. To simplify we assume $R$ is nonsingular.

• The problem is that $y_{1-p} \ldots y_{0}$ are outside the observation window. Usually, this problem is addressed by one of three windowing methods:
  1. Covariance method Delete columns of $T$ (and corresponding elements of $H$) until no data is accessed outside $n=1 \ldots N$, to obtain:
     

     $$T_{C} = \left[\begin{array}{cccc} y_{p} & y_{p+1} & \ldots & y_{N-1} \\ ... ...dots & \ddots & \vdots \\ y_{1} & y_{2} & \ldots & y_{N-p} \end{array} \right]$$

     $$H_{C} = \left[y_{p+1} \ldots y_{N}\right]$$ (7)

     
     

  2. Prewindowed method Set the inaccessible data at the beginning of the observation window $y_{1-p} \ldots y_{0}$ to zero. Since the first column of $T$ is identically zero, we delete it (and the corresponding element in $H$) because it offers no useful information.
     

     $$T_{P} = \left[\begin{array}{cccc} y_{1} & y_{2} & \ldots & y_{N-1} \\ 0 ... ...ots & \vdots & \ddots & \vdots \\ 0 & 0 & \ldots & y_{N-p} \end{array} \right]$$

     $$H_{P} = \left[y_{2} \ldots y_{N}\right]$$ (8)

     
     
  3. Autocorrelation method Window the data symmetrically at both ends, setting inaccessible elements $y_{1-p} \ldots y_{0}$, $y_{N+1} \ldots y_{N+p}$ to zero:
     

     $$T_{A} = \left[\begin{array}{ccccc} y_{1} & y_{2} & \ldots & 0 & 0 \\ 0 &... ...s & \ddots & y_{N} & 0 \\ 0 & 0 & \ldots & y_{N-1} & y_{N} \end{array} \right]$$

     $$H_{A} = \left[y_{2} \ldots y_{N} \ 0 \ldots 0\right]$$ (9)

     
     

• Remarks
  1. The covariance method has the least bias and should be used whenever conditions allow. Any windowing has the effect of smoothing spectral peaks, causing pole estimates to be over damped.
  2. The autocorrelation method involves the most windowing, but the following properties hold:
     • The estimated ``modeling filter''; e.g. $y(n) = (I + A_{1}z^{-1} + \ldots A_{p}z^{-p})^{-1} e(n)$, where $z^{-1}$ is the delay operator, is guaranteed stable. This holds even in the matrix case.

     • Thanks to the symmetry of the windowing, the matrix $R_{A} = (T_{A}T^{*}_{A})$ is Hermitian and has the ``Toeplitz'' property that the $(i,j)$ block of dimension $(m \times m)$, $1\!<\!i\!<\!p$, $1\!<\!j\!<\!p$ depends only on $i-j$. Hence $R_{A}$ is a valid autocorrelation matrix for $y_{n}$ treated as a stationary vector process (hence the name) The structure of $R_{A}$ leads to the simple (Levinson-Durbin) recursion for updating model parameter estimates recursively in the order ( $p \rightarrow p+1$).

     • In practice, pole frequency estimates are unaffected by autocorrelation windowing. So if you want to estimate groups of formant frequencies as features for speech recognition, but don't care about the bandwidths (commonly assumed irrelevant), use autocorrelation windowing.

  3. The pre-windowed method simplifies the development of adaptive lattice algorithms, which update the least squares solution recursively in time ( $N \rightarrow N+1$) and in the order ( $p \rightarrow p+1$). It is easiest to develop the covariance lattice method as an extension of the pre-windowed method.