Taylor's Theorem

Theorem. (Taylor) Every functional $J:\Re ^{\hat N}\mapsto\Re ^1$ in ${\cal C}_2(\Re ^{\hat N})$ has the representation

$$J({\hat \theta}+\eta) = J({\hat \theta}) + J^\prime({\hat \theta}) \eta + \frac{1}{ 2} \eta^T J^{\prime\prime}({\hat \theta}+\lambda\eta) \eta$$

for some $\lambda$ between 0 and $1$, where $J^\prime({\hat \theta})$ is the ${\hat N}\times 1$ gradient vector evaluated at ${\hat \theta}\in\Re ^n$, and $J^{\prime\prime}({\hat \theta})$ is the ${\hat N}\times{\hat N}$ Hessian matrix of $J$ at ${\hat \theta}$, i.e.,

$$J^\prime({\hat \theta}) \mathrel{\stackrel{\mathrm{\Delta}}{=}} \frac{\partial J(\theta)}{\partial \theta}({\hat \theta})$$ (5)

$$J^{\prime\prime}({\hat \theta}) \mathrel{\stackrel{\mathrm{\Delta}}{=}} \frac{\partial^2 J(\theta)}{\partial {\hat \theta}^2}({\hat \theta})$$ (6)

Proof. See Goldstein [2] p. 119. The Taylor infinite series is treated in Williamson and Crowell [7]. The present form is typically more useful for computing bounds on the error incurred by neglecting higher order terms in the Taylor expansion.