本文为《Linear algebra and its applications》的读书笔记

Quadratic forms

A quadratic form on ${\mathbb{R}}^n$ is a function $Q$ defined on ${\mathbb{R}}^n$ whose value at a vector $\mathbf{x}$ in ${\mathbb{R}}^n$ can be computed by an expression of the form $Q(\mathbf{x})={\mathbf{x}}^{T}A\mathbf{x}$, where $A$ is an $n\times n$ symmetric matrix. The matrix $A$ is called the matrix of the quadratic form (关于二次型的矩阵).

The simplest example of a nonzero quadratic form is $Q(\mathbf{x})={\mathbf{x}}^{T}I\mathbf{x}={\Vert \mathbf{x}\Vert }^2$. Examples 1 and 2 show the connection between any symmetric matrix $A$ and the quadratic form ${\mathbf{x}}^{T}A\mathbf{x}$.

EXAMPLE 1
Let $\mathbf{x}=\left[\begin{array}{c}{x}_1\\ x_2\end{array}\right]$. Compute ${\mathbf{x}}^{T}A\mathbf{x}$ for the following matrices:

SOLUTION
a.

b. There are two $-2$ entries in $A$. Watch how they enter the calculations.

The presence of $-4x_1{x}_2$ in the quadratic form in Example 1(b) is due to the $-2$ entries off the diagonal in the matrix $A$. In contrast, the quadratic form associated with the diagonal matrix $A$ in Example 1(a) has no $x_1{x}_2$ $cross$-$product(交叉乘积)$ term.

EXAMPLE 2
For $\mathbf{x}$ in ${\mathbb{R}}^3$, let $Q(\mathbf{x})=5x_1^2+3x_2^2+2x_3^2-x_1{x}_2+8x_2{x}_3$. Write this
quadratic form as ${\mathbf{x}}^{T}A\mathbf{x}$.
SOLUTION
The coefficients of ${\mathbf{x}}_1^2,{\mathbf{x}}_2^2,{\mathbf{x}}_3^2$ go on the diagonal of $A$. To make $A$ symmetric, the coefficient of $x_i{x}_j$ for $i\ne j$ must be split evenly between the $(i,j)$- and $(j,i)$-entries in $A$. It is readily checked that

In some cases, quadratic forms are easier to use when they have no cross-product terms—that is, when the matrix of the quadratic form is a diagonal matrix. Fortunately, the cross-product term can be eliminated by making a suitable change of variable.

Change of Variable in a Quadratic Form 二次型的变量代换

If $\mathbf{x}$ represents a variable vector in ${\mathbb{R}}^n$, then a change of variable is an equation of the form

where $P$ is an invertible matrix and $\mathbf{y}$ is a new variable vector in ${\mathbb{R}}^n$. Here $\mathbf{y}$ is the coordinate vector of $\mathbf{x}$ relative to the basis of ${\mathbb{R}}^n$ determined by the columns of $P$.

If the change of variable (1) is made in a quadratic form ${\mathbf{x}}^{T}A\mathbf{x}$, then

and the new matrix of the quadratic form is $P^{T}AP$. Since $A$ is symmetric, there is an orthogonal matrix $P$ such that $P^{T}AP$ is a diagonal matrix $D$, and the quadratic form in (2) becomes ${\mathbf{y}}^{T}D\mathbf{y}$.

主轴定理

The columns of $P$ in the theorem are called the principal axes (主轴) of the quadratic form ${\mathbf{x}}^{T}A\mathbf{x}$. The vector $\mathbf{y}$ is the coordinate vector of $\mathbf{x}$ relative to the orthonormal basis of ${\mathbb{R}}^n$ given by these principal axes.

A Geometric View of Principal Axes

Suppose $Q(\mathbf{x})={\mathbf{x}}^{T}A\mathbf{x}$, where $A$ is an invertible $2\times 2$ symmetric matrix, and let $c$ be a constant. It can be shown that the set of all $\mathbf{x}$ in ${\mathbb{R}}^2$ that satisfy

either corresponds to an ellipse (or circle), a hyperbola(双曲线), two intersecting lines, or a single point, or contains no points at all. If $A$ is a diagonal matrix, the graph is in standard position, such as in Figure 2.

If $A$ is not a diagonal matrix, the graph of equation (3) is rotated out of standard position, as in Figure 3. Finding the principal axes (determined by the eigenvectors of $A$) amounts to finding a new coordinate system with respect to which the graph is in standard position.

The positive $y_1$-axis in Figure 3(b) is in the direction of the first column of the matrix $P$, and the positive $y_2$-axis is in the direction of the second column of $P$.

Classifying Quadratic Forms 二次型的分类

When $A$ is an $n\times n$ matrix, the quadratic form $Q(\mathbf{x})={\mathbf{x}}^{T}A\mathbf{x}$ is a real-valued function with domain ${\mathbb{R}}^n$. Figure 4 displays the graphs of four quadratic forms with domain ${\mathbb{R}}^2$. For each point $\mathbf{x}=(x_1,x_2)$ in the domain of a quadratic form $Q$, the graph displays the point $(x_1,x_2,z)$ where $z=Q(\mathbf{x})$. Notice that except at $\mathbf{x}=0$, the values of $Q(\mathbf{x})$ are all positive in Figure 4(a) and all negative in Figure 4(d). The horizontal cross-sections(水平截面) of the graphs are ellipses in Figures 4(a) and 4(d) and hyperbolas in Figure 4$(c)$.

The simple $2\times 2$ examples in Figure 4 illustrate the following definitions

positive definite (正定的)
negative definite (负定的)
indefinite (不定的)

Also,$Q$ is said to be positive semidefinite (半正定的) if $Q(\mathbf{x})\ge 0$ for all $\mathbf{x}$, and to be negative semidefinite if $Q(\mathbf{x})\le 0$ for all $\mathbf{x}$.

Theorem 5 characterizes some quadratic forms in terms of eigenvalues.

PROOF
By the Principal Axes Theorem, there exists an rthogonal change of variable $\mathbf{x}=P\mathbf{y}$ such that

where ${\lambda }_1,...,{\lambda }_n$ are the eigenvalues of $A$. Since $P$ is invertible, there is a one-to-one correspondence between all nonzero $\mathbf{x}$ and all nonzero $\mathbf{y}$. Thus the values of $Q(\mathbf{x})$ for $\mathbf{x}\ne 0$ coincide with the values of the expression on the right side of (4), which is obviously controlled by the signs of the eigenvalues ${\lambda }_1,...,{\lambda }_n$, in the three ways described in the theorem.

The classification of a quadratic form is often carried over to the matrix of the form. Thus a positive definite matrix(正定矩阵) $A$ is a symmetric matrix for which the quadratic form ${\mathbf{x}}^{T}A\mathbf{x}$ is positive definite. Other terms, such as positive semidefinite matrix, are defined analogously.

Another useful way to characterize quadratic forms, often used in multivariable calculus courses.
Let $A=\left[\begin{array}{cc}a& b\\ c& d\end{array}\right]$. If ${\lambda }_1$ and ${\lambda }_2$ are the eigenvalues of $A$, then the characteristic polynomial is $det(A-\lambda I)={\lambda }^2-(a+d)\lambda +ad-b^2$. Thus ${\lambda }_1+{\lambda }_2=a+d$ and ${\lambda }_1{\lambda }_2=detA$.
Then the following statements can be easily verified:
$a$. $Q$ is positive definite if $detA>0$ and $a>0$.
$b$. $Q$ is negative definite if $detA>0$ and $a<0$.
$c$. $Q$ is indefinite if $detA<0$.
(The $2\times 2$ case can be generalized to $n\times n$ matrices.)

EXERCISE 25
Show that if $B$ is $m\times n$, then $B^{T}B$ is positive semidefinite; and if $B$ is $n\times n$ and invertible, then $B^{T}B$ is positive definite.
SOLUTION
[Hint: ${\mathbf{x}}^T{B}^{T}B\mathbf{x}$]

EXERCISE 26
Show that if an $n\times n$ matrix $A$ is positive definite, then there exists a positive definite matrix $B$ such that $A=B^{T}B$.
SOLUTION
[Hint: Use the orthogonal decomposition]

EXERCISE 27
Let $A$ and $B$ be symmetric $n\times n$ matrices whose eigenvalues are all positive. Show that the eigenvalues of $A+B$ are all positive.
SOLUTION
[Hint: Consider quadratic forms.]

EXERCISE
If $A$ is $m\times n$, then the matrix $G=A^{T}A$ is called the $Gram$ $matrix$ of $A$. Show that the Gram matrix of any matrix $A$ is positive semidefinite, with the same rank as $A$.
SOLUTION
[Hint: Section 6.5 Theorem 14]

楚列斯基分解

EXERCISE
Prove that an $n\times n$ matrix $A$ is positive definite if and only if $A$ admits a Cholesky factorization, namely, $A=R^{T}R$ for some invertible upper triangular matrix $R$ whose diagonal entries are all positive.
SOLUTION
[Hint: Use a QR factorization and Exercise 26.]
If $A=R^{T}R$, where R is invertible, then ${\mathbf{x}}^{T}A\mathbf{x}=(R\mathbf{x}{)}^T(R\mathbf{x})\ge 0$ when $\mathbf{x}\ne 0$. Thus $A$ is positive definite.

Conversely, suppose that $A$ is positive definite. Then by Exercise 26, $A=B^{T}B$ for some positive definite matrix $B$. Since the eigenvalues of $B$ are positive, 0 is not an eigenvalue and so $B$ is invertible. In particular, the columns of $B$ are linearly independent. By Theorem 12 in Section 6.4, $B=QR$ for some $n\times n$ matrix $Q$ with orthonormal columns and some upper triangular matrix $R$ with positive elements on its diagonal. Since $Q$ is square, $Q^{T}Q=I$. So