These notes are adapted from Prof. Alexis Drouot's MATH 480 course at the University of Washington, Seattle (Winter 2025). They are my own notes, not official course materials. I have done my best to ensure correctness, but errors may remain.
The goal of these notes is to develop one route from the spectral theorem to principal component analysis (PCA). The main theme is that orthogonal decompositions reveal the geometrically important directions of a matrix or data set. We first prove the spectral theorem for real symmetric matrices, use it to obtain the polar and singular value decompositions, and then study optimal low-rank approximation. The final section applies these ideas to dimensionality reduction and PCA.
1The spectral theorem
Theorem 1.1 (Spectral theorem).
Let be symmetric. Then
where is orthogonal and is diagonal. Equivalently, has an orthonormal basis consisting of eigenvectors of .
We begin with the variational fact used in the proof. This route is deliberately independent of the characteristic polynomial: the existence of a real eigenvalue will follow from compactness and the Lagrange multiplier theorem, so no appeal to the Fundamental Theorem of Algebra is needed.
Lemma 1.2 (Extrema of the Rayleigh quotient).
Let be symmetric. The problem
has a solution . Its maximum value is the largest eigenvalue of , and is a corresponding unit eigenvector. Likewise, the minimum is the smallest eigenvalue of .
Proof. The unit sphere is compact and is continuous, so a maximizer exists. Let
On the unit sphere, , so the Lagrange multiplier condition gives
Thus . Multiplying on the left by yields
so is the maximum value.
If is any eigenvalue with unit eigenvector , then
Therefore is the largest eigenvalue. Replacing “maximum” by “minimum” proves the final assertion.
Remark 1.3.
If one assumes the Fundamental Theorem of Algebra, the existence of an eigenvalue can instead be obtained from the characteristic polynomial, after which symmetry shows that the eigenvalue may be taken real. The variational argument above avoids that assumption and additionally characterizes the largest and smallest eigenvalues through the Rayleigh quotient.
Proof of the spectral theorem. We argue by induction on . The result is immediate for .
Assume the result holds in dimension . By Lemma 1.2, has a unit eigenvector with eigenvalue . Set
The subspace is invariant under . Moreover, is invariant: for ,
Choose an orthonormal basis of and put . In this basis the matrix of has block form
Because is symmetric, is symmetric. By the induction hypothesis there are an orthogonal and a diagonal such that . Hence
The matrix
is orthogonal, and the middle matrix is diagonal. This proves the theorem.
2Positive square roots and polar decomposition
Corollary 2.1 (Positive-definite square root).
Let be symmetric positive definite. Then there is a unique symmetric positive-definite matrix such that .
Proof. By the spectral theorem,
Define
Then is symmetric positive definite and .
For uniqueness, suppose is another symmetric positive-definite square root. Since , commutes with and preserves every eigenspace of . Let be the eigenspace of corresponding to . The restriction is symmetric positive definite and satisfies . By the spectral theorem, has an orthonormal basis of eigenvectors of . If for one of these eigenvectors, then
Since is positive definite, , and hence . Therefore on . Doing this on every eigenspace of gives .
The same construction gives a unique symmetric positive-semidefinite square root when is merely positive semidefinite.
Theorem 2.2 (Thin polar decomposition).
Let with . Then
where is symmetric positive semidefinite and has orthonormal columns whenever has full column rank. If is rank deficient, one can still choose such a , but it is not unique.
Proof. First suppose has full column rank. Then is positive definite: for ,
Let and define . Then
and .
For the rank-deficient case, use the spectral theorem to write
and let denote the columns of . Whenever , define . These vectors are orthonormal, because
Complete them to an orthonormal -tuple in ; this is possible because . Set
Then . If , then , so . It follows column by column that . The choice of the corresponding to is not unique.
Remark 2.3.
In the rank-deficient case, the polar factor is canonically determined only on . Our convention extends it arbitrarily on the remaining singular directions so that has orthonormal columns; this is why need not be unique.
3Singular value decomposition
Theorem 3.1 (Singular value decomposition).
Let . Then
where and are orthogonal and is rectangular diagonal. Its nonnegative diagonal entries can be ordered as
The are the singular values of .
Proof. Assume first that . Apply the polar decomposition to write , with having orthonormal columns and symmetric positive semidefinite. By the spectral theorem,
where is orthogonal and with . Reorder the eigenvectors so that the singular values decrease. Since also has orthonormal columns, complete it to an orthogonal matrix
With
we obtain
If , apply the preceding argument to and transpose the resulting factorization.
Definition 3.2 (Induced operator norm).
For ,
Definition 3.3 (Frobenius norm).
For ,
Equivalently, if has singular values , then
Remark 3.4.
The following properties are immediate:
If is orthogonal, then for every , and .
If and have compatible dimensions, then .
Only direction matters in the supremum, because for .
For ,
In particular, is a best approximation to among matrices of rank at most .
Proof. For any with , set . Orthogonal invariance of the operator norm and rank gives
Let . Since while , there is a unit vector . Therefore
Thus every rank- approximation has error at least . On the other hand,
which attains the lower bound.
4Dimensionality reduction and principal components
Data often lie approximately near a lower-dimensional subspace. Replacing the ambient space by a smaller set of important directions can reduce computation and storage and can make a model easier to interpret. Given data points , we first seek a -dimensional linear subspace , where , that loses as little information as possible under orthogonal projection. This initial formulation constrains the approximating subspace to pass through the origin. Standard PCA first centers the observations; we return to this distinction below.
4.1The projection objective
Let be an orthonormal basis of . The orthogonal projection onto is
We wish to minimize
Because is orthogonal to ,
Moreover, using ,
Define the data matrix and unnormalized second-moment matrix by
Then
Thus minimizing reconstruction error is equivalent to maximizing
over all orthonormal .
Lemma 4.1 (Maximizing a weighted sum).
Suppose and with , where . Then
If , equality forces and .
Proof. If some has and some has , move units of mass from to . This preserves the constraints and changes the objective by . Repeating finitely many times yields , proving the inequality. If , any mass below the cutoff makes at least one such move strictly improve the objective, which gives the equality statement.
Theorem 4.2 (Principal subspace via the SVD).
Let
be an SVD of the column-oriented data matrix , with singular values , where . For convenience, set for . A minimizer of the projection objective is
where are the first left singular vectors of . The minimum reconstruction error is
If , the optimal subspace is unique, although its choice of orthonormal basis is not.
Proof. Since
the eigenvectors of are the left singular vectors of , and the corresponding eigenvalues are . For any orthonormal , put . The vectors are still orthonormal, and
where the zero-padding convention from the theorem is in force.
Let . Because , the matrix is an orthogonal projector. Hence
Lemma 4.1 now gives
Equality is attained by for . Therefore the top-left singular subspace minimizes the reconstruction error. Finally,
so the minimum error is . When , the equality condition in Lemma 4.1 forces the unique optimal subspace to be .
4.2Explained variance and choosing the dimension
The trace identity makes the optimal error especially transparent. Since ,
Consequently, retaining the first principal directions captures units of squared energy and leaves the optimal residual
The corresponding mean squared reconstruction error is
For standard PCA, first center the observations:
Applying Theorem 4.2 to finds the best -dimensional affine subspace . Without centering, the analysis finds the best linear subspace constrained to pass through the origin. For centered data, is the empirical variance in the th principal direction under the convention. If sample variance is defined with instead, replace by ; the ratios below do not change.
Provided the centered data are not all zero, the fraction of total variance explained by the first components is
One common way to choose is to select the smallest value for which this ratio reaches a prescribed target. In practice, targets such as 90%, 95%, or 99% are often used, but the appropriate threshold is application dependent. The principal directions are , the first left singular vectors of the column-oriented data matrix .