Symmetric matrices, principal axes and the SVD

Symmetric matrices, those equal to their own transpose, appear everywhere in engineering: stiffness and mass matrices, stress and strain tensors, inertia tensors, covariance matrices of measurements, and the matrix \(A^TA\) of least squares (Lesson 10). They have exceptionally well-behaved eigenvalues and eigenvectors, a result known as the spectral theorem, which gives every symmetric matrix a set of perpendicular principal axes. Principal axes explain why a loaded material has directions of maximum and minimum stress, why a structure vibrates in distinct mode shapes, why a spinning body has stable axes of rotation, and why the scatter of measured data has a dominant direction. This lesson develops the spectral theorem and its applications, connects symmetric matrices to quadratic forms and energy, and then introduces the singular value decomposition (SVD), which extends the idea of principal axes to every matrix and underlies data compression, model reduction and the reliable solution of difficult least squares problems.

The spectral theorem

Lesson 9 showed that eigenvalues can be complex and that some matrices cannot be diagonalized. Symmetric matrices have neither problem. For a real symmetric matrix \(S = S^T\):

• All eigenvalues are real.
• Eigenvectors belonging to different eigenvalues are orthogonal.
• There is always a full set of orthonormal eigenvectors, so \(S\) can be diagonalized by an orthogonal matrix.

In matrix form, the spectral theorem states:

where \(\Lambda\) is the diagonal matrix of eigenvalues and the columns of the orthogonal matrix \(Q\) (Lesson 8) are the corresponding unit eigenvectors. Geometrically, \(S\) acts by rotating to the eigenvector axes (\(Q^T\)), stretching along each axis by its eigenvalue (\(\Lambda\)), and rotating back (\(Q\)). The eigenvector directions are the principal axes of \(S\). Equivalently, \(S = \sum \lambda_i \mathbf{q}_i\mathbf{q}_i^T\): a symmetric matrix is a weighted sum of projections onto its principal axes.

Principal stresses

The state of stress at a point in a loaded plate is described by a symmetric matrix, the stress tensor, containing normal stresses \(\sigma_x\), \(\sigma_y\) and the shear stress \(\tau_{xy}\):

Its eigenvalues are the principal stresses, the largest and smallest normal stresses on any plane through the point, and its eigenvectors are the principal directions, on whose planes the shear stress is zero. Materials such as concrete and cast iron crack perpendicular to the direction of maximum tensile principal stress, and design codes for metals compare combinations of principal stresses with the material's strength.

Quadratic forms and definiteness

A symmetric matrix defines a quadratic form \(Q(\mathbf{x}) = \mathbf{x}^T S\mathbf{x}\), a homogeneous quadratic expression in the components of \(\mathbf{x}\); for example, \(5x^2 + 4xy + 2y^2\) corresponds to the matrix in the first example. Quadratic forms represent energies: the strain energy of a structure with stiffness matrix \(K\) and displacements \(\mathbf{u}\) is \(\tfrac{1}{2}\mathbf{u}^TK\mathbf{u}\), and the kinetic energy of a system with mass matrix \(M\) is \(\tfrac{1}{2}\dot{\mathbf{u}}^TM\dot{\mathbf{u}}\).

In the principal axes, the quadratic form has no cross terms: writing \(\mathbf{x} = Q\mathbf{y}\) gives \(Q(\mathbf{x}) = \lambda_1 y_1^2 + \lambda_2 y_2^2 + \cdots\). The signs of the eigenvalues therefore classify the form. If all eigenvalues are positive, the form is positive for every non-zero \(\mathbf{x}\) and \(S\) is positive definite, the property of stiffness matrices of properly supported structures noted in Lesson 5. If some eigenvalues are negative, the form takes both signs; if some are zero, there are directions with no energy, the mechanisms of Lesson 7. The level set \(\mathbf{x}^TS\mathbf{x} = 1\) of a positive definite form is an ellipse (an ellipsoid in three dimensions) whose axes lie along the eigenvectors, with semi-axis lengths \(1/\sqrt{\lambda_i}\).

Vibration modes

A structure with mass matrix \(M\) and stiffness matrix \(K\), both symmetric and positive definite, vibrates freely according to \(M\ddot{\mathbf{u}} + K\mathbf{u} = \mathbf{0}\). Seeking motions in which every point moves in step, \(\mathbf{u} = \boldsymbol{\phi}\cos\omega t\), gives the generalized eigenvalue problem \(K\boldsymbol{\phi} = \omega^2 M\boldsymbol{\phi}\). Its eigenvalues are the squares of the natural frequencies, and its eigenvectors are the mode shapes. Because \(K\) and \(M\) are symmetric, the mode shapes are orthogonal with respect to the mass matrix, so any free vibration is a superposition of independent modes, each oscillating at its own frequency.

Principal components of data

Measurements of several quantities, such as the temperatures at several sensors or the dimensions of manufactured parts, are summarized by their covariance matrix, a symmetric positive semi-definite matrix whose diagonal entries are variances and whose off-diagonal entries measure how pairs of quantities vary together. Its eigenvectors, the principal components, are the directions of greatest variation, and its eigenvalues are the variances along them. Principal component analysis (PCA) uses the few components with the largest eigenvalues to describe data with many correlated variables compactly, for example to detect abnormal operation of a plant from dozens of correlated sensor readings, or to find the dominant patterns of variation in a set of shapes. AI-210 and STAT-235 develop these methods further.

The singular value decomposition

Non-symmetric and even non-square matrices also have principal axes, though in general two different sets of them: one in the input space and one in the output space. The singular value decomposition states that every \(m \times n\) matrix can be factored as:

where \(U\) (\(m \times m\)) and \(V\) (\(n \times n\)) are orthogonal and \(\Sigma\) is an \(m \times n\) diagonal matrix whose non-negative diagonal entries \(\sigma_1 \ge \sigma_2 \ge \cdots \ge 0\) are the singular values. Geometrically, every matrix rotates (\(V^T\)), stretches along perpendicular axes by the singular values (\(\Sigma\)), and rotates again (\(U\)); it maps the unit sphere to an ellipsoid whose semi-axes are the singular values. The singular values are the square roots of the eigenvalues of the symmetric matrix \(A^TA\), and the columns of \(V\) are its eigenvectors, which connects the SVD to the spectral theorem.

The SVD reveals everything important about a matrix:

• The rank is the number of non-zero singular values, and in numerical work, the number that are not negligibly small.
• The condition number (Lesson 5) is \(\sigma_1/\sigma_n\), the ratio of the largest to the smallest singular value.
• The best approximation of \(A\) by a matrix of rank \(k\) keeps the \(k\) largest singular values and their vectors: \(A_k = \sum_{i=1}^{k}\sigma_i\mathbf{u}_i\mathbf{v}_i^T\). This is the basis of image compression, noise reduction and the reduced-order models used to simulate large structures and flows quickly.
• The pseudo-inverse built from the SVD gives the least squares solution of minimum length even when the columns of \(A\) are dependent, handling the rank-deficient cases where the normal equations fail.

The inertia tensor

The rotational inertia of a rigid body about different axes is described by a symmetric \(3 \times 3\) matrix, the inertia tensor. Its eigenvalues are the principal moments of inertia, and its eigenvectors are the principal axes of the body. A body spins smoothly about a principal axis; about any other axis it wobbles, and the bearings of an unbalanced rotor feel a rotating force. Balancing a car wheel or a turbine rotor means adding small masses so that a principal axis lines up with the axis of rotation. A further result of mechanics, illustrated by a tossed book or a tumbling satellite, is that rotation about the axes of largest and smallest moment is stable, while rotation about the intermediate axis is not, a fact that spacecraft engineers must respect when designing attitude control.