2.3 Computer Application Considerations for Linear Least-Squares

Condition Numbers

In the previous section, it was shown that the problem

has a unique solution, provided that the square matrix $\,\boldsymbol{\rm X}^t\boldsymbol{\rm X}\,$ is invertible.

Theoretically (using exact arithmetic), the invertibility of a square matrix $\,A\,$ is easily characterized in a variety of ways: for example, $\,A\,$ is invertible if and only if the determinant of $\,A\,$ (denoted by $\,\text{det}(A)\,$) is nonzero, and in this case, the unique solution to $\,Ax = b\,$ is given by $\,x = A^{-1}b\,.$

However, when the entries of a matrix are represented only to the degree of accuracy of a digital computer, the question of invertibility is more subtle: there are invertible matrices $\,A\,$ for which correct numerical solutions to $\,Ax = b\,$ are difficult to obtain, stemming from the fact that small changes in the matrices $\,A\,$ and $\,b\,$ can lead to large changes in the solution $\,x\,.$

Example: MATLAB Casts Suspicion on a Solution Vector

To illustrate this point, reconsider the MATLAB example of the previous section, where a $4^{\text{th}}$-order polynomial least-squares approximate of the data

was found. If, alternatively, a $7^{\text{th}}$-order polynomial approximate was sought, the MATLAB user would have been faced with this warning, casting suspicion on the resulting vector $\,\boldsymbol{\rm b}\,$:

The size of $\,\boldsymbol{\rm X}^t\boldsymbol{\rm X}\,$ is only $\,8\times8\,$; however, the entries of $\,\boldsymbol{\rm X}^t\boldsymbol{\rm X}\,$ are such that the matrix does not lend itself nicely to numerical computation. This problem is the subject of the current section.

More precisely, this section gives an analysis of the system $\,Ax = b\,$ in order to understand the sensitivity of solutions $\,x\,$ to small changes in the matrices $\,A\,$ and $\,b\,.$ A number $\,\text{cond}(A)\,$ will be defined that measures this sensitivity.

For a ‘sensitive’ system, correct numerical solution can be challenging, but the problem can often be overcome by replacing the system $\,Ax = b\,$ with an equivalent system $\,\tilde Ax = b\,,$ where the new matrix $\,\tilde A\,$ is more suitable for numerical computations. This replacement requires a knowledge of discrete orthogonal functions, which are also discussed in this section.

Let $\,\phi\, :\, X\rightarrow Y\,$ be a function between normed spaces. The reader is referred to Appendix 2 for a review of normed spaces. The symbol $\,\|\cdot\|\,$ will be used to denote both the $\,X\,$ and $\,Y\,$ norms, with context determining the correct interpretation.

Let $\,x\in X\,,$ and let $\,\tilde x\,$ be an approximation to $\,x\,.$ For $\,x \ne 0\,,$ the relative error of $\,\tilde x\,$ (as an approximation to $\,x\,$) is defined to be the number:

Similarly, for $\,\phi(x)\ne 0\,,$ the relative error of $\,\phi(\tilde x)\,$ (as an approximation to $\,\phi(x)\,$) is the number:

A desirable situation for the function $\,\phi\,$ is this: when the relative error in $\,\tilde x\,$ is small, so is the relative error in $\,\phi(\tilde x)\,.$ In other words, small input errors lead to small output errors. To quantify this input/output relationship, the idea of a condition number is introduced:

This definition shows that $\,c\,$ gives information about the amplification of relative input errors. If $\,c\,$ is small, then small input errors must produce small output errors. However, if $\,c\,$ is large, or if no such positive number $\,c\,$ exists, then small input errors may produce large output errors.

In numerical problems, where numbers often inherently contain error (due to a forced finite computer representation), such considerations become important.

Note that the definition of condition number is dependent on both the norm in the domain $\,X\,,$ and the norm in the codomain $\,Y\,.$

For an invertible matrix $\,A\,,$ the system $\,Ax = b\,$ will be analyzed as follows: the matrices $\,A\,$ and $\,b\,$ are taken as the ‘inputs’ to the system, and a solution vector $\,x\,$ is the ‘output’.

When the inputs $\,A\,$ and $\,b\,$ change slightly (say, by replacing theoretical matrices $\,A\,$ and $\,b\,$ by their computer representations), then it is desired to study how the solution $\,x\,$ changes.

The Euclidean norm $\,\|\cdot\|\,$ will be used to measure ‘changes’ in the $\,n \times 1\,$ matrix $\,b\,.$ It is also necessary to have a means by which to measure changes in the matrix $\,A\,$; this means will be provided by a matrix norm, defined next. In this next definition, $\,\text{sup}\,$ denotes the supremum, i.e., the least upper bound.

It can be shown that the $2$-norm is indeed a norm on the vector space of all $\,n\times n\,$ matrices. Consequently, if $\,\|A\|_2 = 0\,,$ then $\,A = \boldsymbol{0}\,,$ where $\,\boldsymbol{0}\,$ denotes the $\,n \times n\,$ zero matrix. Thus, if $\,\|A\|_2 = 0\,,$ then $\,A\,$ is not invertible. Equivalently, if $\,A\,$ is invertible, then $ \,\|A\|_2 \ne 0\,.$

There are other matrix norms (see, e.g., [S&B, p. 178]), many of which are more easily calculated than $\,\|\cdot\|_2\,,$ but it will be seen that the $2$-norm is particularly well suited to the current situation.

The reader versed in functional analysis will recognize the $2$-norm as the norm of the bounded linear operator $\,A: \Bbb R^n\rightarrow \Bbb R^n\,,$ when the Euclidean norm is used in $\,\Bbb R^n\,.$

It is immediate from the definition of $\,\|A\|_2\,$ that for any $\,x\ne 0\,,$

so that for all $\,x\,$:

Thus, the number $\|A\|_2\,$ gives the greatest magnification that a vector may attain under the mapping determined by $\,A\,.$ That is, the norm of an output $\,\|Ax\|\,$ cannot exceed $\,\|A\|_2\,$ times the norm of the corresponding input $\,\|x\|\,.$

The $2$-norm is not easy to compute. ($\bigstar$ When $\,A\,$ has real number entries, $\|A\|_2\,$ is the square root of the largest eigenvalue of the matrix $\,A^tA\,.$) However, it is easy to obtain an upper bound for $\,\|A\|_2\,$ as the square root of the sum of the squared entries from $\,A\,,$ as follows:

Let $\,x = (x_1,\ldots,x_n)^t\,,$ so that $\,x\,$ is a column vector, and:

Let $\,\alpha_{ij}\,$ denote the element in row $\,i\,$ and column $\,j\,$ of the matrix $\,A\,.$ The $\,i^{\text{th}}\,$ component of the column vector $\,Ax\,$ is denoted by $\,(Ax)_i\,,$ and is the inner product of the $\,i^{\text{th}}\,$ row of $\,A\,$ with $\,x\,$:

Then:

(The Cauchy-Schwarz Inequality is reviewed in Appendix $2\,.$)

Thus, for any $\,x\ne 0\,,$ division by $\,\|x\|^2\,$ yields

so that taking square roots gives:

The square root of the sum of the squared entries from $\,A\,$ gives another norm of $\,A\,,$ called the Frobenius norm, and denoted by $\,\|A\|_F\,$ [G&VL, p. 56].

With the matrix $2$-norm in hand, the analysis of the system $\,Ax = b\,$ can now begin. It is assumed that $\,A\,$ is an invertible $\,n \times n\,$ matrix with real number entries. It is desired to study the sensitivity of solutions $\,x\,$ of this system to small changes in $\,A\,$ and $\,b\,.$

[S&B, 178–180]   Begin by supposing that only the column vector $\,b\,$ changes slightly; $\,A\,$ remains fixed. What is the corresponding effect on the solution $\,x\,$? The analysis of this situation will motivate the definition of the condition number of a matrix.

Let

be true statements; that is, $\,x\,$ denotes the system solution when column vector $\,b\,$ is used, and $\,\tilde x\,$ denotes the system solution when column vector $\,\tilde b\,$ is used. Define

so that:

Since $\,A\,$ is invertible (by hypothesis):

Taking norms and using property ($1$) gives:

Let $\,x\ne 0\,$; division by the positive number $\,\|x\|\,$ gives:

Since $\,b = Ax\,,$ it follows that $\,\|b\| \le \|A\|_2\cdot\|x\|\,$ and hence $\,\frac1{\|x\|} \le \frac{\|A\|_2}{\|b\|}\,$; substitution into the previous display yields:

This inequality shows that the constant $\,\|A^{-1}\|_2\,\|A\|_2\,$ gives information about how an error in $\,b\,$ is potentially ‘amplified’ to produce an error in the solution $\,x\,$.

Thus, the number $\,\|A^{-1}\|_2\,\|A\|_2\,$ is a condition number for the function that ‘inputs’ a column vector $\,b\,,$ and ‘outputs’ a solution to the system $\,Ax = b\,.$ With this motivation, the next definition should come as no surprise:

When $\,\text{cond}(A)\,$ is close to $\,1\,,$ $\,A\,$ is said to be well-conditioned, and when $\,\text{cond}(A)\,$ is large, $\,A\,$ is said to be ill-conditioned.

It is always true that $\,\text{cond}(A) \ge 1\,,$ as follows. Since $\,\|A\|_2\ge\frac{\|Ay\|}{\|y\|}\,$ for all $\,y\ne 0\,,$ take $\,y\,$ to be $\,A^{-1}x\,,$ thus obtaining

from which:

If $\,A\,$ is an orthogonal matrix (that is, $\,A^t A = I\,,$ or equivalently, $\,A^{-1} = A^t\,$), then it can be shown that $\,\text{cond}(A) = 1\,.$

When MATLAB issues warnings regarding ill-conditioned matrices, it reports the information in terms of the reciprocal of the condition number:

rcond(A) := 1/(cond(A))

If $\,A\,$ is well-conditioned, then  rcond(A)  is near $\,1.0\,$; if $\,A\,$ is ill-conditioned, then  rcond(A)  is near $\,0\,.$

It is important to note that, in the system $\,Ax=b\,,$ the sensitivity of solutions $\,x\,$ to small changes in $\,b\,$ clearly has more to do with $\,A\,$ than with $\,b\,.$

Since $\,A\,$ is invertible if and only if $\,\text{det}(A) \ne 0\,,$ one might conjecture that when the determinant of $\,A\,$ is close to $\,0\,,$ then solutions to $\,Ax = b\,$ are sensitive to small changes in $\,A\,.$ However, this is not the case. The examples below show that $\,\text{det}(A)\,$ can be made as small as desired, while keeping $\,\text{cond}(A) = 1\,.$

The definition of the condition number of a matrix was motivated by studying the sensitivity of solutions to $\,Ax = b\,$ to small changes in $\,b\,.$ Does the number $\,\text{cond}(A)\,,$ as defined, also measure the sensitivity of solutions to changes in both $\,A\,$ and $\,b\,$? The following discussion shows that the answer is ‘Yes’.

[G&VL, 79–80] Allow both $\,A\,$ and $\,b\,$ to vary in the system $\,Ax = b\,$; recall that $\,A\,$ is assumed to be invertible.

Denote a change in $\,A\,$ by $\,\epsilon F\,,$ where $\,F\,$ is an $\,n \times n\,$ constant matrix and $\epsilon\gt 0\,.$ Similarly, denote a change in $\,b\,$ by $\,\epsilon f\,,$ where $\,f\,$ is a fixed $\,n \times 1\,$ matrix. Different values of $\,\epsilon\,$ lead to different approximations $\,\tilde A := A + \epsilon F\,$ and $\,\tilde b := b + \epsilon f\,$ of $\,A\,$ and $\,b\,,$ respectively.

Let $\,x(\epsilon)\,$ denote the solution of the ‘perturbed’ system $\,\tilde Ax = \tilde b\,$; the situation is then summarized by:

Differentiating both sides with respect to $\,\epsilon\,$ yields:

Evaluation at $\,\epsilon = 0\,$ and a slight rearrangement yields:

Or, since $\,A\,$ is invertible:

IN PROGRESS