G1BINM Introduction to Numerical Methods 71

7 Iterative methods for matrix

equations
7.1 The need for iterative methods
We have seen that Gaussian elimination provides a method for nding the exact solution
(if rounding errors can be avoided) of a system of equations Ax = b. However Gaussian
elimination requires approximately n
3
/3 operations (where n is the size of the system),
which may become prohibitively time-consuming if n is very large. Another weakness is
that Gaussian elimination requires us to store all the components of the matrix A. In
many real applications (especially the numerical solution of dierential equations), the
matrix A is sparse, meaning that most of its elements are zero, in which case keeping
track of the whole matrix is wasteful.
In situations like these it may be preferable to adopt a method which produces an
approximate rather than exact solution. We will describe three iterative methods, which
start from an initial guess x
0
and produce successively better approximations x
1
, x
2
, . . . .
The iteration can be halted as soon as an adequate degree of accuracy is obtained, and
the hope is that this takes a signicantly shorter time than the exact method of Gaussian
elimination would require.
7.2 Splitting the matrix
All the methods we will consider involve splitting the matrix A into the dierence
between two new matrices S and T:
A = S T.
Thus the equation Ax = b gives
Sx = Tx +b,
based on which we can try the iteration
Sx
k+1
= Tx
k
+b. (7.1)
Now if this procedure converges, say x
k
x as k , then clearly x solves the
original problem Ax = b, but it is not at all clear from the outset whether a scheme
like (7.1) converges or not.
Evidently there are many possible ways to split the matrix A. The tests of a good
choice are:
72 School of Mathematical Sciences University of Nottingham
the new vector x
k+1
should be easy to compute, that is S should be easily invertible
(for example S might be diagonal or triangular);
the scheme should converge as rapidly as possible towards the true solution.
These two requirements are conicting: a choice of splitting which is particularly easy
to invert (see e.g. Jacobis method below) may not converge especially rapidly (or at
all). At the other extreme we can converge exactly, in just one step, by using S = A,
T = 0; but S = A is usually dicult to invert: thats the whole point of splitting!
It is convenient to introduce the notation
A = L+D+U (= S T),
where L is strictly lower triangular,
D is diagonal,
U is strictly upper triangular.
For example, if
A =
_
_
1 2 3
4 5 6
7 8 9
_
_
,
then
L =
_
_
0 0 0
4 0 0
7 8 0
_
_
, D =
_
_
1 0 0
0 5 0
0 0 9
_
_
, U =
_
_
0 2 3
0 0 6
0 0 0
_
_
.
We will consider three possible splittings.
1. Jacobis method S = D, T = (L+U);
2. The Gauss-Seidel method S = L+D, T = U;
3. Successive over-relaxation (SOR) a combination of 1 and 2.
7.3 Jacobis method
In Jacobis method, S is simply the diagonal part of A. We illustrate it with a simple
two-dimensional example.
Example 7.1 Consider the system
2x y = 3,
x + 2y = 0.
We solve the rst equation for x and the second for y:
x = y/2 + 3/2,
y = x/2,
G1BINM Introduction to Numerical Methods 73
which suggests the iterative scheme
x
k+1
= y
k
/2 + 3/2
y
k+1
= x
k
/2.
Note that this corresponds to Sx
k+1
= Tx
k
+b where the splitting
S =
_
2 0
0 2
_
, T =
_
0 1
1 0
_
,
has been employed.
Suppose we start with the initial guess
_
x
0
y
0
_
=
_
1
1
_
.
Then we generate successively
_
x
1
y
1
_
=
_
2
1/2
_
,
_
x
2
y
2
_
=
_
7/4
1
_
,
_
x
3
y
3
_
=
_
2
7/8
_
,
and so forth, which appears to be converging nicely to the solution x = 2, y = 1.
7.4 The Gauss-Seidel method
A drawback with Jacobis method is that it requires us to store all the components of
x
k
until we have nished computing the next iteration x
k+1
. For very large systems,
the memory required may become a problem. In any case, it would appear to make
more sense to use the new values of x (which presumably are more accurate than the
old values) as soon as weve calculated them. The scheme which results from doing so is
called the Gauss-Seidel method. We illustrate it with the same two-dimensional system
as in example 7.1.
Example 7.2 Consider the same system
2x y = 3
x + 2y = 0,
as in example 7.1. As in Jacobis method we use the rst equation to nd x
k+1
in terms of y
k
:
x
k+1
= y
k
/2 + 3/2.
But now that weve found x
k+1
we use it when working out y
k+1
:
y
k+1
= x
k+1
/2
(Jacobi has x
k
/2 on the right-hand side). In terms of splitting this corresponds to the choice
S =
_
2 0
1 2
_
, T =
_
0 1
0 0
_
,
74 School of Mathematical Sciences University of Nottingham
that is Gauss-Seidel corresponds to making S lower triangular and T strictly upper triangular.
Again we start with the initial guess
_
x
0
y
0
_
=
_
1
1
_
,
and nd that
_
x
1
y
1
_
=
_
2
1
_
,
so for this example, Gauss-Seidel converges to the exact solution after just one iteration.
It is a uke that the scheme in example 7.2 converges in one step, but it is generally
the case that Gauss-Seidel converges better than Jacobi, and on the whole its a better
method.
7.5 Implementation of Gauss-Seidel
Now consider the general n n system
a
11
x
1
+ a
12
x
2
+ . . . + a
1n
x
n
= b
1
,
a
21
x
1
+ a
22
x
2
+ . . . + a
2n
x
n
= b
2
,
.
.
.
.
.
.
.
.
.
a
n1
x
1
+ a
n2
x
2
+ . . . + a
nn
x
n
= b
n
.
The Gauss-Seidel scheme proceeds exactly as in example 7.2. We begin by using the
rst equation to update x
1
:
(x
1
)
new
=
1
a
11
_
_
b
1

j=2
a
1j
x
j
_
_
.
Next we use the second equation to update x
2
:
(x
2
)
new
=
1
a
22
_
_
b
2

j=3
a
2j
x
j
_
_

a
21
a
22
x
1
.
Note that the value of x
1
used in this second step is the new updated value: we dont
bother to store the old value of x
1
.
The right-hand side can be rearranged to
(x
2
)
new
=
1
a
22
_
_
b
2

j=1
a
2j
x
j
+ a
22
x
2
_
_
,
or
(x
2
)
new
= x
2
+
1
a
22
_
_
b
2

j=1
a
2j
x
j
_
_
.
G1BINM Introduction to Numerical Methods 75
This form persists for the subsequent steps, so the general formula for updating x
i
is
(x
i
)
new
= x
i
+
1
a
ii
_
_
b
i

j=1
a
ij
x
j
_
_
. (7.2)
On the right-hand side of equation (7.2), the rst term is the old value of x
i
, and
the second term is the correction, which is added to x
i
to obtain the improved value
(x
i
)
new
. Note also from the form of (7.2) that for the scheme to work we must set up the
system in such a way that its diagonal entries are all nonzero.
We can estimate the number of operation required for the Gauss-Seidel method as
follows. Each step of the form (7.2) requires n multiplications and 1 division, making a
total of n +1 operations. These are required for each of the n components of x, so that
n(n +1) n
2
operations are needed for one complete iteration. Thus the total number
of operations required is approximately n
2
times the number of iterations required for
satisfactory convergence. This should be compared with the n
3
/3 operations needed
for Gaussian elimination. We can observe that Gauss-Seidel (or something similar)
may indeed be preferable to Gaussian elimination if n is very large and the number
of iterations required can be kept reasonably low. We should also observe that in the
operations count carried out above, the number of multiplications required may be
greatly reduced if the matrix A is sparse.
The question arises: how can we tell if the scheme has converged adequately? That
is, how do we decide when enough iterations have been performed? One possibility
is to look at the dierence between successive values of x
k
, since these become closer
as x
k
approaches the solution x. Thus we might, for example, choose to continue the
iterations until |x
k+1
x
k
| is smaller than some prescribed small amount. A slightly
better approach is to use the relative dierence (i.e. compared to the actual size of
the solution) as a measure of how well the scheme has converged. Thus the iteration is
halted when
|x
k+1
x
k
|
|x
k+1
|
< ,
where is some prescribed small number. In the Gauss-Seidel step this is very easy to
calculate, since (x
i
)
k+1
(x
i
)
k
is just the second term on the right-hand side of (7.2).
We implement these ideas in the following fragment of code in FORTRAN90 which
performs a single iteration of the Gauss-Seidel scheme.
!
! This code performs a single Gauss-Seidel iteration
!
error = 0.0
do i = 1, n
temp = b(i)
do j = 1, n
temp = temp - a(i,j) * x(j)
end do
76 School of Mathematical Sciences University of Nottingham
temp = temp / a(i,i)
!
! Now temp contains the correction to be added to x(i),
! that is (x(i))_(k+1) - (x(i))_k
!
error = error + temp**2
x(i) = x(i) + temp
end do
error = sqrt(error)
!
! Now error contains |x_(k+1) - x_k|
!
All that remains is to perform this loop until the relative error is suciently small.
We should also allow for the fact that the scheme may not converge at all. The following
code demonstrates a straightforward implementation, where the iteration step is the
code fragment given above.
. . . preamble . . .
do k = 1, 30
. . . iteration step . . .
if (error / sqrt(DOT_PRODUCT(x,x)) < 0.001) exit
end do
if (k > 30) then ! Scheme hasnt converged
write(*,*) Method failed to converge after 30 iterations
else ! Scheme has converged
write(*,*) Method converged after, k, iterations
write(*,*) Solution is
do i = 1, n
write(*,*) x(i)
end do
end if
. . . etc. . . .
Note that here we use the function DOT PRODUCT to calculate |x
k
|.
7.6 Convergence of Jacobi and Gauss-Seidel
In general our iterative process takes the form (7.1). Of course, for the method to be at
all sensible S must be invertible, so we can write
x
k+1
= S
1
Tx
k
+S
1
b.
G1BINM Introduction to Numerical Methods 77
The combination
_
S
1
T
_
is known as the iteration matrix.
Now we set
x
k
= x +
k
,
where x is the exact solution (i.e. it satises x = S
1
Tx + S
1
b) and
k
is the error
after k iterations. Then
k
satises

k+1
=
_
S
1
T
_

k
,
and the question is does
k
0 as k ?
We will answer this question under the assumption that the iteration matrix S
1
T
has a complete set of n linearly independent eigenvectors, say e
1
, . . . , e
n
, with corre-
sponding eigenvalues
1
, . . . ,
n
. Thus we can use {e
i
} as a basis and write

0
=
n

i=1
c
i
e
i
,
for some constants c
i
, where
0
is the error in the initial guess x
0
. Then

1
= S
1
T
0
=
n

i=1
c
i

i
e
i
,
and in general

k
=
n

i=1
c
i

k
i
e
i
. (7.3)
So
k
tends to zero as k if and only if |
i
| < 1 for all i. That is, the iteration (7.1)
converges if and only if the eigenvalues of the iteration matrix S
1
T all have absolute
value less than one.
Another way of stating this result is as follows. Dene the spectral radius (M) of a
matrix M to be the largest value of |
i
| over the eigenvalues
i
of M:
(M) = max
i
|
i
| .
Then the iterative scheme (7.1) converges if and only if (S
1
T) < 1.
Another important aspect of the spectral radius is its implications for the rate at
which the scheme converges. If (without loss of generality)
1
is the largest eigenvalue
of S
1
T in absolute value (with |
1
| = ) then clearly (7.3) implies that the behaviour
of the error
k
is dominated by
k
1
as k :

k
c
1

k
1
e
1
as k
|
k
| const.
k
as k .
Thus the error is reduced by roughly a factor of at each iteration and so the smaller
is, the faster the scheme converges.
With this in mind, the rate of convergence is often dened as log when < 1.
Finally we reiterate that we have assumed throughout this section that the itera-
tion matrix has a complete set of eigenvectors. This may seem at rst glance to be a
dangerous assumption, but the following points should be borne in mind.
78 School of Mathematical Sciences University of Nottingham
A matrix is guaranteed to have a complete set of eigenvectors so long as all its
eigenvalues are distinct.
Thus the case in which two or eigenvalues are repeated, in which there is the
possibility that the set of eigenvectors might be incomplete, is not generic.
In any case, the results of this section can, with a little more work, be shown to
hold even when S
1
T does not have a complete set of eigenvectors.
Now we illustrate the calculation of the spectral radius for the Jacobi and Gauss-
Seidel schemes considered in examples 7.1 and 7.2.
Example 7.3 The Jacobi iteration from example 7.1,
x
k+1
= y
k
/2 + 3/2,
y
k+1
= x
k
/2,
can be written as x
k+1
= S
1
Tx
k
+S
1
b, where
S
1
T =
_
0 1/2
1/2 0
_
, S
1
b =
_
3/2
0
_
.
The eigenvalues of S
1
T are given by

1/2
1/2

= 0
2
= 1/4 = 1/2.
Thus the spectral radus
Jacobi
= 1/2 < 1 and the scheme therefore converges.
Example 7.4 For the Gauss-Seidel iteration from example 7.2, we have
S =
_
2 0
1 2
_
, T =
_
0 1
0 0
_
S
1
T =
1
4
_
2 0
1 2
__
0 1
0 0
_
=
_
0 1/2
0 1/4
_
,
using the usual formula for the inverse of a 2 2 matrix. The eigenvalues are given by

1/2
0 1/4

= 0 ( 1/4) = 0 = 0 or 1/4,
and the spectral radius
G-S
= 1/4.
Thus for the system considered in these examples the Gauss-Seidel scheme converges
twice as fast as the Jacobi scheme. Recall that the error is roughly speaking reduced
by a factor of at each iteration. Thus it takes two Jacobi steps to reduce the error by
25% while Gauss-Seidel does the same job in one step.
This rule of thumb each Gauss-Seidel step is worth two Jacobi steps holds
for many examples, in particular all convergent two-dimensional systems. However it is
possible to construct larger systems for which Jacobi converges and Gauss-Seidel does
not (or vice versa).
One nal point worth emphasising is that it may not be immediately obvious how
to order the equations. Apart from ensuring that the diagonal entries are all nonzero,
there appears to be quite a bit of freedom (there are n! ways of arranging n equations).
G1BINM Introduction to Numerical Methods 79
Example 7.5 The system from example 7.1 can be rearranged to
x + 2y = 0,
2x y = 3,
which suggests the Jacobi scheme
x
k+1
= 2y
k
,
y
k+1
= 2x
k
3.
The iteration matrix for this scheme is
S
1
T =
_
0 2
2 0
_
,
whose eigenvalues are = 2. Thus = 2 > 1 and, with the equations ordered this way, Jacobis
method diverges. It is straightforward to show that Gauss-Seidel does too.
The situation demonstrated by example 7.5 is quite general. For two-dimensional
systems it is usually the case that one of the obvious Jacobi schemes converges while the
other diverges, and the corresponding Gauss-Seidel schemes behave likewise. For larger
systems it is not obvious in advance whether or not a particular Jacobi or Gauss-Seidel
scheme will converge, or what permutation of the equations gives the best convergence.
However it is clearly a good idea to try to make the diagonal be the largest entry in each
row this will make S
1
T as small as possible.
In practice it is usually not feasible to compute the eigenvalues of S
1
T when the
system is large. Therefore some trial and error may be required to formulate a scheme
like Jacobi or Gauss-Seidel in such a way that it converges. However, for many classes
of matrices which crop up particularly frequently (e.g. in the numerical solution of
dierential equations) the spectral radii of the Jacobi and Gauss-Seidel iteration matrices
are known, and can be shown to be smaller than one.
7.7 Successive over-relaxation (SOR)
The origin of the SOR method is the observation that Gauss-Seidel converges monoton-
ically: the approximations x
k
stay on the same side of the solution x as k increases.
This suggests going slightly further than Gauss-Seidel tells us to at each step. We in-
troduce an over-relaxation or acceleration parameter : when = 1 SOR is identical to
Gauss-Seidel and acceleration is achieved by increasing . The best choice of depends
on the problem being solved, but it always lies between 1 and 2.
For SOR the matrix splitting is
(D+ L) x
k+1
= [(1 )DU] x
k
+ b. (7.4)
Notice that
1. if the method converges, i.e. x
k
x as k , then x satises the original
problem Ax = (L+D+U)x = b;
710 School of Mathematical Sciences University of Nottingham
2. when = 1, (7.4) is identical to the splitting for Gauss-Seidel.
The iteration matrix is
M

= (D+ L)
1
[(1 )DU] ,
and the point of SOR is to discover the value of which minimises the spectral radius
of M

.
Example 7.6 For the same system as in example 7.1,
A =
_
2 1
1 2
_
L =
_
0 0
1 0
_
, D =
_
2 0
0 2
_
, U =
_
0 1
0 0
_
.
Thus the SOR iteration matrix is
M

=
_
2 0
2
_
1
_
2(1 )
0 2(1 )
_
=
1
4
_
2 0
2
__
2(1 )
0 2(1 )
_
=
_
1 /2
(1 )/2 1 +
2
/4
_
.
The eigenvalues of M

are given by

1 /2
(1 )/2 1 +
2
/4

= 0
and thus

2
(2 2 +
2
/4) + ( 1)
2
= 0. (7.5)
When = 1 this gives = 0 and 1/4 as in example 7.4. As is increased, the smaller eigenvalue
increases while the larger one decreases (their product is ( 1)
2
) and it can be shown that the
optimal value of occurs when the two eigenvalues are equal.
Therefore set
1
=
2
= . From the quadratic equation (7.5) we can read o the sum and
product of the eigenvalues:

2
=
2
= ( 1)
2
= 1,
and

1
+
2
= 2 = 2 2 +
2
/4 2( 1) = 2 2 +
2
/4

2
16 + 16 = 0.
The roots of this quadratic are = 4(2

3), of which we take the one between 1 and 2:

= 4(2

3) 1.072
is the optimal value of the acceleration parameter for this system.
At this value of the two equal eigenvalues of M

are
= 1 0.072,
and thus the spectral radius
0.072.
Notice the improvement on Gauss-Seidel, which had = 1/4. For this example SOR is nearly
twice as good again as Gauss-Seidel (since (1/4)
2
= 0.0625 is only slightly less than 0.072).
G1BINM Introduction to Numerical Methods 711
The observation that the optimal value of occurs when two eigenvalues coincide
holds for other systems also. However for large systems it is not practicable to nd the
eigenvalues explicitly and thus calculate the optimum value of as shown in example
7.6. Instead the best choice of must often be determined by trial and error, but its
certainly worth the eort. For large systems the improvement in convergence compared
with Gauss-Seidel is typically much more dramatic than that seen in the two-dimensional
example 7.6: each SOR step could be worth thirty or more Gauss-Seidel steps!
7.8 Implementation of SOR
In general, if A has elements a
ij
then the SOR method is
_
_
_
_
_
a
11
0 0 . . .
a
21
a
22
0 . . .
a
31
a
32
a
33
. . .
.
.
.
.
.
.
.
.
.
_
_
_
_
_
_
_
_
_
_
x
1
x
2
x
3
.
.
.
_
_
_
_
_
k+1
=
_
_
_
_
_
b
1
b
2
b
3
.
.
.
_
_
_
_
_
+
+
_
_
_
_
_
(1 )a
11
a
12
a
13
. . .
0 (1 )a
22
a
23
. . .
0 0 (1 )a
33
. . .
.
.
.
.
.
.
.
.
.
_
_
_
_
_
_
_
_
_
_
x
1
x
2
x
3
.
.
.
_
_
_
_
_
k
.
By rearranging this as we did for Gauss-Seidel, we see that each step of SOR is of the
form
(x
i
)
new
= x
i
+

a
ii
_
_
b
i

j=1
a
ij
x
j
_
_
. (7.6)
Now the connection with Gauss-Seidel is much clearer: at each step we simply multiply
the Gauss-Seidel correction by the acceleration parameter . Thus it is a simple matter
to modify the code given in section 7.5 to produce an SOR code.
Exercises
7.1. Consider the linear system Ax = b, where A is the matrix
A =
_
_
3 2 1
2 3 2
1 2 3
_
_
.
(a) Derive the iteration matrices for the Jacobi, Gauss-Seidel and SOR methods.
(b) Hence show that the Jacobi method diverges, but the Gauss-Seidel method
converges.
712 School of Mathematical Sciences University of Nottingham
7.2. (a) Write a subroutine which generates the n n nite dierence matrix F
n
whose elements f
ij
are dened by
f
ij
=
_
_
_
2 i = j,
1 i = j 1,
0 otherwise,
for any value of n. For example, when n = 4,
F
4
=
_
_
_
_
2 1 0 0
1 2 1 0
0 1 2 1
0 0 1 2
_
_
_
_
.
Write a program which prints out the matrix F
10
to check that your subrou-
tine works.
(b) Write a program (using the subroutine from 7.2a) which solves the system
of equations F
n
x = b
n
for x using the SOR method, where F
n
is the
nite dierence matrix dened above and b
n
is the n-dimensional vector
b
n
= (1, 1, . . . , 1)
T
. The dimension n, acceleration parameter and desired
accuracy of the solution should be set by the user at run time, and the pro-
gram should output the solution x and the number of iterations required for
convergence. Test your program with the values n = 8, = 1.3, desired
accuracy = 0.0001.
(c) Modify the program written in 7.2b to tabulate the number of iterations
needed to achieve an accuracy of 0.0001 versus , where = 1, 1.05, . . . , 1.95, 2.
Print out the table of values generated when n = 10, and thus estimate the
optimal value of when n = 10. How much better than Gauss-Seidel is SOR
in this case?
7.3. [1998/99 Question 1]
(a) Dene the spectral radius (M) of a square matrix M.
An iterative scheme is given by x
n+1
= b+Mx
n
, where b and x
n
are vectors.
Give the condition under which the iterative scheme converges, in terms of
(M).
(b) Consider the system of equations
2x + 3y = 7,
x y = 1.
(*)
(i). Show that a Jacobi scheme based on writing x = (7 3y)/2, y = x 1
fails to converge.
(ii). Reformulate the problem in such a way that a Jacobi scheme does con-
verge.
G1BINM Introduction to Numerical Methods 713
(iii). Formulate a convergent Gauss-Seidel method for the system (*), showing
that the method is convergent. Compute two iterations of the method,
starting from x = 0, y = 0.
(iv). Use the method obtained in (iii) to write down an S.O.R. scheme for
(*) and determine the value of the acceleration parameter that gives the
fastest convergence.