7.
2 Direct Methods for Linear Systems with Banded Structure 385
⎛ ⎞
b1 c1 0
⎜ a2 b2 c2 ⎟
⎜ ⎟
⎜ .. ⎟
A=⎜
⎜ a b
3 3 . ⎟
⎟ (7.20)
⎜ . . ⎟
⎝ .. .. c ⎠
n−1
0 an bn
for the matrix elements and starting with u 1 = b1 , these computations are
lk+1 = ak+1 /u k , (7.21)
u k+1 = bk+1 − lk+1 · ck .
After n − 1 steps an LU decomposition A = LU of matrix (7.20) with
⎛ ⎞ ⎛ ⎞
1 0 u 1 c1 0
⎜ l2 1 ⎟ ⎜ .. .. ⎟
⎜ ⎟ ⎜ . . ⎟
L=⎜ . . U =⎜
.. .. ⎟ ⎟
and
⎝ ⎠ ⎝ u n−1 cn−1 ⎠
0 ln 1 0 un
results. The right-hand side y is transformed correspondingly according to
ỹk+1 = yk+1 − lk+1 · ỹk .
The solution x is computed from the upper triangular matrix U by a backward sub-
stitution, starting with xn = ỹn /u n and solving the equations u i xi + ci xi+1 = ỹi one
after another resulting in
ỹi ci
xi = − xi+1 for i = n − 1, . . . , 1 .
ui ui
The computational complexity of the Gaussian elimination is reduced to O(n) for
tridiagonal systems. However, the elimination phase computing lk and u k according
to Eq. (7.21) is inherently sequential, since the computation of lk+1 depends on u k
and the computation of u k+1 depends on lk+1 . Thus, in this form the Gaussian elimi-
nation or LU decomposition has to be computed sequentially and is not suitable for
a parallel implementation.
[Link] Recursive Doubling for Tridiagonal Systems
An alternative approach for solving a linear equation system with tridiagonal matrix
is the method of recursive doubling or cyclic reduction. The methods of recursive
doubling and cyclic reduction also use elimination steps but contain potential par-
allelism [72, 71]. Both techniques can be applied if the coefficient matrix is either
symmetric and positive definite or diagonal dominant [115]. The elimination steps
386 7 Algorithms for Systems of Linear Equations
in both methods are applied to linear equation systems Ax = y with the matrix
structure shown in (7.20), i.e.,
b1 x1 + c1 x2 = y1 ,
ai xi−1 + bi xi + ci xi+1 = yi for i = 2, . . . , n − 1,
an xn−1 + bn xn = yn .
The method, which was first introduced by Hockney and Golub in [91], uses two
equations i − 1 and i + 1 to eliminate the variables xi−1 and xi+1 from equation
i. This results in a new equivalent equation system with a coefficient matrix with
three non-zero diagonals where the diagonals are moved to the outside. Recursive
doubling and cyclic reduction can be considered as two implementation variants for
the same numerical idea of the method of Hockney and Golub. The implementation
of recursive doubling repeats the elimination step, which finally results in a matrix
structure in which only the elements in the principal diagonal are non-zero and the
solution vector x can be computed easily. Cyclic reduction is a variant of recursive
doubling which also eliminates variables using neighboring rows. But in each step
the elimination is only applied to half of the equations and, thus, less computations
are performed. On the other hand, the computation of the solution vector x requires
a substitution phase.
We would like to mention that the terms recursive doubling and cyclic reduction
are used in different ways in the literature. Cyclic reduction is sometimes used for
the numerical method of Hockney and Golub in both implementation variants, see
[60, 115]. On the other hand the term recursive doubling (or full recursive doubling)
is sometimes used for a different method, the method of Stone [168]. This method
applies the implementation variants sketched above in Eq. (7.21) resulting from
the Gaussian elimination, see [61, 173]. In the following, we start the description of
recursive doubling for the method of Hockney and Golub according to [61] and [13].
Recursive doubling considers three neighboring equations i − 1, i, i + 1 of
the equation system Ax = y with coefficient matrix A in the form (7.20) for
i = 3, 4, . . . , n − 2. These equations are
ai−1 xi−2 + bi−1 xi−1 + ci−1 xi = yi−1 ,
ai xi−1 + bi xi + ci xi+1 = yi ,
ai+1 xi + bi+1 xi+1 + ci+1 xi+2 = yi+1 .
Equation i − 1 is used to eliminate xi−1 from the ith equation and equation i + 1 is
used to eliminate xi+1 from the ith equation. This is done by reformulating equations
i − 1 and i + 1 to
yi−1 ai−1 ci−1
xi−1 = − xi−2 − xi ,
bi−1 bi−1 bi−1
yi+1 ai+1 ci+1
xi+1 = − xi − xi+2
bi+1 bi+1 bi+1
and inserting those descriptions of xi−1 and xi+1 into equation i. The resulting new
equation i is
7.2 Direct Methods for Linear Systems with Banded Structure 387
ai(1) xi−2 + bi(1) xi + ci(1) xi+2 = yi(1) (7.22)
with coefficients
ai(1) = αi(1) · ai−1 ,
bi(1) = bi + αi(1) · ci−1 + βi(1) · ai+1 ,
ci(1) = βi(1) · ci+1 , (7.23)
yi(1) = yi + αi(1) · yi−1 + βi(1) · yi+1 ,
and
αi(1) := −ai /bi−1 ,
βi(1) := −ci /bi+1 .
For the special cases i = 1, 2, n − 1, n, the coefficients are given by
b1(1) = b1 + β1(1) · a2 , y1(1) = y1 + β1(1) · y2 ,
bn(1) = bn + αn(1) · cn−1 , yn(1) = bn + αn(1) · yn−1 ,
a1(1) = a2(1) = 0, and cn−1 (1)
= cn(1) = 0 .
(1)
The values for an−1 , an(1) , b2(1) , bn−1
(1)
, c1(1) , c2(1) , y2(1) , and yn−1
(1)
are defined as in
Eq. (7.23). Equation (7.22) forms a linear equation system A(1) x = y (1) with a
coefficient matrix
⎛ (1) ⎞
b1 0 c1(1) 0
⎜ 0 b (1)
0 c2 (1) ⎟
⎜ 2 ⎟
⎜ (1) . . ⎟
⎜a 0 b3 (1) . . . . ⎟
⎜ 3 ⎟
A =⎜
(1)
.. .. .. ⎟.
⎜ (1)
. . (1) ⎟
. cn−2 ⎟
⎜ a4
⎜ .. .. .. ⎟
⎝ . . . 0 ⎠
0 an(1) 0 bn(1)
Comparing the structure of A(1) with the structure of A, it can be seen that the
diagonals are moved to the outside.
In the next step, this method is applied to the equations i − 2, i, i + 2 of the
equation system A(1) x = y (1) for i = 5, 6, . . . , n − 4. Equation i − 2 is used to
eliminate xi−2 from the ith equation and equation i + 2 is used to eliminate xi+2
from the ith equation. This results in a new ith equation
ai(2) xi−4 + bi(2) xi + ci(2) xi+4 = yi(2) ,
which contains the variables xi−4 , xi , and xi+4 . The cases i = 1, . . . , 4, n − 3, . . . , n
are treated separately as shown for the first elimination step. Altogether a next equa-
tion system A(2) x = y (2) results in which the diagonals are further moved to the
outside. The structure of A(2) is
388 7 Algorithms for Systems of Linear Equations
⎛ ⎞
b1(2) 0 0 0 c1(2) 0
⎜ ⎟
⎜ 0 b2(2) c2(2) ⎟
⎜ ⎟
⎜ .. .. ⎟
⎜ 0 . . ⎟
⎜ ⎟
⎜ .. (2) ⎟
⎜ 0 . cn−4 ⎟
A(2) =⎜
⎜ (2) ..
⎟ .
⎟
⎜ a5 . 0 ⎟
⎜ ⎟
⎜ .. ⎟
⎜ a6(2) . 0 ⎟
⎜ ⎟
⎜ .. .. ⎟
⎝ . . 0 ⎠
0 an(2) 0 0 0 bn(2)
The following steps of the recursive doubling algorithm apply the same method
to the modified equation system of the last step. Step k transfers the side diagonals
2k − 1 positions away from the main diagonal, compared to the original coefficient
matrix. This is reached by considering equations i − 2k−1 , i, i + 2k−1 :
(k−1) (k−1) (k−1) (k−1)
ai−2k−1 x i−2k + bi−2k−1 x i−2k−1 + ci−2k−1 x i = yi−2k−1 ,
ai(k−1) xi−2k−1 + bi(k−1) xi + ci(k−1) xi+2k−1 = yi(k−1) ,
(k−1) (k−1) (k−1) (k−1)
ai+2k−1 x i + bi+2k−1 x i+2k−1 + ci+2k−1 x i+2k = yi+2k−1 .
Equation i − 2k−1 is used to eliminate xi−2k−1 from the ith equation and equation
i + 2k−1 is used to eliminate xi+2k−1 from the ith equation. Again, the elimination is
performed by computing the coefficients for the next equation system. These coef-
ficients are
ai(k) = αi(k) · ai−2
(k−1)
k−1 for i = 2k + 1, . . . , n, and ai(k) = 0 otherwise,
ci(k) = βi(k) · ci+2
(k−1)
k−1 for i = 1, . . . , n − 2k , and ci(k) = 0 otherwise, (7.24)
bi(k) = αi(k) · (k−1)
ci−2 k−1 + bi(k−1) + βi(k) · ai+2
(k−1)
k−1 for i = 1, . . . , n ,
yi(k) = αi(k) · (k−1)
yi−2k−1 + yi(k−1) (k) (k−1)
+ βi · yi+2k−1 for i = 1, . . . , n
with
αi(k) := −ai(k−1) /bi−2
(k−1)
k−1 for i = 2k−1 + 1, . . . , n , (7.25)
βi(k) := −ci(k−1) /bi+2
(k−1)
k−1 for i = 1, . . . , n − 2 k−1
.
The modified equation i results by multiplying equation i − 2k−1 from step k − 1
with αi(k) , multiplying equation i + 2k−1 from step k − 1 with βi(k) , and adding both
to equation i. The resulting ith equation is
ai(k) xi−2k + bi(k) xi + ci(k) xi+2k = yi(k) (7.26)
7.2 Direct Methods for Linear Systems with Banded Structure 389
with the coefficients (7.24). The cases k = 1, 2 are special cases of this formula.
The initialization for k = 0 is the following:
ai(0) = ai for i = 2, . . . , n ,
bi(0) = bi for i = 1, . . . , n ,
ci(0) = ci for i = 1, . . . , n − 1 ,
yi(0) = yi for i = 1, . . . , n .
and a1(0) = 0, cn(0) = 0. Also, for the steps k = 0, . . . , log n and i ∈ Z \ {1, . . . , n}
the values
ai(k) = ci(k) = yi(k) = 0 ,
bi(k) = 1 ,
xi = 0
are set. After N = log n steps, the original matrix A is transformed into a diagonal
matrix A(N )
A(N ) = diag(b1(N ) , . . . , bn(N ) )
in which only the main diagonal contains non-zero elements. The solution x of the
linear equation system can be directly computed using this matrix and the corre-
spondingly modified vector y (N ) :
xi = yi(N ) /bi(N ) for i = 1, 2, . . . , n .
To summarize, the recursive doubling algorithm consists of two main phases:
1. Elimination phase: Compute the values ai(k), bi(k), ci(k), and yi(k) for k =1, . . . ,log n
and i = 1, . . . , n according to Eqs. (7.24) and (7.25).
2. Solution phase: Compute xi = yi(N ) /bi(N ) for i = 1, . . . , n with N = log n.
The first phase consists of log n steps where in each step O(n) values are com-
puted. The sequential asymptotic runtime of the algorithm is therefore O(n · log n)
which is asymptotically slower than the O(n) runtime for the Gaussian elimination
approach described earlier. The advantage is that the computations in each step of
the elimination and the substitution phase are independent and can be performed in
parallel. Figure 7.10 illustrates the computations of the recursive doubling algorithm
and the data dependencies between different steps.
[Link] Cyclic Reduction for Tridiagonal Systems
The recursive doubling algorithm offers a large degree of potential parallelism but
has a larger computational complexity than the Gaussian elimination caused by
390 7 Algorithms for Systems of Linear Equations
i=1
i=2
i=3
i=4
i=5
i=6
i=7
i=8
k=0 k=1 k=2 k=3
Fig. 7.10 Dependence graph for the computation steps of the recursive doubling algorithm in the
case of three computation steps and eight equations. The computations of step k are shown in
column k of the illustration. Column k contains one node for each equation i, thus representing the
computation of all coefficients needed in step k. Column 0 represents the data of the coefficient
matrix of the linear system. An edge from a node i in step k to a node j in step k + 1 means that
the computation at node j needs at least one coefficient computed at node i
computational redundancy. The cyclic reduction algorithm is a modification of
recursive doubling which reduces the amount of computations to be performed. In
each step, half the variables in the equation system are eliminated which means that
only half of the values ai(k) , bi(k) , ci(k) , and yi(k) are computed. A substitution phase
is needed to compute the solution vector x. The elimination and the substitution
phases of cyclic reduction are described by the following two phases:
1. Elimination phase: For k = 1, . . . , log n compute ai(k) , bi(k) , ci(k) , and yi(k) with
i = 2k , . . . , n and step size 2k . The number of equations of the form (7.26) is
reduced by a factor of 1/2 in each step. In step k = log n there is only one
equation left for i = 2 N with N = log n .
2. Substitution phase: For k = log n , . . . , 0 compute xi according to Eq. (7.26)
for i = 2k , . . . , n with step size 2k+1 :
yi(k) − ai(k) · xi−2k − ci(k) · xi+2k
xi = . (7.27)
bi(k)
Figure 7.11 illustrates the computations of the elimination and the substitution
phases of cyclic reduction represented by nodes and their dependencies represented
by arrows. In each computation step k, k = 1, . . . , log n , of the elimination phase,
7.2 Direct Methods for Linear Systems with Banded Structure 391
i=1 x1
i=2 x2
i=3 x3
i=4 x4
i=5 x5
i=6 x6
i=7 x7
i=8 x8
k=0 k=1 k=2 k=3
Fig. 7.11 Dependence graph illustrating the dependencies between neighboring computation steps
of the cyclic reduction algorithm for the case of three computation steps and eight equations in
analogy to the representation in Fig. 7.10. The first four columns represent the computations of the
coefficients. The last columns in the graph represent the computation of the solution vector x in
the second phase of the cyclic reduction algorithm, see (7.27)
there are n/2k nodes representing the computations for the coefficients of one equa-
tion. This results in
n n n n
log n
1
+ + + ··· + N = n · i
≤n
2 4 8 2 i=1
2
computation nodes with N = log n and, therefore, the execution time of cyclic
reduction is O(n). Thus, the computational complexity is the same as for the
Gaussian elimination; however, the cyclic reduction offers potential parallelism
which can be exploited in a parallel implementation as described in the following.
The computations of the numbers αi(k) , βi(k) require a division by bi(k) and, thus,
cyclic reduction as well as recursive doubling is not possible if any number bi(k) is
zero. This can happen even when the original matrix is invertible and has non-zero
diagonal elements or when the Gaussian elimination can be applied without pivot-
ing. However, for many classes of matrices it can be shown that a division by zero
is never encountered. Examples are matrices A which are symmetric and positive
definite or invertible and diagonally dominant, see [61] or [115] (using the name
odd–even reduction). (A matrix A is symmetric if A = A T and positive definite if
x T Ax > 0 for all x. A matrix is diagonally dominant if in each row the absolute
value of the diagonal element exceeds the sum of the absolute values of the other
elements in the row without the diagonal in the row.)
