OpenFOAM Matrix Solution

Mesh & matrix Sparse adressing Preconditioner Krylov-solvers CG BiCG Tutorial:PBiCGSTAB
Linear solvers & preconditioners

GOFUN 2018, Braunschweig
Dr. Thorsten Grahs
Feb. 22, 2018
move http://www.move-csc.de | Dr. Thorsten Grahs — Linear solvers & preconditioners | 1/77
Matrix solver
In this talk we cover
Assembling the matrix

Correspondence between
mesh & matrix
Class lduMatrix
Solver in OpenFOAM
Krylov sub-space methods
BiCG method
Tutorial BiCGSTAB
Matrix solver
Results directly from the discretisation of the governing equa-

tions
Assembled as
fvScalarMatrix
fvVectorMatrix
fvTensorMatrix
Example
fvVectorMatrix UEqn
(
fvm::ddt(U)
+ fvm::div(phi, U)
- fvm::laplacian(nu, U)
);
Matrix solver
An instance of the class

fvScalarMatix (e.g. for pEqn) or
fvVectorMatrix (e.g. for UEqn)
is built, which represents the matrix
(and also rhs, preconditioner and solver)
The coefficients results from the discretisation of the fvm oper-
ators
diagonal entries f.i. from the temporal discretisation
off-diagonal entries from the fluxes from the neighbouring cells
Right hand side from source terms (fvc-operators) or old time
steps
Solving an equation
Solving an equation in OpenFOAM

always follows the same pattern:
Initialisation
Assembling the matrix
Solving the system
Correcting the solution
In the following, we consider the approach
exemplified by the solver icoFoam
Matrix creation & initialisation

Example icoFoam
#include "fvCFD.H"
// * * * * * * * * * * * * * * * * * * * *//
int main(int argc, char *argv[])
Creating the instance
{
#include "setRootCase.H"
fvVectorMatrix UEqn from
#include "createTime.H"
#include "createMesh.H"
the class fvVectorMatrix
#include "createFields.H"
#include "initContinuityErrs.H"
Initialisation with
Info<< "\nStarting time loop\n" << endl; fvm::ddt(U) + fvm::div(phi,U) -
while (runTime.loop()) fvm::laplacian(nu,U)

{
Info<< "Time = " << runTime.timeName() All operators are from fvm
<< nl << endl;
#include "readPISOControls.H" namespace (implicit), i.e. all con-
#include "CourantNo.H"
tribute to the matrix
fvVectorMatrix UEqn
(
fvm::ddt(U)
Solved with rhs -fvc::grad(p) (ex-
+ fvm::div(phi, U) plicit)
- fvm::laplacian(nu, U)
);
solve(UEqn == -fvc::grad(p));
Correcting the solution with the
PISO-Loop
Mesh & matrix structure
Correspondence
Mesh ⇔ Au = b
Obviously, there is a strong correspondence between

mesh and matrix
The number of cells corresponds with the
rank n of the matrix A
And so naturally with the dimension of the solution vector u.
The entries of the matrix stems from the
weights of the numerical schemes
In general, A is a sparse matrix
Mesh example
Relation Matrix & Mesh
The number of diagonal elements equals the number of cells of

the mesh.
The number of non-zero upper and lower elements equal the
number of internal faces of the mesh.
The upper matrix contains values belonging to the owners of
faces.
The lower matrix contains values belonging to the neighbour
cells.
Off-diagonal matrix element contain a non-zero value if a face
is shared between an owner and neighbour cell on that position
(in the mesh)
Matrix structure
Matrix according to the mesh example
10 volumes/cells
36 nodes/corners
47 faces, 13 internal faces
a0,0 a0,1 0 0 0 a0,5 0 0 0 0
 
a1,0 a1,1 a1,2 0 0 0 a1,6 0 0 0 
 
 0 a2,1 a2,2 a2,3 0 0 0 a2,7 0 0 
 
 0 0 a 3,2 a 3,3 a3,4 0 0 0 a3,8 0 
 
 0 0 0 a 4,3 a4,4 0 0 0 0 a4,9 
 
a5,0 0 0 0 0 a5,5 a5,6 0 0 0 
 
 0 a6,1 0 0 0 a6,5 a6,6 a6,7 0 0 
 
 0 0 a7,2 0 0 0 a7,6 a7,7 a7,8 0 
 
 0 0 0 a8,3 0 0 0 a8,7 a8,8 a8,9 
0 0 0 0 a9,4 0 0 0 a9,8 a9,9
lduMatrix
The system, resulting from the FV discretisation, is stored as

type lduMatrix
fvMatrix is a sub class from lduMatrix
In general we deal with sparse matrices.
The non-zero coefficients are stored in three arrays:
lower()
diag()
upper()
For our example we have the following entries for the arrays:
lower () = (a1,0 , a5,0 , a2,1 , a6,1 , a3,2 , a7,2 , a4,3 , a8,3 , a9,4 , a6,5 , a7,6 , a8,7 , a9,8 )
diag () = (a0,0 , a1,1 , a2,2 , a3,3 , a4,4 , a5,5 , a6,6 , a7,7 , a8,8 , a9,8 )
upper () = (a0,1 , a0,5 , a1,2 , a1,6 , a2,3 , a2,7 , a3,4 , a3,8 , a4,9 , a5,6 , a6,7 , a7,8 , a8,9 )
Run order
The order of accessing matrix elements in OpenFOAM is as

follows:
lower(), diag(), upper()
Addressing
The addressing for the non-zero coefficients are stored in three

lists of the class lduAdressing:
lowerAddr()
Keeps the row number of the according element in array lower()
and the column number for array upper()
upperAddr()
Vice versa, i.e. keeps column number for lower() and row
number for upper() entries
ownerStartAddr()
Keeps the information, at which position in
upper() a new row starts
lower() a new column starts
Addressing
The address arrays for the example matrix looks as follows:

upperAddr()
= (0, 0, 1, 1, 2, 2, 3, 3, 4, 5, 6, 7, 8)
lowerAddr()
= (1, 5, 2, 6, 3, 7, 4, 8, 9, 6, 7, 8, 9)
ownerStartAddr() = (0, 2, 4, 6, 8, 9, 10, 11, 12)
lduMatrix
lower(), diag(), upper() and lduAdressing
are function of the matrix class lduMatrix.
lduMatrix has further functions for solving algorithms and statis-
tics i.e. solver, solverPerformance, smoother & preconditioners
Class definition can be found at
$FOAM SRC/OpenFOAM/matrices/lduMatrix
Linear system solver functions

Solver sub-classes in lduMatrix
lduMatrix::solverPerformance (solver statistics)

Class which is returned by each matrix solver. Keeps informa-
tions about the solver characteristics, e.g.
solver name
field/variable name
Initial residual
Final residual
Number of iterations
Sub-classes solver, smoother & preconditioner are virtual
base classes:
They define an interface, which has to be implemented by a
concrete matrix solver
This allows run-time selection for the matrix solver
An implementation of a concrete matrix solver algorithm has to
be derived from the class solver
Matrix manipulation
fvMatrix defines function for accessing matrix elements

tmp<volScalarField> D() returns diagonal elements
tmp<volScalarField> A() returns diagonal elements weighted
with inverse of cell volume
tmp<volScalarField> H() returns off-diagonal elements
The assembled matrix correspondence directly with the fvm
operators
This enables the construction of the matrix system directly from
the implementation of the equation in top-level code
fvm :: ddt(U) + fvm :: div(phi, U) ⇒ Au = b
Test case
Consider heat conduction in a thin rod
The temperature distribution is governed by a Laplace-equation

(From Fourier’s Law)

d dT
k = 0
dx dx
Diameter D = 0.001m2 , Conductivity k = 1000Wm−1 K −1
Test case
Expectations
What do we expect?
Linear distribution inside the rod.

myLaplacianFoam
We change the original OpenFOAM solver laplacianFoam

to a customized (simplified) version: myLaplacianFoam
Clone solver
> run
> mkdir programming
> cd programming
> cp -r $FOAM_SOLVERS/basic/laplacianFoam myLaplacianFoam
Adapt source code & make-files

in myLaplacianFoam/Make/files:
laplacianFoam.C mylaplacianFoam.C
⇒
EXE = $(FOAM_APPBIN)/laplacianFoam EXE = $(FOAM_USER_APPBIN)/mylaplacianFoam
Adapt code
myLaplacianFoam.c
Change name
> mv laplacianFoam.C myLaplacianFoam.C
In myLaplacianFoam.C change
while (simple.loop())
{
Info<< "Time = " << runTime.timeName()
while (runTime.loop())
<< nl << endl;
{
fvScalarMatrix
while (simple.correctNonOrthogonal())
TEqn(fvm::laplacian(k, T));
{
TEqn.solve();
solve
(
forAll(T, cellI)
fvm::ddt(T)
{
);
- fvm::laplacian(DT, T) ⇒ Info<< "X = " <<
mesh.C()[cellI].component(vector::X)
}
<< ", T = " << T[cellI] << endl;
#include "write.H"
}
Info<< "ExecutionTime = "
<< runTime.elapsedCpuTime()
runTime.writeAndEnd();
<< " s"
}
<< " ClockTime = "
<< runTime.elapsedClockTime()
<< " s"
<< nl << endl;
}
Adapt code
createFields.C
In create.C change
Info<< "Reading diffusivity DT\n" << endl; Info<< "Reading diffusivity k\n" << endl;
dimensionedScalar DT
(
⇒ dimensionedScalar k
(
transportProperties.lookup("DT") transportProperties.lookup("k")
); );
Compile the modified code:

> cd myLaplacianFoam
> wmake
Run the test case
> cd rodTemperature
> blockMesh
> myLaplacianFoam
...
Calculating temperature distribution
DICPCG: Solving for T,Initial residual = 1,Final residual = 1.21266e-16,No Iterations 1

X = 0.05, T = 140
X = 0.15, T = 220
X = 0.25, T = 300
X = 0.35, T = 380
X = 0.45, T = 460
End
Result
Result in accordance with our expectations
Extension
Change in myLaplacianFoam.C
#include "fvCFD.H"
#include "simpleMatrix.H"
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
int main(int argc, char *argv[])

{
#include "setRootCase.H"
#include "createTime.H"
#include "createMesh.H"
#include "createFields.H"
// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Info<< "\nCalculating temperature distribution\n" << endl;
while (runTime.loop())
{
fvScalarMatrix TEqn(fvm::laplacian(k, T));
TEqn.solve();
forAll(T, cellI)
{
Info<< "X = " <<
mesh.C()[cellI].component(vector::X)
<< ", T = " << T[cellI] << endl;
}
#include "writeMatCoeffs.H"
Adapt code
Create writeMatCoeffs.C
Investigating the coefficient matrix
// Assigning contribution from BC
label NC = mesh.nCells(); //Number of cells
forAll(T.boundaryField(), patchI)
simpleMatrix<scalar> A(NC); //Coeff.matrix
{
// Initialization of matrix
const fvPatch &pp =
for(label i=0; i<NC; i++)
T.boundaryField()[patchI].patch();
{
forAll(pp, faceI)
A.source()[i] = 0.0;
{
for(label j=0; j<NC; j++)
label cellI = pp.faceCells()[faceI];
{
A[cellI][cellI]
A[i][j] = 0.0;
+= TEqn.internalCoeffs()[patchI][faceI];
}
A.source()[cellI]
}
+= TEqn.boundaryCoeffs()[patchI][faceI];
// Assigning diagonal coefficients
}
}
{
Info << "\n==Coefficients of Matrix A=="
A[i][i] = TEqn.diag()[i];
<< endl;
}
// Assigning off-diagonal coefficients
{
for(label faceI=0;
for(label j=0; j<NC; j++)
faceI<TEqn.lduAddr().lowerAddr().size(); faceI++)
{
{
Info<< A[i][j] << " ";
label l = TEqn.lduAddr().lowerAddr()[faceI];
}
label u = TEqn.lduAddr().upperAddr()[faceI];
Info<< A.source()[i] << endl;
A[l][u] = TEqn.upper()[faceI];
}
A[u][l] = TEqn.upper()[faceI];
Info<< "\n==> Solution: " << A.solve()
}
<< endl;
Running adapted code
Recompile and rerun test case rodTemperature

Solver output
Calculating temperature distribution
DICPCG: Solving for T, Initial residual = 1, Final residual = 1.21266e-16, No Iterations 1

X = 0.05, T = 140
X = 0.15, T = 220
X = 0.25, T = 300
X = 0.35, T = 380
X = 0.45, T = 460
==Coefficient Matrix==
-300 100 0 0 0 -20000
100 -200 100 0 0 0
0 100 -200 100 0 0
0 0 100 -200 100 0
0 0 0 100 -300 -100000
Solution: 5(140 220 300 380 460)
End
Linear systems
Iterative solvers
The system of equation in OpenFOAM is solved by iterative
matrix algorithms
Based on the starting vector x0 the iteration rule
xk+1 = Axk
is applied
The iteration stops, when the residual is smaller than a prescribed
tolerance tol:
kAxk+1 − bk < tol
Sometimes one tries to accelerate the solution behaviour of the

iterative process.
This goal can be achieved by pre-conditioning
Preconditioning
Goal
Enhance matrix properties in order to optimize condition number
(cond(A) close to 1)
⇒ faster convergence of the solver
One is looking for a matrix M with
M−1 Ax = M−1 b
and M easy to invert.

The easiest choice would be M = I
(To simple - yields just the original system)
The best choice would be M = A−1
(To costly to invert A)
Preconditioner in OpenFOAM
diagonalPreconditioner
for symmetric & nonsymmetric matrices (not very effective)
DICPreconditioner
Diagonal Incomplete Cholesky preconditioner
for symmetric matrices
DILUPreconditioner
Diagonal Incomplete LU preconditioner
for nonsymmetric matrices
FDICPreconditioner
Fast Diagonal Incomplete Cholesky preconditioner
GAMGPreconditioner
Geometric Agglomerated algebraic MultiGrid preconditioner
noPreconditioner
Preconditioners
Diagonal preconditioner M = D
Uses the diagonal as preconditioner. Not very efficient.
Incomplete LU preconditioner A = LU + R, M = LU
Apply Gaussian elimination algorithm, but only on allowed pat-
tern ⇒ incomplete LU factorization called ILU.
Reduce in the for-loops the indices to the indices with
– allowed pattern, e.g. ILU(0) for pattern of A
– values that are not too small, ILUT for ILU with treshold
Leads to approximate LU factorization
A = LU + R, preconditioner M = LU
with all ignored fill-in entries collected in R.

Incomplete Cholesky preconditioner A = GGT +R, M = GGT
Similar to ILU, but based on incompleted Cholesky factorization
Matrix solver in OpenFOAM
BICCG
Diagonal incomplete LU preconditioned BiCG solver
diagonalSolver
Solver for symmetric and nonsymmetric matrices

GAMG
Geometric Agglomerated algebraic Multi-Grid solver

ICC
Incomplete Cholesky Conjugate Gradient solver

PBiCG
Bi-Conjugate Gradient solver with preconditioner

PCG
Conjugate Gradient solver with preconditioner

smoothSolver
Iterative solver with run-time selectable smoother

Most of these solvers are Krylov-subspace solvers
Krylov subspace methods
A Krylov1 subspace method is an iterative matrix algorithm for

solving the linear system Ax = b of the form:
xm ∈ x0 +Km , Km = KM (A, r0 ) = span(r0 , Ar0 , . . . , Am−1 r0 )
with
r0 = b − Ax0 , (b − Axm ) ⊥ Lm , Km , Lm ∈ Rm ⊂ Rn
Km represents the Krylov subspace where the solution is sought.

Why is this desirable?
1
Aleksey Nikolaevich Krylov (1863 - 1945), russian naval engineer and mathe-
matician
Krylov method
Motivation
Idea of a Krylov method stems from the fact that

the inverse of a N × N non-singular Matrix A can be expressed
by a polynomial in A of degree (N − 1).
i.e.
A−1 = c0 I + c1 A + c2 A2 + . . . + cN−1 AN−1
This is a direct conclusion from the Cayley-Hamilton theorem
Since the true solution x? is
x? = A−1 b = c0 b + c1 Ab + c2 A2 b + . . . + cN−1 AN−1 b = p(A)
with polynomial p(A) = N−1 i

P
i=0 ci A b
Hence we can approximate the solution x? = A−1 b by applying
a polynomial with m n
We only need matrix-vector multiplication!
Krylov method
The iteration approach
Krylov subspace methods for the linear system
Ax = b
attempt to approximate the solution x = A−1 b with
xj ∈ x0 + Kj , Kj (A, r0 ), r0 = b − Ax0 .
The specific choice of the polynomial (i.e. the coefficients) depends

on the concrete method.
Krylov methods
Remarks
If A has a dominant eigenvector, then it is likely that
Wj = [b, Ab, A2 b, . . . , Aj−1 b]
is very, very ill conditioned for large values of j.

In general, one needs orthonormal vectors {vi }ji=0 such that
Kj (A, b) = span[v0 , v1 , . . . , vj ]
in order to avoid numerical problems.

The vectors vi can be computed using an standard orthogonal-
ization approach, f.i. the Arnoldi algorithm.
Krylov methods
Standard methods
CG: The Conjugate Gradient algorithm applies to systems

where A is symmetric positive definite (SPD)
GMRES: The Generalized Minimal RESidual algorithm is the
first method to try if A is not SPD.
BiCG: The BiConjugate Gradient algorithm applies to general
linear systems, but the convergence can be quite erratic.
BiCGstab: The stabilized version of the BiConjugate Gradient
algorithm.
Remark
We focus her on CG and BiCG. In the tutorial we show how to
extend BiCG to BiCGstab.
The Conjugated Gradient Method (CGM)
CG is the most popular iterative method for solving large systems

of linear equation of the form
Ax = b
where A ∈ Rn×n be symmetric positive definite and b ∈ Rn .

If A would be dense, the best way would be to factor
(LU, QR,..) and solve by back-substitution
For sparse systems CG is your method.
Conjugated Gradients
Associated with the system Ax = b we have a quadratic function,

f , given by
1
f (x) := xT Ax − xT b
2
The most instructive explanation of the CG method starts from
the fact, that (if A is symmetric positive definite) the solution
of
Ax = b
minimizes the quadratic function f .
Why????
To this uses we calculate the first and second order derivatives

of f :
∇f (x) = Ax − b
Hf (x) = A
with
the gradient ∇f = (∂1 f , . . . , ∂n f )
the Hessian Hf = (∂ij2 f )ij .
Since all higher order derivatives vanish, we can rewrite f as its
Taylor polynomial.
Taylor polynomial
We rewrite f as its Taylor polynomial of degree two, i.e.
1
f (x) = f (y) − (x − y)T ∇f (y) + (x − y)T Hf (y)(x − y)
2
1
= f (y) − (x − y) (Ay − b) + (x − y)T A(x − y)
T
2
T
≥ f (y) − (x − y) (Ay − b)
In the last step, we have used that A is positive definite.

If y = x∗ solves Ax = b ⇒ f (x) ≥ f (x∗ ), i.e. x∗ minimises f .
On the other hand, every minimiser y of f has to satisfy the
necessary condition 0 = ∇f (y) = Ay − b.
This is only possible for y = x∗
Example: Graph, Contours Gradient of f
Here we show the graph of

a quadratic form f .
The minimum point of this
surface is x with Ax = b.
Contours of f are ellipsoidals
(i.e. f=const.)
Constructing an iterative method
Now we construct an iterative method which tries to find the

minimum of f .
We start from the iteration rule
xj+1 = xj + αj pj
with search direction pj and step length αj
Question?
How to chose α.
How to choose pj
Determine step length α
Assume for a moment that we already have chosen the search direc-
tion pj .
With given pj the best possible step-length can be determined
by finding the minimum of f along xj + αpj .
Setting ψ(α) = f (xj + αpj ) the necessary condition for a mini-
mum is
0 = ψ 0 (α) = pT
j ∇f (xj + αpj )
= pT
j [A(xj + αpj ) − b]
= pT T T
j Axj + αpj Apj − pj b.
Step-length αj
Theorem
Assume that the search direction pj 6= 0 has already been determined.
Then we can compute the optimal step-length α for the new iteration
xj+1 = xj + αj pj
by
pT
j (b − Axj ) pT
j rj
αj = = (1)
pT
j Apj pT
j Apj
Thus, we have to determine the search direction.
Search direction – 1st try
Use the unity vectors ei as search directions, i.e.
p0 = e1 , p1 = e2 , . . . , pn−1 = en , pn = e1 , pn+1 = e2 , . . .
Than we have
n
X
eT
i Aei = aii and eT
i (Ax − b)ei = aij xj − bi
j=1
Inserting (1) – the definition for α – yields

 
n
1 X
xkk = xk−1 + αk−1 pk−1 = xk−1 −  akj xk−1
j − bl  e k
akk
j=1
for k = 0, 1, . . . , n − 1.
Search direction – 1st try
Problematic:
Only the k-th component of the vector xk is updated.
If we consider one cycle, we get
 
n
1  X X
xk = bk − akj xjk − akj xj0  ,
akk
j<k
xik = xik−1 , k 6= i
Therefore, one cycle is one Gauss-Seidel iteration.
Search direction – 2nd try
One natural way to choose the search direction is the direction

of the steepest descent.
This is given by the negative gradient of the target function:
pj = −∇f (xj )
= −(Axj − b)
= rj
Steepest Descent
Algorithm
Choose x1 and set p1 =
b − Ax1
for j = 1, 2, . . . do
pT rj
αj = pTjAp
j j
xj+1 = xj + αj pj
pj+1 = b−Axj+1
end
Orthogonal search direction
Notice that, by choosing αj and pj as constructed, we have
0 = pT T
j (Axj+1 − b) = −pj pj+1
Thus, the derived search directions are orthogonal

Furthermore, there are successive orthogonal.
On a first glance ...
orthogonal search direction are very good.
On a second view
steepest descent converges rather slowly.
Steepest descent method
Bad if λmax /λmin is large.

Consider Ax = b with

λ1 0 λ1
A= , b= .
0 λ2 λ2
Use start vector x0 = (−9, 1)T . The solution is (1, 1)T .

Looking for = krk k < 10−4 :
Example
Example Let us consider the following setting:

1 0 0 9
A = , b = , x = .
0 9 0 1
Convergence history:

j−1 9
xj = (0.8)
(−1)j−1
Search direction 3rd approach
Idea:
Determine search direction pk such, that pk is
w.r.t. the previous directions p0 , p1 , . . . pk+1
Definition
Two vectors x, y ∈ Rn are called
A-conjugated
if
xT Ay = 0
holds.
A-conjugates search directions
Level curves of f (x) = xT Ax = 12 (x12 + 9x22 )

The level curses of f are el-
lipses.
The orthogonality forces the
search direction tangential
to the level sets
If we introduce A-conjugated
search directions, i.e. sj =
A1/2 pj one finds the mini-
mum in two steps.
We have 0 = sT T
j sj+1 = pj Apj+1
Conjugated Gradient (CG) method
Theorem
Assume p0 , p1 , . . . , pn−1 6= 0 are pairwise A-conjugated vectors.
Then, the scheme
xk+1 = xk + αk pk
with
(rk )T rk
αk = −
(pk )T Apk
converges in at most n steps against the exact solution.
Algorithm: CG method (Hestenes & Stiefel 1952)

Choose x0
Set p0 = r0 = b − Ax0 and j = 0.
while krj k2 > ε do
r T rj
vj = Apj , αj = vTj p
j j
xj+1 = xj + αj pj
rj+1 = rj − αj vj
kr k2
βj = krj+1k2 2
j 2
pj+1 = rj+1 + βj pj
j =j +1
end
Remarks
It can be proved, that the vectors pk are pair-wise A-conjugated.
Theoretically, after at most n steps the solution can be computed.
Due to rounding errors in practice you will not get the solution
after n steps.
In practice we have n 1. Therefore, the CG method is used
as an iterative method.
In each iteration step there are
1 matrix-vector product,
2 scalar products
3 scalar multiplications
necessary.
CG as Krylov subspace method
Theorem
The k-th iteration xk of the CG method minimizes the functional
f (·) w.r.t. the subspace
Kk (A, r0 ) = span{r0 , Ar0 , A2 r0 , . . . , Ak−1 r0 },
i.e. there holds

k−1
X
f (xk ) = min f (x0 + ci Ai r0 ).
ci
i=0
The subspace Kk (A, r0 ) is called the k-th Krylov subspace.
CG method: Error estimate
The error ek := xk − x∗ is measured in the energy norm
kxkA := (xT Ax)1/2
We get the error estimate

Theorem
p !k
∗ κ(A) − 1
k
kx − x kA ≤ 2 p kx0 − x∗ kA
κ(A) + 1
λmax(A)
with κ(A) := cond2 (A) = λmin(A) ≥ 1.
CG method: Example
Consider Ax = b with

λ1 0 λ1
A= , b= .
0 λ2 λ2
Use start vector x0 = (−9, 1)T . The solution is (1, 1)T .

Looking for = krk k < 10−4 :
BiCG solver
Bi-Conjugated Gradient method
The BiCG method belongs to the class of Krylov subspace

methods.
It represents an further development of the CG method.
Results from the simultaneous evaluation of Ax = b and AT x =
b
In the subsequent tutorial we extend the BiCG method to
BiCGstab
Due to this intention we examine the implementation of the
BiCG method in OpenFOAM.
BiCG solverPseudocode BiCG method
Choose ψ 0 ∈ Rn and > 0

r0 = p0 := b − Aψ 0 , w0 := PL r0 , r?0 = p?0 := b − AT ψ 0 , w?0 := PL r?0
r1 = r0 , r?1 = r?0 , j := 1
While krj k2 >
wj := PL rj , w?j := PL r?j
(wj , r?j )2
βj :=
(wj−1 , r?j−1 )2
pj := wj + βj pj−1 , p?j := w?j + βj p?j−1
(wj , r?j )2
αj :=
(Apj , p?j )2
rj+1 := rj − αj Apj , r?j+1 := r?j − αj AT p?j
ψ j := ψ j−1 + αj pj
j := j + 1
PBiCG solver code

PBICG.C
The PBiCG method in OpenFOAM is a modified version. The

order of calculation has been inverted.
ψn in the code equates to the iteration vector xn
// --- Precondition residuals
preconPtr-precondition(wA, rA, cmpt); wj := PL rj
preconPtr->preconditionT(wT, rT, cmpt);
... w?j := PL r?j
// --- Update search directions: (wj , r?j )2
wArT = gSumProd(wA, rT); βj :=
scalar beta = wArT/wArTold; (wj−1 , r?j−1 )2
PBiCG solver code

PBICG.C
for (register label cell=0;
cell<nCells; cell++)
{
pAPtr[cell] = wAPtr[cell]
+ beta*pAPtr[cell]; pj := wj + βj pj−1 ,
pTPtr[cell] = wTPtr[cell]
p?j := w?j + βj p?j−1
+ beta*pTPtr[cell];
}
// --- Update preconditioned residuals
matrix_.Amul(wA,pA interfaceBouCoeffs_,
interfaces_, cmpt);
matrix_.Tmul(wT,pT interfaceIntCoeffs_,
interfaces_, cmpt);
(wj , r?j )2
αj :=
scalar wApT = gSumProd(wA, pT); (Apj , p?j )2
...
// --Update solution and residual:
scalar alpha = wArT/wApT;
PBiCG solver code (cont.)

PBICG.C
for (register label cell=0;

cell<nCells; cell++)
{
psiPtr[cell]+= alpha*pAPtr[cell]; ψ j := ψ j−1 + αj pj
rAPtr[cell] -= alpha*wAPtr[cell];
rj+1 := rj − αj Apj
rTPtr[cell] -= alpha*wTPtr[cell];
}
r?j+1 := r?j − αj AT p?j
BiCG method
For the BiCG method, orthogonality is assured by the Petrov-

Galerkin condition i.e.
r0 residual vector with r0 := b − Ax0
xm ∈ x0 + Km and
Lm = KmT = span{r0 , AT r0 , . . . , (AT )m−1 r0 }
Disadvantages BICG method

There are multiplications with AT needed.
For a regular matrix, the method could terminate without solu-
tion
The method has no minimization properties for the iteration
vector. This could cause an oscillatory behaviour in the conver-

gence
Tutorial PBiCGstab
Motivation
We have just seen that the method has some disadvantages.
We consider the BiCGstab method from van der Vorst [8] which
avoids these disadvantages.
In general, BiCGstab has smoother convergence properties
Multiplication with AT is not necessary.
Furthermore, we want to know how to expand a linear system
solver in OpenFOAM.
Tutorial PBiCGstab
Goal
Goal
Extension of the BiCG solver (PBiCG.C) to the stabilized version
BiCGStab
Proceeding
Change to the directory $WM_PROJECT_USER_DIR$
Clone the folder PBICG
cp -r $FOAM_SRC/OpenFOAM/matrices/lduMatrice/solver/PBICG .
mv PBiCG PBiCGSTAB
cd PBiCGSTAB
Change all names, pre- & post-fixes in PBiCGSTAB.C/.H: string

PBICGSTAB ⇒ PBICGSTAB:
rename ’s/PBiCG/PBiCGSTAB/g’ *
sed -i ’s/PBiCG/PBiCGSTAB/g’ *
Tutorial BiCGstab
Pseudo-code BiCGstab (from: Meister [9])
Choose x0 ∈ Rn und > 0,
r0 = p0 := b − Ax0 , rP0 = pP0 := PL r0 , ρP0 := (rP0 , rP0 )2 , j := 0
While krj k2 >
vj := ApPj , vPj := PL vj
ρPj
αjP := , sj := rj − αjP vj , sPj := PL sj
(vPj , rP0 )2
tj := AsPj , tPj := PL tj
(tPj , sPj )2
ωjP :=
(tPj , tPj )2
xPj+1 := xPj + αjP pPj + ωjP sPj
rj+1 := sj − ωjP tj , rPj+1 := sPj − ωjP tPj
αjP ρPj+1
ρPj+1 := (rPj+1 , rP0 )2 , βjP := P P
ω j ρj
pPj+1 := rPj+1 + βjP (pPj − ωjP vPj ), j := j + 1
xj = PR xPj
Tutorial BiCGstab
Procedure
Translate the pseudo-code of the previous slide into source code

You only have to change the file PBiCGSTAB.C
Use the code from PBiCG.C and his pseudo-code as pattern
If you get in trouble look after the example code BiCGSTAB.C
Try to track the algorithm in BiCGSTAB.C
In OpenFOAM code:
x = psi
ρ = alpha1
r0 = r0
rP = rP
...
Tutorial BiCGstab
Make
Add the following entries to your Make files

files
./PBiCGSTAB.C
LIB = \$(FOAM_USER_LIBBIN)/PBiCGSTAB
options
EXE_INC = -I$(OBJECTS_DIR)
LIB_LIBS = \
$(FOAM_LIBBIN)/libOSspecific.o \
-L$(FOAM_LIBBIN)/dummy -lPstream \
-lz
Or add the entries to the Make file for our common user-defined
library libmylib
Tutorial BiCGstab
PBiCGSTAB.C -- ::solve I
// * * * * * * * * * * * * * * * Member Functions * * * * * * * * * * * * * //
Foam::lduMatrix::solverPerformance Foam::PBiCGSTAB::solve
(
scalarField& psi,
const scalarField& source,
const direction cmpt
) const
{
// --- Setup class containing solver performance data
lduMatrix::solverPerformance solverPerf
(
lduMatrix::preconditioner::getName(controlDict_) + typeName,
fieldName_
);
register label nCells = psi.size();
scalar* __restrict__ psiPtr = psi.begin();

scalarField pA(nCells);
scalar* __restrict__ pAPtr = pA.begin();
scalarField wA(nCells);
scalar* __restrict__ wAPtr = wA.begin();
scalarField vA(nCells);
scalar* __restrict__ vAPtr = vA.begin();
scalarField sA(nCells);
scalar* __restrict__ sAPtr = sA.begin();
scalarField tA(nCells);
scalar* __restrict__ tAPtr = tA.begin();
// --- Calculate A.psi
matrix_.Amul(wA, psi, interfaceBouCoeffs_, interfaces_, cmpt);
Tutorial BiCGstab
PBiCGSTAB.C -- ::solve II
// --- Calculate initial residual
scalarField rA(source - wA);
scalar* __restrict__ rAPtr = rA.begin();
scalarField rP(nCells);
scalar* __restrict__ rPPtr = rP.begin();
scalarField r0(nCells);
scalar* __restrict__ r0Ptr = r0.begin();
// --- Define help-fields

scalarField vP(nCells);
scalar* __restrict__ vPPtr = vP.begin();
scalarField sP(nCells);
scalar* __restrict__ sPPtr = sP.begin();
scalarField tP(nCells);
scalar* __restrict__ tPPtr = tP.begin();
// --- Calculate normalisation factor

scalar normFactor = this->normFactor(psi, source, wA, pA);
if (lduMatrix::debug >= 2)
{
Info<< " Normalisation factor = " << normFactor << endl;
}
// --- Calculate normalised residual norm

solverPerf.initialResidual() = gSumMag(rA)/normFactor;
solverPerf.finalResidual() = solverPerf.initialResidual();
Tutorial BiCGstab
PBiCGSTAB.C -- ::solve III
// --- Check convergence, solve if not converged
if (!solverPerf.checkConvergence(tolerance_, relTol_))
{
// --- Select and construct the preconditioner
autoPtr<lduMatrix::preconditioner> preconPtr =
lduMatrix::preconditioner::New
(
*this,
controlDict_
);
// --- precondition r0
preconPtr->precondition(rP, rA, cmpt);
// --- initialize "pA and r0" with the preconditioned r0

for (register label cell=0; cell<nCells; cell++)
{
pAPtr[cell] = rPPtr[cell];
r0Ptr[cell] = rPPtr[cell];
}
// --- initialize rPr0

scalar rPr0 = gSumProd(rP,r0);
// --- Solver iteration

do
{
// --- Calculate v=A*p and precondition v
matrix_.Amul(vA, pA, interfaceBouCoeffs_, interfaces_, cmpt);
preconPtr->precondition(vP, vA, cmpt);
Tutorial BiCGstab
PBiCGSTAB.C -- ::solve IV
// --- Calculate alpha=rPr0/vPr0
scalar alpha2 = gSumProd(vP, r0);
scalar alpha = rPr0/alpha2;
// --- Calculate s=rA-alpha*v

{
sAPtr[cell] = rAPtr[cell] - alpha*vAPtr[cell];
}
// --- Precondition s
preconPtr->precondition(sP, sA, cmpt);
// --- Calculate t=A*sP and precondition t

matrix_.Amul(tA, sP, interfaceBouCoeffs_, interfaces_, cmpt);
preconPtr->precondition(tP, tA, cmpt);
// --- Calculate alpha1=tPsP/tPtP

scalar tPsP=gSumProd(tP,sP);
scalar tPtP = gSumProd(tP,tP);
if (solverPerf.checkSingularity(mag(tPtP)/normFactor)) break;
scalar alpha1 = tPsP/tPtP;
// --- Update solution and residual

{
psiPtr[cell] += alpha*pAPtr[cell] + alpha1*sPPtr[cell];
rAPtr[cell] = sAPtr[cell] - alpha1*tAPtr[cell];
rPPtr[cell] = sPPtr[cell] - alpha1*tPPtr[cell];
}
Tutorial BiCGstab
PBiCGSTAB.C -- ::solve V
solverPerf.finalResidual() = gSumMag(rA)/normFactor;
// --- Update rPr0 and calculate beta

scalar rPr0old = rPr0;
rPr0 = gSumProd(rP, r0);
scalar beta = (alpha*rPr0)/(alpha1*rPr0old);
// --- Update search directions

{
pAPtr[cell] = rPPtr[cell] + beta*(pAPtr[cell] - alpha1*vPPtr[cell]);
}
} while
(
solverPerf.nIterations()++ < maxIter_
&& !(solverPerf.checkConvergence(tolerance_, relTol_))
);
}
return solverPerf;
}
// ************************************************************************* //
Tutorial BiCGstab
Test case pitzDaily
Solver simpleFoam
Settings
"U|k|epsilon|R"
{
solver PBiCGSTAB;
preconditioner DILU;
tolerance 1e-05;
relTol 0.1;
}
BiCG analogue
Convergence after
BiCG 952 Iterations
BiCGSTAB 754 Iterations
Tutorial BiCGstab
Comparison BiCG/BiCGSTAB – Residuals
Comparison of the residuals BiCG/BiCGSTAB

Literature I
[1] Hirt, C. W., Nichols, B. D.Volume of Fluid (VOF) Method for the Dynamic sof Free Boundaries, Journal of
Computational Physics, 39: 201-225-65, 1981.
[2] Ubbink, O, Numerical prediction of two fluid systems with sharp interfaces, Ph.D Thesis, Imperial College of
Science, Technology and Medicine, London, 1997.
[3] Rusche, H. Computational Fluid Dynamics of Dispersed Two-Phase Flows at High Phase Fractions, Ph.D Thesis,
Imperial College of Science, Technology and Medicine, London, 2002.
[4] Damián, S. M. Description and utilization of interFoam multiphase solver, -Final Work-Computational Fluid
Dynamics. http://infofich.unl.edu.ar/upload/5e6dfd7ff282e2deabe7447979a16d49b0b1b675.pdf
[5] Eslamdoost, A. Forced Roll Motion of a 2D Box and Interaction with Free-Surface, PhD course in CFD with
OpenSource software, Chalmers University, 2009 http://www.tfd.chalmers.se/~hani/kurser/OS_CFD_2009/
ArashEslamdoost/RollMotionofaBoxandInteractionwithFreeSurface.pdf
[6] Issa, R. I. Solution of the implicitly discretised fluid flow equations by operator splitting, Journal of Computational
Physics, 62: 40-65, 1985.
[7] Van Leer, B. Towards the ultimate conservative difference scheme III. Upstream-centered finite-difference
schemes for ideal compressible flow, J. Comp. Phys. 23 (3): 263–275, 1977.
[8] van der Vorst, H. A. BI-CGSTAB: A fast and smoothly converging variant of BI-CG for the solution of nonsym-
metric linear systems, SIAM J. Sci. Stat. Comput., 13: 631-644, 1992.
[9] Meister, A. Numerik linearer Gleichungssysteme, Vieweg Verlag, 2011.

OpenFOAM Matrix Solution

Uploaded by

OpenFOAM Matrix Solution

Uploaded by

Mesh & matrix Sparse adressing Preconditioner Krylov-solvers CG BiCG Tutorial:PBiCGSTAB

Linear solvers & preconditioners

Dr. Thorsten Grahs

Feb. 22, 2018

In this talk we cover

Assembling the matrix

Results directly from the discretisation of the governing equa-

An instance of the class

Solving an equation in OpenFOAM

Matrix creation & initialisation

while (runTime.loop()) fvm::laplacian(nu,U)

Mesh & matrix structure

Obviously, there is a strong correspondence between

Relation Matrix & Mesh

The number of diagonal elements equals the number of cells of

The system, resulting from the FV discretisation, is stored as

The order of accessing matrix elements in OpenFOAM is as

lower(), diag(), upper()

The addressing for the non-zero coefficients are stored in three

The address arrays for the example matrix looks as follows:

Linear system solver functions

lduMatrix::solverPerformance (solver statistics)

fvMatrix defines function for accessing matrix elements

Consider heat conduction in a thin rod

The temperature distribution is governed by a Laplace-equation

Linear distribution inside the rod.

We change the original OpenFOAM solver laplacianFoam

Adapt source code & make-files

Compile the modified code:

DICPCG: Solving for T,Initial residual = 1,Final residual = 1.21266e-16,No Iterations 1

Result in accordance with our expectations

int main(int argc, char *argv[])

Info<< "\nCalculating temperature distribution\n" << endl;

Running adapted code

Recompile and rerun test case rodTemperature

DICPCG: Solving for T, Initial residual = 1, Final residual = 1.21266e-16, No Iterations 1

kAxk+1 − bk < tol

Sometimes one tries to accelerate the solution behaviour of the

and M easy to invert.

with all ignored fill-in entries collected in R.

Matrix solver in OpenFOAM

Solver for symmetric and nonsymmetric matrices

Geometric Agglomerated algebraic Multi-Grid solver

Incomplete Cholesky Conjugate Gradient solver

Bi-Conjugate Gradient solver with preconditioner

Conjugate Gradient solver with preconditioner

Iterative solver with run-time selectable smoother

Krylov subspace methods

A Krylov1 subspace method is an iterative matrix algorithm for

xm ∈ x0 +Km , Km = KM (A, r0 ) = span(r0 , Ar0 , . . . , Am−1 r0 )

Km represents the Krylov subspace where the solution is sought.

Idea of a Krylov method stems from the fact that

x? = A−1 b = c0 b + c1 Ab + c2 A2 b + . . . + cN−1 AN−1 b = p(A)

with polynomial p(A) = N−1 i

Krylov subspace methods for the linear system

attempt to approximate the solution x = A−1 b with

The specific choice of the polynomial (i.e. the coefficients) depends

If A has a dominant eigenvector, then it is likely that

Wj = [b, Ab, A2 b, . . . , Aj−1 b]

is very, very ill conditioned for large values of j.

in order to avoid numerical problems.

CG: The Conjugate Gradient algorithm applies to systems

The Conjugated Gradient Method (CGM)

CG is the most popular iterative method for solving large systems

where A ∈ Rn×n be symmetric positive definite and b ∈ Rn .

Associated with the system Ax = b we have a quadratic function,

To this uses we calculate the first and second order derivatives

We rewrite f as its Taylor polynomial of degree two, i.e.

In the last step, we have used that A is positive definite.

Example: Graph, Contours Gradient of f

Here we show the graph of

Constructing an iterative method

Now we construct an iterative method which tries to find the

with search direction pj and step length αj

Determine step length α

Thus, we have to determine the search direction.

Choose ψ 0 ∈ Rn and > 0

scalar* restrict psiPtr = psi.begin();