[petsc-users] unreliable AMG in PETSc

[Arne Morten Kvarving]

hi there.

we (sintef ict) use (parts of) PETSc in our (isogeometric) finite 
element library.
sadly i find that the AMG in petsc is somewhat unreliable and it's a 
pain to deal with users (students) due to this fact.

now, we all know that linear algebra, and AMG in particular, is more art 
than craft at times, certainly to the uneducated, and you can only do 
this much to automate it. but segfaults are bad. preconditioners which 
break the symmetry of the systems are bad. preconditioners that breaks 
the definiteness of systems is also bad. and worse; here it depends on 
the number of MPI processes. meshes are typically targeted a given world 
size and if you are unlucky, this number is one that does not work.

as an example to illustrate this, consider

http://www.math.ntnu.no/~arnemort/petsc-amg-bug.tar.bz2

it's about 30MB due to a somewhat large matrix, hopefully your pipes 
will survive.

this is a (isogeometric) standard Poisson problem using cubic splines, 
with homogenous dirichlet boundaries and a uniform source (f = 1) on a 
mesh intended for simulating flow around wings, but i used the Poisson 
to keep it simple. see

http://www.math.ntnu.no/~arnemort/wing.png

if you want to get an idea.

the tarball has the matrix and vector, as well as a cmake-based 
buildsystem for a simple test app (hacked ex23.c). it contains a number 
of tests (64 in total) to illustrate the unreliable behavior.
in particular it solves the stored linear system using 1 through 32 
processors, with either amg + sor or asm + ilu. the former fails and 
crashes for several runs, while the latter runs through for all (there 
to illustrate that the systems are solvable so that it is not drawn into 
question).

i have tried this with v3.4.2, v3.4.4 and v3.5.2 - they all fail. in 
v3.4.2 one more test fails compared to the others. i have also confirmed 
that i can produce across systems and across toolchains (gcc 4.8 and icc 
14.0). the fails i get are

27:amg_14
39:amg_20
41:amg_21
43:amg_22
45:amg_23
49:amg_25
51:amg_26
53:amg_27
55:amg_28
57:amg_29
61:amg_31
63:amg_32

do not put much thought into the configurations, or the high iteration 
counts. in the "real" code stronger smoothers etc are employed. but 
again i wanted to keep it simple.

i do suspect that eventually the cause is ML. disclaimer: i have never 
been able to (find the energy to) decode the "slightly unreadable" code 
of petsc so i haven't tried to dig.

i do realize this "bug report" has a number of holes (as far as the 
information it contains is entailed), and i can provide whatever you 
need upon request. but i have already sunk too much time into this and 
are behind on my projects :)

usage instructions below.

regards,

arnem

---

usage: requires petsc >= 3.4, some c++ compiler and cmake.

export PETSC_DIR=<dir to petsc>
export PETSC_ARCH=<petsc configuration>

basic usage:
tar jxvf petsc-amg-bug.tar.bz2
cd petsc-amg-bug
cmake . -DCMAKE_BUILD_TYPE=Release
make
make test

for verbose output of the process - replace make test with 'ctest -V'

if your system does not provide the mpi spawner as 'mpirun', you can 
pass -DMPIRUN_COMMAND=<path-to-binary> to cmake. DO NOTE: you cannot use 
petscmpiexec as that does not return the error code to the terminal 
(probably should report that as a separate bug). or well, you can use 
it, but if you do, a failed tests with be reported as a success.

you can execute particular tests only using ctest -R <re to match> -V

to run only the amg tests do

ctest -R amg.* -V

to run only a given amg test do

ctest -R amg_5 -V