[petsc-dev] problem with hypre with '--with-openmp=1'

[Mark Adams]

We are using KNL (Cori) and hypre is not working when configured
with  '--with-openmp=1', even when not using threads (as far as I can tell,
I never use threads).

Hypre is not converging, for instance with an optimized build:

srun -n 1 ./ex56 -pc_type hypre -ksp_monitor -ksp_converged_reason
-ksp_type cg -pc_hypre_type boomeramg
OMP: Warning #239: KMP_AFFINITY: granularity=fine will be used.
  0 KSP Residual norm 7.366251922394e+22
  1 KSP Residual norm 3.676434682799e+22
Linear solve did not converge due to DIVERGED_INDEFINITE_PC iterations 2

Interestingly in debug mode it almost looks good but it is dying:

05:09 nid02516 maint *=
~/petsc_install/petsc/src/ksp/ksp/examples/tutorials$ make
PETSC_DIR=/global/homes/m/madams/petsc_install/petsc-cori-knl-dbg64-intel-omp
PETSC_ARCH="" run
srun -n 1 ./ex56 -pc_type hypre -ksp_monitor -ksp_converged_reason
-ksp_type cg -pc_hypre_type boomeramg
OMP: Warning #239: KMP_AFFINITY: granularity=fine will be used.
  0 KSP Residual norm 7.882081712007e+02
  1 KSP Residual norm 2.500214073037e+02
  2 KSP Residual norm 3.371746347713e+01
  3 KSP Residual norm 2.918759396143e+00
  4 KSP Residual norm 9.006505495017e-01
Linear solve did not converge due to DIVERGED_INDEFINITE_PC iterations 5

This test runs fine on Xeon nodes. I assume that Hypre has been tested on
KNL. GAMG runs fine, of coarse and the initial residual is similar to this
debug run.

Could PETSc be messing up the matrix conversion to hypre '--with-openmp=1' ?

Any ideas?

Thanks,
Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20180622/7b9bf19a/attachment.html>