[petsc-users] Receiving DIVERGED_PCSETUP_FAILED

[Faraz Hussain]

I have been following ex52.c ksp/ksp/examples/tutorials to use MUMPS to directly solve Ax=b. My matrix is symmetric and positive definite. I built a small cantilever beam model with matrix of 5000^2 size. It solves in 2 seconds and gives correct answer. But when I use a finer mesh of the cantilever beam with 3.3 million^2 matrix, I get the following error:

  Mumps row pivot threshhold = 1e-06
  Mumps determinant = (0., 0.) * 2^0
Linear solve did not converge due to DIVERGED_PCSETUP_FAILED iterations 0
               PCSETUP_FAILED due to FACTOR_OUTMEMORY
Norm of error inf. iterations 0

It runs for more than an hour before aborting with this message. I am running it with this command:

mpiexec -hostfile ./hostfile -np 48 ./ex12 -ksp_converged_reason

My machines have 24 cpus and 125GB Ram. When I do "top" I see it correctly spans 48 processes on 2 nodes. The memory usage of each process is no more than 1-2GB. So I do not understand why it gives FACTOR_OUTMEMORY ?

The same matrix solves in under 5 minutes in Intel Pardiso using 24 cpus on one host. 

I thought maybe mumps thinks it is ill-conditioned? The model does converge in the iterative solver in 4000 iterations. I also tried running with these options per the FAQ on 

 " How can I determine the condition number of a matrix? ".

mpiexec -hostfile ./hostfile -np 48 ./ex12 -pc_type none -ksp_type gmres -ksp_monitor_singular_value -ksp_gmres_restart 1000 -ksp_converged_reason -ksp_monitor_true_residual

After 1337 iterations I cancelled it, and the output was:

1337 KSP preconditioned resid norm 5.647402411074e-02 true resid norm 5.647402411074e-02 ||r(i)||/||b|| 3.993316540960e-03
1337 KSP Residual norm 5.647402411074e-02 % max 1.070324243277e+05 min 1.220336631740e-01 max/min 8.770729448238e+05