[petsc-users] Meaning of error message (gamg & fieldsplit related)
Barry Smith
bsmith at mcs.anl.gov
Wed Aug 17 00:44:25 CDT 2016
I first set the option -ksp_converged_reason and obtained the message
$ ./ex1
Residual norms for fieldsplit_PA_ solve.
0 KSP unpreconditioned resid norm 2.288897733893e+03 true resid norm 2.288897733893e+03 ||r(i)||/||b|| 1.000000000000e+00
Linear solve did not converge due to DIVERGED_PCSETUP_FAILED iterations 0
PCSETUP_FAILED due to SUBPC_ERROR
so it is failing in building the preconditioner since it didn't even work for the first fieldsplit I guessed that it failed in the gamg on the first fieldsplit so ran with
-fieldsplit_PA_ksp_error_if_not_converged
and got
$ ./ex1
Residual norms for fieldsplit_PA_ solve.
0 KSP unpreconditioned resid norm 2.288897733893e+03 true resid norm 2.288897733893e+03 ||r(i)||/||b|| 1.000000000000e+00
Linear solve did not converge due to DIVERGED_PCSETUP_FAILED iterations 0
PCSETUP_FAILED due to SUBPC_ERROR
~/Src/petsc/test-dir (master=) arch-master-basic
$ ./ex1 -fieldsplit_PA_ksp_error_if_not_converged
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR:
[0]PETSC ERROR: KSPSolve has not converged
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.7.3-1182-g00a02d5 GIT Date: 2016-08-16 15:09:17 -0500
[0]PETSC ERROR: ./ex1 on a arch-master-basic named Barrys-MacBook-Pro.local by barrysmith Wed Aug 17 00:35:16 2016
[0]PETSC ERROR: Configure options --with-mpi-dir=/Users/barrysmith/libraries
[0]PETSC ERROR: #1 KSPSolve() line 850 in /Users/barrysmith/Src/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #2 PCGAMGOptProlongator_AGG() line 1221 in /Users/barrysmith/Src/petsc/src/ksp/pc/impls/gamg/agg.c
looking at the code and seeing the options prefix for this KSP solve I ran with
-fieldsplit_PA_ksp_error_if_not_converged -fieldsplit_PA_gamg_est_ksp_monitor_true_residual
and got
$ ./ex1 -fieldsplit_PA_ksp_error_if_not_converged -fieldsplit_PA_gamg_est_ksp_monitor_true_residual
Residual norms for fieldsplit_PA_gamg_est_ solve.
0 KSP none resid norm 7.030417576826e+07 true resid norm 1.006594247197e+01 ||r(i)||/||b|| 1.000000000000e+00
1 KSP none resid norm 6.979279406029e+07 true resid norm 1.150138107009e+01 ||r(i)||/||b|| 1.142603497100e+00
2 KSP none resid norm 6.979246564783e+07 true resid norm 6.970771666727e+03 ||r(i)||/||b|| 6.925105807166e+02
3 KSP none resid norm 6.978033367036e+07 true resid norm 9.958555706490e+02 ||r(i)||/||b|| 9.893316730368e+01
4 KSP none resid norm 6.977995917588e+07 true resid norm 1.095475380870e+04 ||r(i)||/||b|| 1.088298869103e+03
5 KSP none resid norm 6.954940040289e+07 true resid norm 2.182804459638e+04 ||r(i)||/||b|| 2.168504802919e+03
6 KSP none resid norm 6.905975832912e+07 true resid norm 4.557801389945e+04 ||r(i)||/||b|| 4.527943014412e+03
7 KSP none resid norm 6.905788649989e+07 true resid norm 4.476060162996e+04 ||r(i)||/||b|| 4.446737278162e+03
8 KSP none resid norm 5.464732207984e+07 true resid norm 7.801211607942e+04 ||r(i)||/||b|| 7.750105496491e+03
9 KSP none resid norm 5.393328767072e+07 true resid norm 8.529925739695e+04 ||r(i)||/||b|| 8.474045787013e+03
10 KSP none resid norm 5.294387823310e+07 true resid norm 8.380999411358e+04 ||r(i)||/||b|| 8.326095082197e+03
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR:
[0]PETSC ERROR: KSPSolve has not converged
the smoother is simply not working AT ALL on your first sub matrix so I ran with -fieldsplit_PA_gamg_est_ksp_pmat_view and loaded the resulting matrix from the file binaryoutput into matlab and checked its eigenvalues
>> e = eig(full(a))
e =
5.5260 +85.4938i
5.5260 -85.4938i
-11.6673 + 0.0000i
-6.5409 + 0.0000i
6.5240 + 0.0000i
-2.5377 + 2.0951i
-2.5377 - 2.0951i
3.0712 + 0.0000i
-0.5365 + 0.9521i
-0.5365 - 0.9521i
1.0710 + 0.0000i
-0.9334 + 0.0000i
0.7608 + 0.0000i
0.4337 + 0.0000i
-0.4558 + 0.0000i
-0.4011 + 0.0000i
0.1212 + 0.0000i
0.0327 + 0.0338i
0.0327 - 0.0338i
-0.0426 + 0.0000i
-0.0360 + 0.0334i
-0.0360 - 0.0334i
0.0308 + 0.0000i
-0.0003 + 0.0327i
-0.0003 - 0.0327i
-0.0323 + 0.0000i
0.0120 + 0.0185i
0.0120 - 0.0185i
-0.0256 + 0.0000i
-0.0225 + 0.0000i
0.0152 + 0.0000i
-0.0083 + 0.0125i
-0.0083 - 0.0125i
-0.0177 + 0.0000i
-0.0175 + 0.0000i
-0.0177 + 0.0000i
-0.0158 + 0.0000i
-0.0176 + 0.0000i
-0.0136 + 0.0038i
-0.0136 - 0.0038i
0.0125 + 0.0000i
-0.0080 + 0.0069i
-0.0080 - 0.0069i
0.0066 + 0.0075i
0.0066 - 0.0075i
0.0097 + 0.0000i
0.0039 + 0.0085i
0.0039 - 0.0085i
0.0070 + 0.0000i
-0.0095 + 0.0011i
-0.0095 - 0.0011i
-0.0064 + 0.0000i
0.0024 + 0.0036i
0.0024 - 0.0036i
0.0042 + 0.0000i
0.0042 - 0.0000i
0.0040 + 0.0000i
-0.0035 + 0.0021i
-0.0035 - 0.0021i
-0.0004 + 0.0038i
-0.0004 - 0.0038i
-0.0040 + 0.0000i
-0.0037 + 0.0000i
-0.0036 + 0.0000i
-0.0037 + 0.0000i
-0.0033 + 0.0000i
-0.0033 + 0.0000i
-0.0033 + 0.0000i
-0.0029 + 0.0000i
0.0015 + 0.0014i
0.0015 - 0.0014i
0.0004 + 0.0022i
0.0004 - 0.0022i
0.0003 + 0.0024i
0.0003 - 0.0024i
-0.0012 + 0.0018i
-0.0012 - 0.0018i
0.0003 + 0.0023i
0.0003 - 0.0023i
-0.0022 + 0.0000i
-0.0024 + 0.0000i
-0.0024 - 0.0000i
-0.0021 + 0.0000i
0.0010 + 0.0010i
0.0010 - 0.0010i
0.0009 + 0.0009i
0.0009 - 0.0009i
-0.0002 + 0.0014i
-0.0002 - 0.0014i
-0.0013 + 0.0011i
-0.0013 - 0.0011i
-0.0011 + 0.0012i
-0.0011 - 0.0012i
-0.0014 + 0.0008i
-0.0014 - 0.0008i
-0.0012 + 0.0008i
-0.0012 - 0.0008i
-0.0014 + 0.0004i
-0.0014 - 0.0004i
-0.0005 + 0.0011i
-0.0005 - 0.0011i
0.0007 + 0.0009i
0.0007 - 0.0009i
-0.0010 + 0.0001i
-0.0010 - 0.0001i
0.0008 + 0.0006i
0.0008 - 0.0006i
0.0011 + 0.0001i
0.0011 - 0.0001i
0.0010 + 0.0001i
0.0010 - 0.0001i
0.0010 + 0.0000i
0.0010 - 0.0000i
0.0003 + 0.0008i
0.0003 - 0.0008i
-0.0001 + 0.0009i
-0.0001 - 0.0009i
-0.0008 + 0.0000i
-0.0008 + 0.0005i
-0.0008 - 0.0005i
-0.0004 + 0.0007i
-0.0004 - 0.0007i
-0.0008 + 0.0002i
-0.0008 - 0.0002i
-0.0000 + 0.0008i
-0.0000 - 0.0008i
0.0008 + 0.0000i
0.0008 - 0.0000i
-0.0008 + 0.0000i
-0.0007 + 0.0000i
-0.0004 + 0.0005i
-0.0004 - 0.0005i
0.0005 + 0.0003i
0.0005 - 0.0003i
0.0006 + 0.0001i
0.0006 - 0.0001i
0.0006 + 0.0000i
-0.0005 + 0.0003i
-0.0005 - 0.0003i
0.0003 + 0.0005i
0.0003 - 0.0005i
-0.0001 + 0.0005i
-0.0001 - 0.0005i
0.0000 + 0.0006i
0.0000 - 0.0006i
-0.0004 + 0.0000i
-0.0004 + 0.0000i
0.0003 + 0.0003i
0.0003 - 0.0003i
0.0003 + 0.0002i
0.0003 - 0.0002i
-0.0004 + 0.0000i
-0.0000 + 0.0004i
-0.0000 - 0.0004i
-0.0002 + 0.0003i
-0.0002 - 0.0003i
-0.0001 + 0.0003i
-0.0001 - 0.0003i
0.0003 + 0.0001i
0.0003 - 0.0001i
-0.0003 + 0.0000i
-0.0003 - 0.0000i
0.0001 + 0.0002i
0.0001 - 0.0002i
0.0002 + 0.0000i
0.0002 + 0.0001i
0.0002 - 0.0001i
0.0001 + 0.0002i
0.0001 - 0.0002i
0.0002 + 0.0000i
-0.0002 + 0.0000i
-0.0002 - 0.0000i
0.0001 + 0.0000i
0.0000 + 0.0001i
0.0000 - 0.0001i
0.0001 + 0.0001i
0.0001 - 0.0001i
-0.0001 + 0.0000i
-0.0001 + 0.0000i
-0.0001 - 0.0000i
0.0001 + 0.0001i
0.0001 - 0.0001i
0.0001 + 0.0000i
0.0001 - 0.0000i
-0.0001 + 0.0001i
-0.0001 - 0.0001i
0.0001 + 0.0001i
0.0001 - 0.0001i
0.0001 + 0.0000i
0.0000 + 0.0001i
0.0000 - 0.0001i
0.0000 + 0.0001i
0.0000 - 0.0001i
-0.0000 + 0.0001i
-0.0000 - 0.0001i
-0.0001 + 0.0000i
-0.0001 - 0.0000i
-0.0001 + 0.0000i
-0.0001 - 0.0000i
-0.0001 + 0.0000i
-0.0001 - 0.0000i
0.0001 + 0.0000i
0.0001 - 0.0000i
-0.0001 + 0.0000i
-0.0001 - 0.0000i
-0.0001 + 0.0000i
0.0000 + 0.0001i
0.0000 - 0.0001i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
0.0000 + 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
0.0000 + 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
-0.0000 + 0.0000i
-0.0000 + 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
-0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
0.0000 + 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
-0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
0.0000 + 0.0000i
-0.0000 + 0.0000i
-0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
0.0000 + 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
-0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
-0.0000 + 0.0000i
-0.0000 - 0.0000i
-0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 - 0.0000i
-0.0000 + 0.0000i
0.0000 + 0.0000i
-0.0000 + 0.0000i
0.0000 + 0.0000i
-0.0000 + 0.0000i
-0.0000 + 0.0000i
0.0000 + 0.0000i
-0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
0.0000 + 0.0000i
The first matrix you are handing off to GAMG is totally inappropriate for GAMG it has a huge range of eigenvalue scales both positive and negative and many zero eigenvectors. The smoother GAMG uses does not work on the matrix nor will GAMG work on the matrix.
Hence the entire nested fieldsplit preconditioner cannot work on the entire matrix since the inner GAMG preconditioner cannot work on the inner matrix, in fact probably next to no preconditioner will work on the inner matrix.
You need to go back and make sure that either 1) the matrices being generated are what they should be and 2) your assumption that GAMG would work on the first submatrix makes sense
Barry
> On Aug 16, 2016, at 5:21 PM, Safin, Artur <aks084000 at utdallas.edu> wrote:
>
> Barry, Mark,
>
> Apologies for taking a while to respond.
>
>> Are you saying it works for a while but fails when the problem is large, or that it never works with fieldsplit_1? And how many processors are you using?
>
> All of these are serial runs. The preconditioner always fails for large problems; but I also found an example where it fails for a fairly small matrix. fieldsplit_1 is the one that fails, although fieldpslit_0 always works.
>
>> This should be fixed in the master branch
>
> I tried the developmental version (from today), but the same bug persists.
>
>
>
> I have some code that reproduces the example. The index sets that I use are non-overlapping; I also checked with matlab that each fieldsplit submatrix is non-singular.
> The issue is with fieldsplit_TA: if I do not use -fieldsplit_TA-pc_type_gamg, then the solver converges. Otherwise, it crashes with the same message.
>
> Artur
>
>
> <A> <A.info><ex.c> <run.sh>
More information about the petsc-users
mailing list