[petsc-users] pcfieldsplit for a composite dm with multiple subfields

Wed Sep 9 14:15:44 CDT 2015

   Thanks. I think you should try running this with __float128 (quad precision) to see what happens.

   Do you have access to a relatively recent (last couple of years) gcc (and gfortran) compiler?

   Select a new PETSC_ARCH name say arch-quad

   and run configure with the additional option -with-precision=__float128 

   not use --download-mumps or other such packages since they don't work with __float128.  If you get everything build then run
with the same options except do not use pc_factor_mat_solver_package mumps  just use the default PETSc solver LU

   Note that if your program is reading in binary files this will be slightly tricky since you will need to convert the result to __float128 format. If you use PETSc VecLoad() or PetscBinaryRead() etc you can just use the flag -binary_read_double and will read from a double binary file and automatically put the results into the PETSc __float128 matrices.

 Interpreting results. You may get convergence which might likely indicate the scaling and conditioning of your problem are very nasty but can be overcome with more precision or you may get the same lack of convergence you get now and I don't have a good explanation for that :-)

  Note: it could also be that your SNES function evaluation has a problem.

   Barry

> On Sep 9, 2015, at 1:48 PM, Gideon Simpson <gideon.simpson at gmail.com> wrote:
> 
> So here’s one grid sequence with these solver flags:
> -snes_max_it 100 -snes_converged_reason -snes_monitor \
> -pc_type lu \
> -pc_factor_mat_solver_package mumps -mat_mumps_icntl_28 1 -mat_mumps_icntl_7 6 \
> -snes_max_funcs 500000 -snes_fd -snes_monitor -ksp_monitor_true_residual
> 
> By the way, the problem, I believe is approaching something that is singular, so it is not unreasonable to think there is some kind of inherent ill conditioning in the problem.
> 
>     0 SNES Function norm 2.977789375068e-12 
>       0 KSP preconditioned resid norm 9.487609584353e-12 true resid norm 2.977789375068e-12 ||r(i)||/||b|| 1.000000000000e+00
>       1 KSP preconditioned resid norm 3.713064216887e-24 true resid norm 9.247299640119e-23 ||r(i)||/||b|| 3.105424351885e-11
>     1 SNES Function norm 2.511921299174e-12 
>   Nonlinear solve converged due to CONVERGED_SNORM_RELATIVE iterations 1
>   0 SNES Function norm 5.066222213175e+03 
>     0 KSP preconditioned resid norm 3.135073899423e+01 true resid norm 5.066222213175e+03 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 4.667575811795e-11 true resid norm 2.582648659172e-07 ||r(i)||/||b|| 5.097780062738e-11
>   1 SNES Function norm 8.484664416715e+02 
>     0 KSP preconditioned resid norm 2.668033649331e+02 true resid norm 8.484664416715e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.096350794450e-10 true resid norm 1.126402252137e-08 ||r(i)||/||b|| 1.327574311505e-11
>   2 SNES Function norm 6.549557200006e+02 
>     0 KSP preconditioned resid norm 1.552337314505e+02 true resid norm 6.549557200006e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.781790941843e-09 true resid norm 5.596253874028e-08 ||r(i)||/||b|| 8.544476677024e-11
>   3 SNES Function norm 5.770704424482e+02 
>     0 KSP preconditioned resid norm 1.348464485234e+02 true resid norm 5.770704424482e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.909040505447e-10 true resid norm 6.992127254821e-09 ||r(i)||/||b|| 1.211659225719e-11
>   4 SNES Function norm 5.237674957045e+02 
>     0 KSP preconditioned resid norm 1.524393824592e+02 true resid norm 5.237674957045e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.664845931493e-10 true resid norm 5.713325641481e-09 ||r(i)||/||b|| 1.090813326206e-11
>   5 SNES Function norm 4.753878819749e+02 
>     0 KSP preconditioned resid norm 1.864281909532e+02 true resid norm 4.753878819749e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.409973828386e-10 true resid norm 4.289369979836e-09 ||r(i)||/||b|| 9.022884559061e-12
>   6 SNES Function norm 4.221755368773e+02 
>     0 KSP preconditioned resid norm 2.551073925659e+02 true resid norm 4.221755368773e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 5.540011959843e-10 true resid norm 1.441047729187e-08 ||r(i)||/||b|| 3.413385199545e-11
>   7 SNES Function norm 3.806522663388e+02 
>     0 KSP preconditioned resid norm 5.183358055470e+02 true resid norm 3.806522663388e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.063740790885e-09 true resid norm 3.530342872323e-08 ||r(i)||/||b|| 9.274456464634e-11
>   8 SNES Function norm 3.762033603907e+02 
>     0 KSP preconditioned resid norm 5.324397409227e+03 true resid norm 3.762033603907e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 6.352530213970e-09 true resid norm 9.590500220593e-08 ||r(i)||/||b|| 2.549286165502e-10
>   9 SNES Function norm 3.758953286234e+02 
>     0 KSP preconditioned resid norm 6.246847657999e+03 true resid norm 3.758953286234e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 5.367419000872e-08 true resid norm 8.213904234688e-07 ||r(i)||/||b|| 2.185157305564e-09
>  10 SNES Function norm 3.756870388529e+02 
>     0 KSP preconditioned resid norm 5.901626862375e+02 true resid norm 3.756870388529e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.330043229228e-09 true resid norm 1.418363475093e-08 ||r(i)||/||b|| 3.775385702481e-11
>  11 SNES Function norm 3.728715884883e+02 
>     0 KSP preconditioned resid norm 1.069749399855e+03 true resid norm 3.728715884883e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.911440821785e-10 true resid norm 9.552819316154e-09 ||r(i)||/||b|| 2.561959562241e-11
>  12 SNES Function norm 3.722328158041e+02 
>     0 KSP preconditioned resid norm 3.064673278570e+03 true resid norm 3.722328158041e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.105090853307e-08 true resid norm 1.085271466536e-07 ||r(i)||/||b|| 2.915571707969e-10
>  13 SNES Function norm 3.720989356426e+02 
>     0 KSP preconditioned resid norm 9.068377570178e+03 true resid norm 3.720989356426e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.784084466576e-08 true resid norm 3.777243066339e-07 ||r(i)||/||b|| 1.015117944322e-09
>  14 SNES Function norm 3.720707403757e+02 
>     0 KSP preconditioned resid norm 7.198056009453e+04 true resid norm 3.720707403757e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 7.120525808630e-08 true resid norm 1.054867000686e-06 ||r(i)||/||b|| 2.835124846476e-09
>  15 SNES Function norm 3.720707293983e+02 
>     0 KSP preconditioned resid norm 4.134314655599e+03 true resid norm 3.720707293983e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.003141582850e-08 true resid norm 2.046203596859e-07 ||r(i)||/||b|| 5.499501667784e-10
>  16 SNES Function norm 3.719027845818e+02 
>     0 KSP preconditioned resid norm 2.545115269075e+03 true resid norm 3.719027845818e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 4.965948884734e-08 true resid norm 5.071610656715e-07 ||r(i)||/||b|| 1.363692574235e-09
>  17 SNES Function norm 3.714400762925e+02 
>     0 KSP preconditioned resid norm 1.109601826815e+03 true resid norm 3.714400762925e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 6.169052076803e-09 true resid norm 6.563603648295e-08 ||r(i)||/||b|| 1.767069324832e-10
>  18 SNES Function norm 3.701626928766e+02 
>     0 KSP preconditioned resid norm 2.447547395077e+02 true resid norm 3.701626928766e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.250579098159e-09 true resid norm 1.218652092692e-08 ||r(i)||/||b|| 3.292206686799e-11
>  19 SNES Function norm 3.556442798407e+02 
>     0 KSP preconditioned resid norm 1.418496111605e+02 true resid norm 3.556442798407e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.627252384359e-10 true resid norm 1.951334080585e-09 ||r(i)||/||b|| 5.486757952241e-12
>  20 SNES Function norm 3.496125058841e+02 
>     0 KSP preconditioned resid norm 6.543042326746e+01 true resid norm 3.496125058841e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.377371559702e-10 true resid norm 1.295256556885e-09 ||r(i)||/||b|| 3.704834738706e-12
>  21 SNES Function norm 3.146776071310e+02 
>     0 KSP preconditioned resid norm 4.667517635820e+01 true resid norm 3.146776071310e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.743789267634e-09 true resid norm 1.644758285998e-08 ||r(i)||/||b|| 5.226804350630e-11
>  22 SNES Function norm 2.855422195269e+02 
>     0 KSP preconditioned resid norm 2.243742065865e+02 true resid norm 2.855422195269e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.047816533727e-10 true resid norm 2.299887594213e-09 ||r(i)||/||b|| 8.054457228859e-12
>  23 SNES Function norm 2.806777486780e+02 
>     0 KSP preconditioned resid norm 1.561899503179e+03 true resid norm 2.806777486780e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.972680147986e-09 true resid norm 7.239772649903e-08 ||r(i)||/||b|| 2.579389596790e-10
>  24 SNES Function norm 2.805893466480e+02 
>     0 KSP preconditioned resid norm 1.722111652119e+03 true resid norm 2.805893466480e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 5.170553469009e-09 true resid norm 6.273773919075e-08 ||r(i)||/||b|| 2.235927341513e-10
>  25 SNES Function norm 2.805171735939e+02 
>     0 KSP preconditioned resid norm 8.024869901472e+03 true resid norm 2.805171735939e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.741187759948e-08 true resid norm 3.211723859501e-07 ||r(i)||/||b|| 1.144929495173e-09
>  26 SNES Function norm 2.805136061483e+02 
>     0 KSP preconditioned resid norm 3.239737262200e+04 true resid norm 2.805136061483e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 2.317708795749e-08 true resid norm 3.672739850558e-07 ||r(i)||/||b|| 1.309291160949e-09
>  27 SNES Function norm 2.805134008969e+02 
>     0 KSP preconditioned resid norm 6.920567009699e+04 true resid norm 2.805134008969e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.101785240910e-07 true resid norm 1.523556762078e-06 ||r(i)||/||b|| 5.431315428092e-09
>  28 SNES Function norm 2.805133530790e+02 
>     0 KSP preconditioned resid norm 3.265182256725e+05 true resid norm 2.805133530790e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.903754744633e-07 true resid norm 6.654431764515e-06 ||r(i)||/||b|| 2.372233510980e-08
>  29 SNES Function norm 2.805133509342e+02 
>     0 KSP preconditioned resid norm 1.312587966212e+06 true resid norm 2.805133509342e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 1.990958223875e-07 true resid norm 1.311133369898e-05 ||r(i)||/||b|| 4.674049793106e-08
>  30 SNES Function norm 2.805133508006e+02 
>     0 KSP preconditioned resid norm 2.049236485623e+07 true resid norm 2.805133508006e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 3.432837714993e-05 true resid norm 4.348842776781e-04 ||r(i)||/||b|| 1.550315792232e-06
>  31 SNES Function norm 2.805133508001e+02 
>     0 KSP preconditioned resid norm 7.694877106483e+06 true resid norm 2.805133508001e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 4.401952388358e-06 true resid norm 1.008598135500e-04 ||r(i)||/||b|| 3.595544142990e-07
>  32 SNES Function norm 2.805133507963e+02 
>     0 KSP preconditioned resid norm 1.116553807875e+08 true resid norm 2.805133507963e+02 ||r(i)||/||b|| 1.000000000000e+00
>     1 KSP preconditioned resid norm 7.718443445086e-06 true resid norm 1.327187421086e-03 ||r(i)||/||b|| 4.731280765492e-06
> Nonlinear solve did not converge due to DIVERGED_LINE_SEARCH iterations 32
> 
> -gideon
> 
>> On Sep 9, 2015, at 1:52 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>> 
>>  Please run with the -snes_fd and one level of grid sequencing (where Newton was having trouble converging on the second level) with -snes_monitor -ksp_monitor_true_residual and send the output. Too much speculation and not enough data.
>> 
> 