<div dir="ltr"><div>Hello</div><div><br></div>May I know if the nested IS is identified based on relative local index or the global one? It's hard to identify that in the attached example.<div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Giang</div></div></div>
<br><div class="gmail_quote">On Thu, Jul 28, 2016 at 10:35 AM, Lawrence Mitchell <span dir="ltr"><<a href="mailto:lawrence.mitchell@imperial.ac.uk" target="_blank">lawrence.mitchell@imperial.ac.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Dear Artur,<br>
<span class=""><br>
On 28/07/16 02:20, Safin, Artur wrote:<br>
> Barry, Lawrence,<br>
><br>
>> I think the SubKSPs (and therefore SubPCs) are not set up until you call KSPSetUp(ksp) which your code does not do explicitly and is therefore done in KSPSolve.<br>
><br>
> I added KSPSetUp(), but unfortunately the issue did not go away.<br>
><br>
><br>
><br>
> I have created a MWE that replicates the issue. The program tries to solve a tridiagonal system, where the first fieldsplit partitions the global matrix<br>
><br>
> [ P x ]<br>
> [ x T ],<br>
><br>
> and the nested fieldsplit partitions P into<br>
><br>
> [ A x ]<br>
> [ x B ].<br>
<br>
</span>Two things:<br>
<br>
1. Always check the return value from all PETSc calls. This will<br>
normally give you a very useful backtrace when something goes wrong.<br>
<br>
That is, annotate all your calls with:<br>
<br>
PetscErrorCode ierr;<br>
<br>
<br>
ierr = SomePetscFunction(...); CHKERRQ(ierr);<br>
<br>
If I do this, I see that the call to KSPSetUp fails:<br>
<span class=""><br>
[0]PETSC ERROR: --------------------- Error Message<br>
------------------------------<wbr>------------------------------<wbr>--<br>
</span>[0]PETSC ERROR: Petsc has generated inconsistent data<br>
[0]PETSC ERROR: Unhandled case, must have at least two fields, not 1<br>
<span class="">[0]PETSC ERROR: See<br>
<a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/<wbr>documentation/faq.html</a> for trouble shooting.<br>
</span>[0]PETSC ERROR: Petsc Development GIT revision: v3.7.2-931-g1e46b98<br>
GIT Date: 2016-07-06 16:57:50 -0500<br>
<br>
...<br>
<br>
[0]PETSC ERROR: #1 PCFieldSplitSetDefaults() line 470 in<br>
/data/lmitche1/src/deps/petsc/<wbr>src/ksp/pc/impls/fieldsplit/<wbr>fieldsplit.c<br>
[0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 487 in<br>
/data/lmitche1/src/deps/petsc/<wbr>src/ksp/pc/impls/fieldsplit/<wbr>fieldsplit.c<br>
[0]PETSC ERROR: #3 PCSetUp() line 968 in<br>
/data/lmitche1/src/deps/petsc/<wbr>src/ksp/pc/interface/precon.c<br>
[0]PETSC ERROR: #4 KSPSetUp() line 393 in<br>
/data/lmitche1/src/deps/petsc/<wbr>src/ksp/ksp/interface/itfunc.c<br>
[0]PETSC ERROR: #5 main() line 65 in /homes/lmitche1/tmp/ex.c<br>
<br>
The reason is you need to call KSPSetUp *after* setting the outermost<br>
fieldsplit ISes.<br>
<br>
If I move the call to KSPSetUp, then things seem to work. I've<br>
attached the working code.<br>
<br>
Cheers,<br>
<br>
Lawrence<br>
<br>
$ cat options.txt<br>
-pc_type fieldsplit<br>
-pc_fieldsplit_type multiplicative<br>
-fieldsplit_T_ksp_type bcgs<br>
<span class="">-fieldsplit_P_ksp_type gmres<br>
-fieldsplit_P_pc_type fieldsplit<br>
-fieldsplit_P_pc_fieldsplit_<wbr>type multiplicative<br>
-fieldsplit_P_fieldsplit_A_<wbr>ksp_type gmres<br>
-fieldsplit_P_fieldsplit_B_pc_<wbr>type lu<br>
-fieldsplit_P_fieldsplit_B_<wbr>ksp_type preonly<br>
</span>-ksp_converged_reason<br>
-ksp_monitor_true_residual<br>
-ksp_view<br>
<br>
$ ./ex -options_file options.txt<br>
<br>
0 KSP preconditioned resid norm 5.774607007892e+00 true resid norm<br>
1.414213562373e+00 ||r(i)||/||b|| 1.000000000000e+00<br>
1 KSP preconditioned resid norm 1.921795888956e-01 true resid norm<br>
4.802975385197e-02 ||r(i)||/||b|| 3.396216464745e-02<br>
2 KSP preconditioned resid norm 1.436304589027e-12 true resid norm<br>
2.435255920058e-13 ||r(i)||/||b|| 1.721985974998e-13<br>
Linear solve converged due to CONVERGED_RTOL iterations 2<br>
KSP Object: 1 MPI processes<br>
type: gmres<br>
GMRES: restart=30, using Classical (unmodified) Gram-Schmidt<br>
Orthogonalization with no iterative refinement<br>
GMRES: happy breakdown tolerance 1e-30<br>
maximum iterations=10000, initial guess is zero<br>
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.<br>
left preconditioning<br>
using PRECONDITIONED norm type for convergence test<br>
PC Object: 1 MPI processes<br>
type: fieldsplit<br>
FieldSplit with MULTIPLICATIVE composition: total splits = 2<br>
Solver info for each split is in the following KSP objects:<br>
Split number 0 Defined by IS<br>
KSP Object: (fieldsplit_P_) 1 MPI processes<br>
type: gmres<br>
GMRES: restart=30, using Classical (unmodified) Gram-Schmidt<br>
Orthogonalization with no iterative refinement<br>
GMRES: happy breakdown tolerance 1e-30<br>
maximum iterations=10000, initial guess is zero<br>
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.<br>
left preconditioning<br>
using PRECONDITIONED norm type for convergence test<br>
PC Object: (fieldsplit_P_) 1 MPI processes<br>
type: fieldsplit<br>
FieldSplit with MULTIPLICATIVE composition: total splits = 2<br>
Solver info for each split is in the following KSP objects:<br>
Split number 0 Defined by IS<br>
KSP Object: (fieldsplit_P_fieldsplit_A_) 1 MPI processes<br>
type: gmres<br>
GMRES: restart=30, using Classical (unmodified)<br>
Gram-Schmidt Orthogonalization with no iterative refinement<br>
GMRES: happy breakdown tolerance 1e-30<br>
maximum iterations=10000, initial guess is zero<br>
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.<br>
left preconditioning<br>
using PRECONDITIONED norm type for convergence test<br>
PC Object: (fieldsplit_P_fieldsplit_A_) 1 MPI processes<br>
type: ilu<br>
ILU: out-of-place factorization<br>
0 levels of fill<br>
tolerance for zero pivot 2.22045e-14<br>
matrix ordering: natural<br>
factor fill ratio given 1., needed 1.<br>
Factored matrix follows:<br>
Mat Object: 1 MPI processes<br>
type: seqaij<br>
rows=25, cols=25<br>
package used to perform factorization: petsc<br>
total: nonzeros=73, allocated nonzeros=73<br>
total number of mallocs used during MatSetValues<br>
calls =0<br>
not using I-node routines<br>
linear system matrix = precond matrix:<br>
Mat Object: (fieldsplit_P_fieldsplit_A_) 1 MPI processes<br>
type: seqaij<br>
rows=25, cols=25<br>
total: nonzeros=73, allocated nonzeros=73<br>
total number of mallocs used during MatSetValues calls =0<br>
not using I-node routines<br>
Split number 1 Defined by IS<br>
KSP Object: (fieldsplit_P_fieldsplit_B_) 1 MPI processes<br>
type: preonly<br>
maximum iterations=10000, initial guess is zero<br>
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.<br>
left preconditioning<br>
using NONE norm type for convergence test<br>
PC Object: (fieldsplit_P_fieldsplit_B_) 1 MPI processes<br>
type: lu<br>
LU: out-of-place factorization<br>
tolerance for zero pivot 2.22045e-14<br>
matrix ordering: nd<br>
factor fill ratio given 5., needed 1.43836<br>
Factored matrix follows:<br>
Mat Object: 1 MPI processes<br>
type: seqaij<br>
rows=25, cols=25<br>
package used to perform factorization: petsc<br>
total: nonzeros=105, allocated nonzeros=105<br>
total number of mallocs used during MatSetValues<br>
calls =0<br>
not using I-node routines<br>
linear system matrix = precond matrix:<br>
Mat Object: (fieldsplit_P_fieldsplit_B_) 1 MPI processes<br>
type: seqaij<br>
rows=25, cols=25<br>
total: nonzeros=73, allocated nonzeros=73<br>
total number of mallocs used during MatSetValues calls =0<br>
not using I-node routines<br>
linear system matrix = precond matrix:<br>
Mat Object: (fieldsplit_P_) 1 MPI processes<br>
type: seqaij<br>
rows=50, cols=50<br>
total: nonzeros=148, allocated nonzeros=148<br>
total number of mallocs used during MatSetValues calls =0<br>
not using I-node routines<br>
Split number 1 Defined by IS<br>
KSP Object: (fieldsplit_T_) 1 MPI processes<br>
type: bcgs<br>
maximum iterations=10000, initial guess is zero<br>
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.<br>
left preconditioning<br>
using PRECONDITIONED norm type for convergence test<br>
PC Object: (fieldsplit_T_) 1 MPI processes<br>
type: ilu<br>
ILU: out-of-place factorization<br>
0 levels of fill<br>
tolerance for zero pivot 2.22045e-14<br>
matrix ordering: natural<br>
factor fill ratio given 1., needed 1.<br>
Factored matrix follows:<br>
Mat Object: 1 MPI processes<br>
type: seqaij<br>
rows=50, cols=50<br>
package used to perform factorization: petsc<br>
total: nonzeros=148, allocated nonzeros=148<br>
total number of mallocs used during MatSetValues calls =0<br>
not using I-node routines<br>
linear system matrix = precond matrix:<br>
Mat Object: (fieldsplit_T_) 1 MPI processes<br>
type: seqaij<br>
rows=50, cols=50<br>
total: nonzeros=148, allocated nonzeros=148<br>
total number of mallocs used during MatSetValues calls =0<br>
not using I-node routines<br>
linear system matrix = precond matrix:<br>
Mat Object: 1 MPI processes<br>
type: seqaij<br>
rows=100, cols=100<br>
total: nonzeros=298, allocated nonzeros=500<br>
total number of mallocs used during MatSetValues calls =0<br>
not using I-node routines<br>
<br>
<br>
</blockquote></div><br></div></div>