<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi, <div class=""><br class=""></div><div class=""> I am setting up a matrix with the following calls. The intent is to solve the system with a 2x2 block size.</div><div class=""><br class=""></div><div class=""> What combinations of KSP/PC will effectively translate to solving this block matrix system? </div><div class=""><br class=""></div><div class=""> I saw a discussion about bjacobi in the manual with the following calls (I omitted the prefixes from my actual command): </div><div class=""><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">-pc_type bjacobi -pc_bjacobi_blocks 2 -sub_ksp_type preonly -sub_pc_type lu -ksp_view</span></div></div><div class=""><br class=""></div><div class="">which provides the following output: </div><div class=""><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">KSP Object:(fluid_complex_) 1 MPI processes</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: gmres</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> GMRES: happy breakdown tolerance 1e-30</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> maximum iterations=10000, initial guess is zero</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> left preconditioning</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> using PRECONDITIONED norm type for convergence test</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">PC Object:(fluid_complex_) 1 MPI processes</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: bjacobi</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> block Jacobi: number of blocks = 2</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Local solve is same for all blocks, in the following KSP and PC objects:</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> KSP Object: (fluid_complex_sub_) 1 MPI processes</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: preonly</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> maximum iterations=10000, initial guess is zero</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> left preconditioning</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> using NONE norm type for convergence test</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> PC Object: (fluid_complex_sub_) 1 MPI processes</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: lu</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> LU: out-of-place factorization</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> tolerance for zero pivot 2.22045e-14</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> matrix ordering: nd</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> factor fill ratio given 5., needed 5.70941</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Factored matrix follows:</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Mat Object: 1 MPI processes</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: seqaij</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> rows=36844, cols=36844</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> package used to perform factorization: petsc</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total: nonzeros=14748816, allocated nonzeros=14748816</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total number of mallocs used during MatSetValues calls =0</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> using I-node routines: found 9211 nodes, limit used is 5</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> linear system matrix = precond matrix:</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Mat Object: (fluid_complex_) 1 MPI processes</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: seqaij</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> rows=36844, cols=36844</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total: nonzeros=2583248, allocated nonzeros=2583248</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total number of mallocs used during MatSetValues calls =0</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> using I-node routines: found 9211 nodes, limit used is 5</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> linear system matrix = precond matrix:</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Mat Object: (fluid_complex_) 1 MPI processes</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: seqaij</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> rows=73688, cols=73688, bs=2</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total: nonzeros=5224384, allocated nonzeros=5224384</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total number of mallocs used during MatSetValues calls =0</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> using I-node routines: found 18422 nodes, limit used is 5</span></div></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span></div><div class=""><br class=""></div><div class="">Likewise, I tried to use a more generic option: </div><div class=""><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">-mat_set_block_size 2 -ksp_type gmres -pc_type ilu -sub_ksp_type preonly -sub_pc_type lu -ksp_view</span></div></div><div class=""><br class=""></div><div class="">with the following output:</div><div class=""><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Linear fluid_complex_ solve converged due to CONVERGED_RTOL iterations 38</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">KSP Object:(fluid_complex_) 1 MPI processes</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: gmres</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> GMRES: happy breakdown tolerance 1e-30</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> maximum iterations=10000, initial guess is zero</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> left preconditioning</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> using PRECONDITIONED norm type for convergence test</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">PC Object:(fluid_complex_) 1 MPI processes</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: ilu</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> ILU: out-of-place factorization</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 0 levels of fill</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> tolerance for zero pivot 2.22045e-14</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> matrix ordering: natural</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> factor fill ratio given 1., needed 1.</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Factored matrix follows:</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Mat Object: 1 MPI processes</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: seqaij</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> rows=73688, cols=73688, bs=2</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> package used to perform factorization: petsc</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total: nonzeros=5224384, allocated nonzeros=5224384</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total number of mallocs used during MatSetValues calls =0</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> using I-node routines: found 18422 nodes, limit used is 5</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> linear system matrix = precond matrix:</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Mat Object: (fluid_complex_) 1 MPI processes</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: seqaij</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> rows=73688, cols=73688, bs=2</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total: nonzeros=5224384, allocated nonzeros=5224384</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total number of mallocs used during MatSetValues calls =0</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> using I-node routines: found 18422 nodes, limit used is 5</span></div></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span></div><div class=""> Are other PC types expected to translate to the block matrices? </div><div class=""><br class=""></div><div class=""> I would appreciate any guidance. </div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Manav<br class=""><div class=""><br class=""></div></div></body></html>