<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">I tried the options "-mat_type baij” and that seemed to change the type of matrix to BAIJ. </div><div class=""><br class=""></div><div class="">I am now experimenting with various preconditioners (ILU, ASM, etc.), and things seem to be working fine so far. </div><div class=""><br class=""></div><div class=""><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">KSP Object:(fluid_complex_) 4 MPI processes</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: gmres</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> GMRES: happy breakdown tolerance 1e-30</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> maximum iterations=10000, initial guess is zero</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> tolerances: relative=1e-10, absolute=1e-50, divergence=10000.</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> left preconditioning</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> using PRECONDITIONED norm type for convergence test</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">PC Object:(fluid_complex_) 4 MPI processes</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: asm</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Additive Schwarz: total subdomain blocks = 4, amount of overlap = 1</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Additive Schwarz: restriction/interpolation type - RESTRICT</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Local solve is same for all blocks, in the following KSP and PC objects:</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> KSP Object: (fluid_complex_sub_) 1 MPI processes</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: gmres</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> GMRES: happy breakdown tolerance 1e-30</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> maximum iterations=10000, initial guess is zero</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> left preconditioning</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> using PRECONDITIONED norm type for convergence test</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> PC Object: (fluid_complex_sub_) 1 MPI processes</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: ilu</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> ILU: out-of-place factorization</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> 3 levels of fill</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> tolerance for zero pivot 2.22045e-14</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> matrix ordering: natural</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> factor fill ratio given 1., needed 2.32913</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Factored matrix follows:</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Mat Object: 1 MPI processes</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: seqbaij</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> rows=294752, cols=294752, bs=2</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> package used to perform factorization: petsc</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total: nonzeros=49050496, allocated nonzeros=49050496</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total number of mallocs used during MatSetValues calls =0</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> block size is 2</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> linear system matrix = precond matrix:</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Mat Object: (fluid_complex_) 1 MPI processes</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: seqbaij</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> rows=294752, cols=294752, bs=2</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total: nonzeros=21059584, allocated nonzeros=21059584</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total number of mallocs used during MatSetValues calls =0</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> block size is 2</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> linear system matrix = precond matrix:</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> Mat Object: (fluid_complex_) 4 MPI processes</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> type: mpibaij</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> rows=1158728, cols=1158728, bs=2</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total: nonzeros=83105344, allocated nonzeros=83266816</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> total number of mallocs used during MatSetValues calls =0</span></div><div style="margin: 0px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> block size is 2</span></div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">-Manav</div><div class=""><br class=""></div><br class=""><div><blockquote type="cite" class=""><div class="">On Nov 15, 2016, at 3:34 PM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov" class="">bsmith@mcs.anl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class=""><br class="Apple-interchange-newline">On Nov 15, 2016, at 3:23 PM, Manav Bhatia <<a href="mailto:bhatiamanav@gmail.com" class="">bhatiamanav@gmail.com</a>> wrote:<br class=""><br class="">I have a complex system, (A + i B) (x + i y) = (f + ig), that I am trying to solve using real matrices:<span class="Apple-converted-space"> </span><br class=""><br class=""> [A -B; B A ] [x; y] = [f; g]<br class=""><br class="">So, the 2x2 block is made of the real and imaginary component of each entry in the complex matrix.<span class="Apple-converted-space"> </span><br class=""><br class="">I am following the discussion in the following paper:<span class="Apple-converted-space"> </span><br class=""><br class="">DAY D. \& HEROUX M.A. 2001. Solving complex-valued linear systems via equivalent real formulations. \textit{SIAM Journal on Scientific Computing} 23: 480-498.<br class=""><br class="">Following is an excerpt.<span class="Apple-converted-space"> </span><br class=""><br class="">**********************************************************************************<br class=""><br class="">The matrix K in the K formulation has a natural 2-by-2 block structure that can be exploited by using block entry data structures. Using the block entry features of these packages has the following benefits.<br class=""><br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>• Applying 2-by-2 block Jacobi scaling to K corresponds exactly to applying point Jacobi scaling to C.<br class=""><br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>• The block sparsity pattern of K exactly matches the point sparsity pattern of C. Thus any pattern-based preconditioners such as block ILU(l) applied to K correspond exactly to ILU(l) applied to C. See section 4 for definitions of block ILU(l) and ILU(l).<br class=""><br class=""><span class="Apple-tab-span" style="white-space: pre;"> </span>• Any drop tolerance-based complex preconditioner has a straightforward K formulation since the absolute value of a complex entry equals the scaled Frobenius norm of the corresponding block entry in K.<span class="Apple-converted-space"> </span><br class=""><br class="">**********************************************************************************<br class=""><br class="">The paper additional outlines the challenges of the poor spectral properties of the equivalent real system.<span class="Apple-converted-space"> </span><br class=""><br class="">So, I am assembling the system with a 2x2 block, but am not sure how to best pick the right solver options in Petsc.<span class="Apple-converted-space"> </span><br class=""><br class="">I agree that I am getting confused by the “block” nomenclature. Particularly, I am not sure how to reconcile the different notions with points 1 and 2 from the paper (noted above).<span class="Apple-converted-space"> </span><br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class=""> In PETSc we call this 2x2 block Jacobi "point-block Jacobi" you can use the option -pc_type pbjacobi. The ILU() in PETSc can also be "point block", this is obtained with the usual -pc_type ilu (that is there is no different preconditioner name for ILU point block). To use all these things you need to make your matrix a BAIJ matrix (not an AIJ) and set its block size to 2.<span class="Apple-converted-space"> </span></span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class=""> Have you tried solving the matrices as complex? Is there a reason you wish to reformulate them as real?<span class="Apple-converted-space"> </span></span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class=""> The convergence of iterative methods (either with real or complex numbers) depends on the properties of the A and B (i.e. C) matrix. Where does the C matrix come from? There are many applications that result in complex matrices that are poorly conditioned for iterative methods.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class=""> Barry</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class=""><br class="">Any guidance would be appreciated!<br class=""><br class="">Thanks,<br class="">Manav<br class=""><br class=""><br class=""><blockquote type="cite" class="">On Nov 15, 2016, at 3:12 PM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov" class="">bsmith@mcs.anl.gov</a>> wrote:<br class=""><br class=""><br class=""> We can help you if you provide more information about what the blocks represent in your problem.<span class="Apple-converted-space"> </span><br class=""><br class=""> Do you have two degrees of freedom at each grid point? What physically are the two degrees of freedom. What equations are you solving?<br class=""><br class=""> I think you may be mixing up the "matrix block size" of 2 with the blocks in "block Jacobi". Though both are called "block" they really don't have anything to do with each other.<span class="Apple-converted-space"> </span><br class=""><br class=""> Barry<br class=""><br class=""><blockquote type="cite" class="">On Nov 15, 2016, at 3:03 PM, Manav Bhatia <<a href="mailto:bhatiamanav@gmail.com" class="">bhatiamanav@gmail.com</a>> wrote:<br class=""><br class="">Hi,<span class="Apple-converted-space"> </span><br class=""><br class=""> I am setting up a matrix with the following calls. The intent is to solve the system with a 2x2 block size.<br class=""><br class=""> What combinations of KSP/PC will effectively translate to solving this block matrix system?<span class="Apple-converted-space"> </span><br class=""><br class=""> I saw a discussion about bjacobi in the manual with the following calls (I omitted the prefixes from my actual command): <br class="">-pc_type bjacobi -pc_bjacobi_blocks 2 -sub_ksp_type preonly -sub_pc_type lu -ksp_view<br class=""><br class="">which provides the following output:<span class="Apple-converted-space"> </span><br class="">KSP Object:(fluid_complex_) 1 MPI processes<br class="">type: gmres<br class=""> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement<br class=""> GMRES: happy breakdown tolerance 1e-30<br class="">maximum iterations=10000, initial guess is zero<br class="">tolerances: relative=1e-05, absolute=1e-50, divergence=10000.<br class="">left preconditioning<br class="">using PRECONDITIONED norm type for convergence test<br class="">PC Object:(fluid_complex_) 1 MPI processes<br class="">type: bjacobi<br class=""> block Jacobi: number of blocks = 2<br class=""> Local solve is same for all blocks, in the following KSP and PC objects:<br class=""> KSP Object: (fluid_complex_sub_) 1 MPI processes<br class=""> type: preonly<br class=""> maximum iterations=10000, initial guess is zero<br class=""> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.<br class=""> left preconditioning<br class=""> using NONE norm type for convergence test<br class=""> PC Object: (fluid_complex_sub_) 1 MPI processes<br class=""> type: lu<br class=""> LU: out-of-place factorization<br class=""> tolerance for zero pivot 2.22045e-14<br class=""> matrix ordering: nd<br class=""> factor fill ratio given 5., needed 5.70941<br class=""> Factored matrix follows:<br class=""> Mat Object: 1 MPI processes<br class=""> type: seqaij<br class=""> rows=36844, cols=36844<br class=""> package used to perform factorization: petsc<br class=""> total: nonzeros=14748816, allocated nonzeros=14748816<br class=""> total number of mallocs used during MatSetValues calls =0<br class=""> using I-node routines: found 9211 nodes, limit used is 5<br class=""> linear system matrix = precond matrix:<br class=""> Mat Object: (fluid_complex_) 1 MPI processes<br class=""> type: seqaij<br class=""> rows=36844, cols=36844<br class=""> total: nonzeros=2583248, allocated nonzeros=2583248<br class=""> total number of mallocs used during MatSetValues calls =0<br class=""> using I-node routines: found 9211 nodes, limit used is 5<br class="">linear system matrix = precond matrix:<br class="">Mat Object: (fluid_complex_) 1 MPI processes<br class=""> type: seqaij<br class=""> rows=73688, cols=73688, bs=2<br class=""> total: nonzeros=5224384, allocated nonzeros=5224384<br class=""> total number of mallocs used during MatSetValues calls =0<br class=""> using I-node routines: found 18422 nodes, limit used is 5<br class=""><br class=""><br class="">Likewise, I tried to use a more generic option:<span class="Apple-converted-space"> </span><br class="">-mat_set_block_size 2 -ksp_type gmres -pc_type ilu -sub_ksp_type preonly -sub_pc_type lu -ksp_view<br class=""><br class="">with the following output:<br class="">Linear fluid_complex_ solve converged due to CONVERGED_RTOL iterations 38<br class="">KSP Object:(fluid_complex_) 1 MPI processes<br class="">type: gmres<br class=""> GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement<br class=""> GMRES: happy breakdown tolerance 1e-30<br class="">maximum iterations=10000, initial guess is zero<br class="">tolerances: relative=1e-05, absolute=1e-50, divergence=10000.<br class="">left preconditioning<br class="">using PRECONDITIONED norm type for convergence test<br class="">PC Object:(fluid_complex_) 1 MPI processes<br class="">type: ilu<br class=""> ILU: out-of-place factorization<br class=""> 0 levels of fill<br class=""> tolerance for zero pivot 2.22045e-14<br class=""> matrix ordering: natural<br class=""> factor fill ratio given 1., needed 1.<br class=""> Factored matrix follows:<br class=""> Mat Object: 1 MPI processes<br class=""> type: seqaij<br class=""> rows=73688, cols=73688, bs=2<br class=""> package used to perform factorization: petsc<br class=""> total: nonzeros=5224384, allocated nonzeros=5224384<br class=""> total number of mallocs used during MatSetValues calls =0<br class=""> using I-node routines: found 18422 nodes, limit used is 5<br class="">linear system matrix = precond matrix:<br class="">Mat Object: (fluid_complex_) 1 MPI processes<br class=""> type: seqaij<br class=""> rows=73688, cols=73688, bs=2<br class=""> total: nonzeros=5224384, allocated nonzeros=5224384<br class=""> total number of mallocs used during MatSetValues calls =0<br class=""> using I-node routines: found 18422 nodes, limit used is 5<br class=""><br class="">Are other PC types expected to translate to the block matrices?<span class="Apple-converted-space"> </span><br class=""><br class="">I would appreciate any guidance.<span class="Apple-converted-space"> </span><br class=""><br class="">Thanks,<br class="">Manav</blockquote></blockquote></blockquote></div></blockquote></div><br class=""></body></html>