<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 28 Jul 2019, at 3:02 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" class="">mfadams@lbl.gov</a>> wrote:</div><div class=""><div dir="ltr" class=""><div dir="ltr" class=""><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Jul 28, 2019 at 2:54 AM Pierre Jolivet via petsc-dev <<a href="mailto:petsc-dev@mcs.anl.gov" class="">petsc-dev@mcs.anl.gov</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class="">Hello,<div class="">I’m facing multiple issues with PCREDUNDANT and MATMPISBAIJ:</div><div class="">1) <a href="https://www.mcs.anl.gov/petsc/petsc-current/src/mat/impls/sbaij/mpi/mpisbaij.c.html#line3354" target="_blank" class="">https://www.mcs.anl.gov/petsc/petsc-current/src/mat/impls/sbaij/mpi/mpisbaij.c.html#line3354</a> shouldn’t this be <span class="">sum != N? I’m running an example where it says that sum (4) != Nbs (60), with a bs=15.</span></div></div></blockquote><div class=""><br class=""></div><div class="">Clearly a bug.</div></div></div></div></blockquote><div><br class=""></div><div>OK, fixed here: <a href="https://bitbucket.org/petsc/petsc/pull-requests/1915/fixes/diff#Lsrc/mat/impls/sbaij/mpi/mpisbaij.cT3324" class="">https://bitbucket.org/petsc/petsc/pull-requests/1915/fixes/diff#Lsrc/mat/impls/sbaij/mpi/mpisbaij.cT3324</a> (once again, this isn’t tested in PETSc for a bs>1, I think, but now my preconditioner is working OK, for a bs>1).</div><br class=""><blockquote type="cite" class=""><div dir="ltr" class=""><div class="gmail_quote"><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class=""><div class=""><span class="">2) when I’m using MATMPIBAIJ, I can do stuff like: </span>-prefix_mat_type baij -prefix_pc_type redundant -prefix_redundant_pc_type ilu, and in the KSPView, I have "package used to perform factorization: petsc”, so the underlying MatType is indeed MATSEQBAIJ. </div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class=""><div class="">However, with MATMPISBAIJ, if I do: -prefix_mat_type sbaij -prefix_pc_type redundant, first, it looks like you are hardwiring a PCLU (MatGetFactor() line 4440 in src/mat/interface/matrix.c </div></div></blockquote><div class=""><br class=""></div><div class="">Using LU as a default for symmetric matrices does seem wrong.</div></div></div></blockquote><div><br class=""></div><div>FWIW, I made a similar remark to Jose because the same matrices are being used by SLEPc for a GEVP with a SINVERT ST in a Krylov—Schur EPS (tentative fix here: <a href="https://bitbucket.org/slepc/slepc/branch/jose/sinvert-cholesky#diff" class="">https://bitbucket.org/slepc/slepc/branch/jose/sinvert-cholesky#diff</a>).</div><br class=""><blockquote type="cite" class=""><div dir="ltr" class=""><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class=""><div class="">Could not locate a solver package.), then, if I append -prefix_redundant_pc_type cholesky, I end up with an error related to MUMPS: MatGetFactor_sbaij_mumps() line 2625 in src/mat/impls/aij/mpi/mumps/mumps.c Cannot use PETSc SBAIJ matrices with block size > 1 with MUMPS Cholesky, use AIJ matrix instead. Why isn’t this call dispatched to PETSc Cholesky for SeqSBAIJ matrices?</div><div class=""><br class=""></div></div></blockquote><div class=""><br class=""></div><div class="">Generally, we don't like to switch parameters under the covers like this. We would rather you get your inputs right so you know what is going on.</div></div></div></blockquote><div><br class=""></div><div>I’m not sure how to interpret your comment, but anyway, I found out that for SBAIJ matrices, the type of the output matrix in MatCreateMPIMatConcatenateSeqMat (used by PCREDUNDANT) was set to MPISBAIJ, whereas for BAIJ, it was BAIJ (and thus SeqBAIJ when used within PCREDUNDANT with “full” redundancy over PETSC_COMM_WORLD).</div><div>Here is the fix: <a href="https://bitbucket.org/petsc/petsc/pull-requests/1915/fixes/diff#Lsrc/mat/impls/sbaij/mpi/mpisbaij.cT3345" class="">https://bitbucket.org/petsc/petsc/pull-requests/1915/fixes/diff#Lsrc/mat/impls/sbaij/mpi/mpisbaij.cT3345</a> (inspired by <a href="https://www.mcs.anl.gov/petsc/petsc-current/src/mat/impls/baij/mpi/mpibaij.c.html#line3957" class="">https://www.mcs.anl.gov/petsc/petsc-current/src/mat/impls/baij/mpi/mpibaij.c.html#line3957</a>)</div><div><br class=""></div><div>Now everything seems OK!</div><div>Thanks,</div><div>Pierre</div><br class=""><blockquote type="cite" class=""><div dir="ltr" class=""><div class="gmail_quote"><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" class=""><div class=""></div><div class="">Thanks,</div><div class="">Pierre</div><div class=""><br class=""></div><div class="">1) I don’t think this is tested right now, at least not in src/ksp/ksp/examples/tutorials</div><div class="">2) reproducer: src/ksp/ksp/examples/tutorials/ex2.c</div><div class="">$ mpirun -np 2 ./ex2 -pc_type redundant -mat_type sbaij</div><div class="">// error because trying to do LU with a symmetric matrix</div><div class="">$ mpirun -np 2 ./ex2 -pc_type redundant -mat_type sbaij -redundant_pc_type cholesky -ksp_view</div><div class="">// you’ll see: that MUMPS is being used, but since bs=1, it’s working, but it won’t for the general case</div><div class="">// the MatType is mpisbaij with "1 MPI processes" whereas with baij, it’s seqbaij</div><div class=""><br class=""></div></div></blockquote></div></div>
</blockquote></div><br class=""></body></html>