<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">Matt:</div><div class="gmail_quote">On mastter, we have</div><div class="gmail_quote"><br></div><div class="gmail_quote"><div class="gmail_quote">  if (C->submat_singleis) { </div><div class="gmail_quote">      /* flag is set in PCSetUp_ASM() to skip several MPIU_Allreduce() */<br></div><div class="gmail_quote">      MatGetSubMatrices_MPIAIJ_SingleIS(C,ismax,isrow,iscol,scall,submat);</div><div class="gmail_quote">    PetscFunctionReturn(0);</div><div class="gmail_quote">  }</div><div class="gmail_quote">...</div><div class="gmail_quote">MatGetSubMatrices_MPIAIJ_Local(C,...)<br></div><div class="gmail_quote"><br></div><div class="gmail_quote">i.e., MatGetSubMatrices_MPIAIJ_SingleIS() is called when ismax=1 for each process.</div><div class="gmail_quote">General case is handled by MatGetSubMatrices_MPIAIJ_Local(C,...)</div><div class="gmail_quote"><br></div><div class="gmail_quote">I've optimized this routine and will merge it to master.</div><div class="gmail_quote"><br></div><div class="gmail_quote">Hong</div></div><div class="gmail_quote"><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">It comes from here:<div><br></div><div>  <a href="https://bitbucket.org/petsc/petsc/commits/c10200c1442b553b7ad65c70101560db4fa22e78" target="_blank">https://bitbucket.org/petsc/<wbr>petsc/commits/<wbr>c10200c1442b553b7ad65c70101560<wbr>db4fa22e78</a></div><div><br></div><div>If we ask for more than 1 matrix, it dispatches to</div><div><br></div><div>  MatGetSubMatrices_MPIAIJ_<wbr>SingleIS()</div><div><br></div><div>but then fails here</div><div><br></div><div>  <a href="https://bitbucket.org/petsc/petsc/annotate/2e559809f9aee9c95ee79eb0939630cfe5502c8d/src/mat/impls/aij/mpi/mpiov.c?at=master&fileviewer=file-view-default#mpiov.c-1306" target="_blank">https://bitbucket.org/petsc/<wbr>petsc/annotate/<wbr>2e559809f9aee9c95ee79eb0939630<wbr>cfe5502c8d/src/mat/impls/aij/<wbr>mpi/mpiov.c?at=master&<wbr>fileviewer=file-view-default#<wbr>mpiov.c-1306</a></div><div><br></div><div>because ismax > 1. I think the ismax check needs to move up to here</div><div><br></div><div>  <a href="https://bitbucket.org/petsc/petsc/annotate/2e559809f9aee9c95ee79eb0939630cfe5502c8d/src/mat/impls/aij/mpi/mpiov.c?at=master&fileviewer=file-view-default#mpiov.c-2012" target="_blank">https://bitbucket.org/petsc/<wbr>petsc/annotate/<wbr>2e559809f9aee9c95ee79eb0939630<wbr>cfe5502c8d/src/mat/impls/aij/<wbr>mpi/mpiov.c?at=master&<wbr>fileviewer=file-view-default#<wbr>mpiov.c-2012</a></div><div><br></div><div>but I don't know for sure. Please fix this since it is breaking PyLith.</div><div><br></div><div>  Matt<span class="gmail-HOEnZb"><font color="#888888"><br clear="all"><div><br></div>-- <br><div class="gmail-m_-6080241473301409647gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</font></span></div></div>
</blockquote></div><br></div></div>