<html><body><div style="font-family: times new roman, new york, times, serif; font-size: 12pt; color: #000000"><div>I have a 3x3 global matrix made of two overlapping 2x2 local matrix (= diagonal with 1.). Each local matrix correspond to one domain (each domain is delegated to one MPI proc, so, I have 2 MPI procs because I have 2 domains).</div><div>This is the simplest possible example: I have two 2x2 (local) diag matrix that overlap so that the global matrix built from them is 1, 2, 1 on the diagonal (local contributions add up in the middle).<br></div><div><br></div><div>Now, I need for each MPI proc to get the assembled local matrix (sometimes called the dirichlet matrix) : this is a local matrix (sequential - not distributed with MPI) that accounts for contribution of neighboring domains (MPI proc).<br></div><div><br></div><div>How to get the local assembled matrix ? MatGetLocalSubMatrix does not work (throw error - see example attached). MatGetSubMatrix returns a MPI distributed matrix, not a local (sequential) one.<br></div><div><ol><li>My understanding is that MatISGetMPIXAIJ should return a local matrix (sequential AIJ matrix) : the MPI in the name recall that you get the assembled matrix (with contributions from the shared border) from the other MPI processus. Correct ? In my simple example, I replaced MatGetLocalSubMatrix with MatISGetMPIXAIJ : I get a deadlock which was surprising to me... Is MatISGetMPIXAIJ a collective call ?<br></li><li>Supposing this is a collective call (and that point 1 is not correct), I ride up MatISGetMPIXAIJ before the "if (rank > 0)" : I don't deadlock now, but it seems I get a global matrix which is not the assembled local matrix I am looking for.<br></li><li>I am supposed to destroy the matrix returned by MatISGetMPIXAIJ ? (I believe yes - not sure as AFAIU wording should associate Destroy methods to Create methods)<br></li></ol></div><div>Franck<br></div><div><br></div><div>The git diff illustrate modifications I tried to add to the initial file attached to this thread:<br></div><div>--- a/matISLocalMat.cpp<br>+++ b/matISLocalMat.cpp<br>@@ -31,6 +31,8 @@ int main(int argc,char **argv) {<br> MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY);<br> MatView(A, PETSC_VIEWER_STDOUT_WORLD); PetscViewerFlush(PETSC_VIEWER_STDOUT_WORLD); // Diag: 1, 2, 1<br> <br>+ Mat assembledLocalMat;<br>+ MatISGetMPIXAIJ(A, MAT_INITIAL_MATRIX, &assembledLocalMat);<br> if (rank > 0) { // Do not pollute stdout: print only 1 proc<br> std::cout << std::endl << "non assembled local matrix:" << std::endl << std::endl;<br> Mat nonAssembledLocalMat;<br>@@ -38,11 +40,10 @@ int main(int argc,char **argv) {<br> MatView(nonAssembledLocalMat, PETSC_VIEWER_STDOUT_SELF); // Diag: 1, 1<br> <br> std::cout << std::endl << "assembled local matrix:" << std::endl << std::endl;<br>- Mat assembledLocalMat;<br>- IS is; ISCreateGeneral(PETSC_COMM_SELF, localSize, localIdx, PETSC_COPY_VALUES, &is);<br>- MatGetLocalSubMatrix(A, is, is, &assembledLocalMat); // KO ?!...<br>- MatView(assembledLocalMat, PETSC_VIEWER_STDOUT_SELF); // Would like to get => Diag: 2, 1<br>+ //IS is; ISCreateGeneral(PETSC_COMM_SELF, localSize, localIdx, PETSC_COPY_VALUES, &is);<br>+ //MatGetLocalSubMatrix(A, is, is, &assembledLocalMat); // KO ?!...<br> }<br>+ MatView(assembledLocalMat, PETSC_VIEWER_STDOUT_WORLD); // Would like to get => Diag: 2, 1<br><br></div><div><br></div><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;" data-mce-style="border-left: 2px solid #1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>De: </b>"Stefano Zampini" <stefano.zampini@gmail.com><br><b>À: </b>"petsc-maint" <knepley@gmail.com><br><b>Cc: </b>"petsc-dev" <petsc-dev@mcs.anl.gov>, "PETSc users list" <petsc-users@mcs.anl.gov>, "Franck Houssen" <franck.houssen@inria.fr><br><b>Envoyé: </b>Dimanche 21 Mai 2017 22:51:34<br><b>Objet: </b>Re: [petsc-dev] Using PETSc MatIS, how to get local matrix (= one domain) before and after assembly ?<br><div><br></div><div dir="auto">To assemble the operator in aij format, use <div dir="auto">MatISGetMPIXAIJ</div><div dir="auto"><a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatISGetMPIXAIJ.html" target="_blank" data-mce-href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatISGetMPIXAIJ.html">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatISGetMPIXAIJ.html</a><br data-mce-bogus="1"></div></div><div class="gmail_extra"><br><div class="gmail_quote">Il 21 Mag 2017 18:43, "Matthew Knepley" <<a href="mailto:knepley@gmail.com" target="_blank" data-mce-href="mailto:knepley@gmail.com">knepley@gmail.com</a>> ha scritto:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex" data-mce-style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Sun, May 21, 2017 at 11:23 AM, Franck Houssen <span dir="ltr"><<a href="mailto:franck.houssen@inria.fr" target="_blank" data-mce-href="mailto:franck.houssen@inria.fr">franck.houssen@inria.fr</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left: 1px solid #cccccc; padding-left: 1ex;"><div><div style="font-family:"times new roman","new york",times,serif;font-size:12pt;color:rgb(0,0,0)" data-mce-style="font-family: 'times new roman','new york',times,serif; font-size: 12pt; color: #000000;"><div>I have a 3x3 global matrix is built (diag: 1, 2, 1): it's made of 2 overlapping 2x2 local matrix (diag: 1, 1).</div><div>Getting non assembled local matrix is OK with MatISGetLocalMat.</div><div>How to get assembled local matrix (initial local matrix + neigbhor contributions on the borders) ? (expected result is diag: 2, 1)</div></div></div></blockquote><div><br></div><div>You can always use</div><div><br></div><div><a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetSubMatrix.html" target="_blank" data-mce-href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetSubMatrix.html">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetSubMatrix.html</a><br></div><div><a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetSubMatrices.html" target="_blank" data-mce-href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetSubMatrices.html">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetSubMatrices.html</a><br></div><div><br></div><div>to get copies, but if you just want to build things, you can use</div><div><br></div><div><a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetLocalSubMatrix.html" target="_blank" data-mce-href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetLocalSubMatrix.html">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetLocalSubMatrix.html</a><br></div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left: 1px solid #cccccc; padding-left: 1ex;"><div><div style="font-family:"times new roman","new york",times,serif;font-size:12pt;color:rgb(0,0,0)" data-mce-style="font-family: 'times new roman','new york',times,serif; font-size: 12pt; color: #000000;"><div>Franck<br></div></div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="m_-6074799036086587145gmail_signature"><div dir="ltr"><div class="m_-6074799036086587145gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div class="m_-6074799036086587145gmail_signature"><br></div><div class="m_-6074799036086587145gmail_signature"><a href="http://www.caam.rice.edu/~mk51/" target="_blank" data-mce-href="http://www.caam.rice.edu/~mk51/">http://www.caam.rice.edu/~mk51/</a><br></div></div></div></div></div></blockquote></div></div></blockquote><div><br></div></div></body></html>