<html><body><div style="font-family: times new roman, new york, times, serif; font-size: 12pt; color: #000000"><div>OK, this is working now ! As the API changed between the latest stable and the master branch, I was actually not using the correct method.<br></div><div><br></div><div>Thanks Stefano,<br></div><div><br></div><div>Franck<br></div><div><br></div><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;" data-mce-style="border-left: 2px solid #1010FF; margin-left: 5px; padding-left: 5px; color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>De: </b>"Stefano Zampini" <stefano.zampini@gmail.com><br><b>À: </b>"Franck Houssen" <franck.houssen@inria.fr><br><b>Cc: </b>"petsc-dev" <petsc-dev@mcs.anl.gov>, "PETSc users list" <petsc-users@mcs.anl.gov>, "petsc-maint" <knepley@gmail.com><br><b>Envoyé: </b>Mercredi 24 Mai 2017 13:42:10<br><b>Objet: </b>Re: [petsc-dev] Using PETSc MatIS, how to get local matrix (= one domain) before and after assembly ?<br><div><br></div><br class=""><div><blockquote class=""><div class="">On May 24, 2017, at 11:46 AM, Franck Houssen <<a href="mailto:franck.houssen@inria.fr" class="" target="_blank" data-mce-href="mailto:franck.houssen@inria.fr">franck.houssen@inria.fr</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class=""><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;" class="" data-mce-style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;"><div class="">The code I sent compile and run at my side with petsc-3.7.6 (on debian/testing with gcc-6.3). The code you sent does not compile at my side. Anyway, no big deal.<br class=""></div><div class=""><br class=""></div></div></div></div></blockquote><div><br class=""></div><div>MatGetSubMatrix/MatGetSubMatrices have been renamed to MatCreateSubMatrix/MatCreateSubMatrices in petsc-dev. I thought you were using the master branch and not the latest release. Sorry for the confusion. </div><div><br class=""></div><div>To compile the code I have sent, just rename MatCreateSubMatrices with MatGetSubMatrices and it should work. </div><div class=""><a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetSubMatrices.html#MatGetSubMatrices" class="" target="_blank" data-mce-href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetSubMatrices.html#MatGetSubMatrices">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetSubMatrices.html#MatGetSubMatrices</a><br data-mce-bogus="1"></div><div class=""><a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetSubMatrix.html#MatGetSubMatrix" class="" target="_blank" data-mce-href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetSubMatrix.html#MatGetSubMatrix">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetSubMatrix.html#MatGetSubMatrix</a><br data-mce-bogus="1"></div><div class=""><br class=""></div></div><div class=""><br></div><div class=""><br class=""><blockquote class=""><div class=""><div class=""><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;" class="" data-mce-style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;"><div class="">The modification you propose as far as I understand is to replace "ISCreateGeneral(PETSC_COMM_WORLD" with "ISCreateGeneral(PETSC_COMM_SELF" : still not working at my side (empty dirichlet local matrix).</div><div class="">I will try to get that with a MPI matrix (that would contain same data that MatIS : that's what I tried to avoid as this doubles allocations - anyway, no big deal).<br class=""></div><div class=""><br class=""></div></div></div></div></blockquote><div><br class=""></div><div>In the code, you are already extracting submatrices from MPIAIJ format, not from MATIS. Attached a code that compiles and runs with petsc-3.7.6 </div><div><div class=""><br class=""></div><div class=""><br class=""></div></div><blockquote class=""><div class=""><div class=""><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;" class="" data-mce-style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;"><div class="">Franck<br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><hr id="zwchr" class=""><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class="" data-mce-style="border-left-width: 2px; border-left-style: solid; border-left-color: #1010ff; margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;"><b class="">De: </b>"Stefano Zampini" <<a href="mailto:stefano.zampini@gmail.com" class="" target="_blank" data-mce-href="mailto:stefano.zampini@gmail.com">stefano.zampini@gmail.com</a>><br class=""><b class="">À: </b>"Franck Houssen" <<a href="mailto:franck.houssen@inria.fr" class="" target="_blank" data-mce-href="mailto:franck.houssen@inria.fr">franck.houssen@inria.fr</a>><br class=""><b class="">Cc: </b>"petsc-dev" <<a href="mailto:petsc-dev@mcs.anl.gov" class="" target="_blank" data-mce-href="mailto:petsc-dev@mcs.anl.gov">petsc-dev@mcs.anl.gov</a>>, "PETSc users list" <<a href="mailto:petsc-users@mcs.anl.gov" class="" target="_blank" data-mce-href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>>, "petsc-maint" <<a href="mailto:knepley@gmail.com" class="" target="_blank" data-mce-href="mailto:knepley@gmail.com">knepley@gmail.com</a>><br class=""><b class="">Envoyé: </b>Mardi 23 Mai 2017 20:23:49<br class=""><b class="">Objet: </b>Re: [petsc-dev] Using PETSc MatIS, how to get local matrix (= one domain) before and after assembly ?<br class=""><div class=""><br class=""></div><br class=""><div class=""><blockquote class=""><div class="">On May 23, 2017, at 6:34 PM, Franck Houssen <<a href="mailto:franck.houssen@inria.fr" class="" target="_blank" data-mce-href="mailto:franck.houssen@inria.fr">franck.houssen@inria.fr</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class=""><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;" class="" data-mce-style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;"><div class="">OK. I am supposed to destroy the matrix returned by MatISGetMPIXAIJ ?<br class=""></div></div></div></div></blockquote>Yes<br class=""><blockquote class=""><div class=""><div class=""><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;" class="" data-mce-style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;"><div class="">Also, my example still not get the final assembled local matrix (the MatCreateSubMatrix returns an empty matrix) but as far as I understand my (global) index set is OK: what did I miss ?<br class=""></div></div></div></div></blockquote><div class=""><br class=""></div>I really doubt you can use the example you have sent. It doesn’t compile, as MatCreateSubMatrix needs an extra argument.</div><div class="">Attached a modified version that does what I guess is what you are looking for (sequential Dirichlet problems on the subdomains).</div><div class=""><br class=""><blockquote class=""><div class=""><div class=""><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;" class="" data-mce-style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;"><div class=""><br class=""></div><div class="">Franck<br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><hr id="zwchr" class=""><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class="" data-mce-style="border-left-width: 2px; border-left-style: solid; border-left-color: #1010ff; margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;"><b class="">De: </b>"Stefano Zampini" <<a href="mailto:stefano.zampini@gmail.com" class="" target="_blank" data-mce-href="mailto:stefano.zampini@gmail.com">stefano.zampini@gmail.com</a>><br class=""><b class="">À: </b>"Franck Houssen" <<a href="mailto:franck.houssen@inria.fr" class="" target="_blank" data-mce-href="mailto:franck.houssen@inria.fr">franck.houssen@inria.fr</a>><br class=""><b class="">Cc: </b>"petsc-dev" <<a href="mailto:petsc-dev@mcs.anl.gov" class="" target="_blank" data-mce-href="mailto:petsc-dev@mcs.anl.gov">petsc-dev@mcs.anl.gov</a>>, "PETSc users list" <<a href="mailto:petsc-users@mcs.anl.gov" class="" target="_blank" data-mce-href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>>, "petsc-maint" <<a href="mailto:knepley@gmail.com" class="" target="_blank" data-mce-href="mailto:knepley@gmail.com">knepley@gmail.com</a>><br class=""><b class="">Envoyé: </b>Mardi 23 Mai 2017 13:16:18<br class=""><b class="">Objet: </b>Re: [petsc-dev] Using PETSc MatIS, how to get local matrix (= one domain) before and after assembly ?<br class=""><div class=""><br class=""></div><div dir="auto" class="">MatISGetMPIXAIJ is collective, as it assembles the global operator. To get the matrices you are looking for, you should call MatCreateSubMatrix on the assembled global operator, with the global indices representing the subdomain problem. Each process needs to call both functions<div dir="auto" class=""><br class=""></div><div dir="auto" class="">Stefano</div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">Il 23 Mag 2017 11:41, "Franck Houssen" <<a href="mailto:franck.houssen@inria.fr" target="_blank" class="" data-mce-href="mailto:franck.houssen@inria.fr">franck.houssen@inria.fr</a>> ha scritto:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex" data-mce-style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;"><div class=""><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;" class="" data-mce-style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;"><div class="">I have a 3x3 global matrix made of two overlapping 2x2 local matrix (= diagonal with 1.). Each local matrix correspond to one domain (each domain is delegated to one MPI proc, so, I have 2 MPI procs because I have 2 domains).</div><div class="">This is the simplest possible example: I have two 2x2 (local) diag matrix that overlap so that the global matrix built from them is 1, 2, 1 on the diagonal (local contributions add up in the middle).<br class=""></div><div class=""><br class=""></div><div class="">Now, I need for each MPI proc to get the assembled local matrix (sometimes called the dirichlet matrix) : this is a local matrix (sequential - not distributed with MPI) that accounts for contribution of neighboring domains (MPI proc).<br class=""></div><div class=""><br class=""></div><div class="">How to get the local assembled matrix ? MatGetLocalSubMatrix does not work (throw error - see example attached). MatGetSubMatrix returns a MPI distributed matrix, not a local (sequential) one.<br class=""></div><div class=""><ol class=""><li class="">My understanding is that MatISGetMPIXAIJ should return a local matrix (sequential AIJ matrix) : the MPI in the name recall that you get the assembled matrix (with contributions from the shared border) from the other MPI processus. Correct ? In my simple example, I replaced MatGetLocalSubMatrix with MatISGetMPIXAIJ : I get a deadlock which was surprising to me... Is MatISGetMPIXAIJ a collective call ?<br class=""></li><li class="">Supposing this is a collective call (and that point 1 is not correct), I ride up MatISGetMPIXAIJ before the "if (rank > 0)" : I don't deadlock now, but it seems I get a global matrix which is not the assembled local matrix I am looking for.<br class=""></li><li class="">I am supposed to destroy the matrix returned by MatISGetMPIXAIJ ? (I believe yes - not sure as AFAIU wording should associate Destroy methods to Create methods)<br class=""></li></ol></div><div class="">Franck<br class=""></div><div class=""><br class=""></div><div class="">The git diff illustrate modifications I tried to add to the initial file attached to this thread:<br class=""></div><div class="">--- a/matISLocalMat.cpp<br class="">+++ b/matISLocalMat.cpp<br class="">@@ -31,6 +31,8 @@ int main(int argc,char **argv) {<br class=""> MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY); MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY);<br class=""> MatView(A, PETSC_VIEWER_STDOUT_WORLD); PetscViewerFlush(PETSC_VIEWER_STDOUT_WORLD); // Diag: 1, 2, 1<br class=""> <br class="">+ Mat assembledLocalMat;<br class="">+ MatISGetMPIXAIJ(A, MAT_INITIAL_MATRIX, &assembledLocalMat);<br class=""> if (rank > 0) { // Do not pollute stdout: print only 1 proc<br class=""> std::cout << std::endl << "non assembled local matrix:" << std::endl << std::endl;<br class=""> Mat nonAssembledLocalMat;<br class="">@@ -38,11 +40,10 @@ int main(int argc,char **argv) {<br class=""> MatView(nonAssembledLocalMat, PETSC_VIEWER_STDOUT_SELF); // Diag: 1, 1<br class=""> <br class=""> std::cout << std::endl << "assembled local matrix:" << std::endl << std::endl;<br class="">- Mat assembledLocalMat;<br class="">- IS is; ISCreateGeneral(PETSC_COMM_SELF, localSize, localIdx, PETSC_COPY_VALUES, &is);<br class="">- MatGetLocalSubMatrix(A, is, is, &assembledLocalMat); // KO ?!...<br class="">- MatView(assembledLocalMat, PETSC_VIEWER_STDOUT_SELF); // Would like to get => Diag: 2, 1<br class="">+ //IS is; ISCreateGeneral(PETSC_COMM_SELF, localSize, localIdx, PETSC_COPY_VALUES, &is);<br class="">+ //MatGetLocalSubMatrix(A, is, is, &assembledLocalMat); // KO ?!...<br class=""> }<br class="">+ MatView(assembledLocalMat, PETSC_VIEWER_STDOUT_WORLD); // Would like to get => Diag: 2, 1<br class=""><div class=""><br class=""></div></div><div class=""><br class=""></div><hr id="m_7985792947083527645zwchr" class=""><blockquote style="border-left-width: 2px; border-left-style: solid; border-left-color: rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;" class="" data-mce-style="border-left-width: 2px; border-left-style: solid; border-left-color: #1010ff; margin-left: 5px; padding-left: 5px; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt;"><b class="">De: </b>"Stefano Zampini" <<a href="mailto:stefano.zampini@gmail.com" target="_blank" class="" data-mce-href="mailto:stefano.zampini@gmail.com">stefano.zampini@gmail.com</a>><br class=""><b class="">À: </b>"petsc-maint" <<a href="mailto:knepley@gmail.com" target="_blank" class="" data-mce-href="mailto:knepley@gmail.com">knepley@gmail.com</a>><br class=""><b class="">Cc: </b>"petsc-dev" <<a href="mailto:petsc-dev@mcs.anl.gov" target="_blank" class="" data-mce-href="mailto:petsc-dev@mcs.anl.gov">petsc-dev@mcs.anl.gov</a>>, "PETSc users list" <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" class="" data-mce-href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>>, "Franck Houssen" <<a href="mailto:franck.houssen@inria.fr" target="_blank" class="" data-mce-href="mailto:franck.houssen@inria.fr">franck.houssen@inria.fr</a>><br class=""><b class="">Envoyé: </b>Dimanche 21 Mai 2017 22:51:34<br class=""><b class="">Objet: </b>Re: [petsc-dev] Using PETSc MatIS, how to get local matrix (= one domain) before and after assembly ?<br class=""><div class=""><br class=""></div><div dir="auto" class="">To assemble the operator in aij format, use <div dir="auto" class="">MatISGetMPIXAIJ</div><div dir="auto" class=""><a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatISGetMPIXAIJ.html" target="_blank" class="" data-mce-href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatISGetMPIXAIJ.html">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatISGetMPIXAIJ.html</a><br class=""></div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">Il 21 Mag 2017 18:43, "Matthew Knepley" <<a href="mailto:knepley@gmail.com" target="_blank" class="" data-mce-href="mailto:knepley@gmail.com">knepley@gmail.com</a>> ha scritto:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex" data-mce-style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;"><div dir="ltr" class=""><div class="gmail_extra"><div class="gmail_quote">On Sun, May 21, 2017 at 11:23 AM, Franck Houssen <span dir="ltr" class=""><<a href="mailto:franck.houssen@inria.fr" target="_blank" class="" data-mce-href="mailto:franck.houssen@inria.fr">franck.houssen@inria.fr</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left: 1px solid #cccccc; padding-left: 1ex;"><div class=""><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;" class="" data-mce-style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;"><div class="">I have a 3x3 global matrix is built (diag: 1, 2, 1): it's made of 2 overlapping 2x2 local matrix (diag: 1, 1).</div><div class="">Getting non assembled local matrix is OK with MatISGetLocalMat.</div><div class="">How to get assembled local matrix (initial local matrix + neigbhor contributions on the borders) ? (expected result is diag: 2, 1)</div></div></div></blockquote><div class=""><br class=""></div><div class="">You can always use</div><div class=""><br class=""></div><div class=""><a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetSubMatrix.html" target="_blank" class="" data-mce-href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetSubMatrix.html">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetSubMatrix.html</a><br class=""></div><div class=""><a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetSubMatrices.html" target="_blank" class="" data-mce-href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetSubMatrices.html">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetSubMatrices.html</a><br class=""></div><div class=""><br class=""></div><div class="">to get copies, but if you just want to build things, you can use</div><div class=""><br class=""></div><div class=""><a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetLocalSubMatrix.html" target="_blank" class="" data-mce-href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetLocalSubMatrix.html">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetLocalSubMatrix.html</a><br class=""></div><div class=""><br class=""></div><div class=""> Thanks,</div><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" data-mce-style="margin: 0px 0px 0px 0.8ex; border-left: 1px solid #cccccc; padding-left: 1ex;"><div class=""><div style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;" class="" data-mce-style="font-family: 'times new roman', 'new york', times, serif; font-size: 12pt;"><div class="">Franck<br class=""></div></div></div></blockquote></div><br class=""><br class="" clear="all"><div class=""><br class=""></div>-- <br class=""><div class="m_7985792947083527645m_-6074799036086587145gmail_signature"><div dir="ltr" class=""><div class="m_7985792947083527645m_-6074799036086587145gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class="m_7985792947083527645m_-6074799036086587145gmail_signature"><br class=""></div><div class="m_7985792947083527645m_-6074799036086587145gmail_signature"><a href="http://www.caam.rice.edu/~mk51/" target="_blank" class="" data-mce-href="http://www.caam.rice.edu/~mk51/">http://www.caam.rice.edu/~mk51/</a><br class=""></div></div></div></div></div></blockquote></div></div></blockquote><div class=""><br class=""></div></div></div></blockquote></div></div></blockquote><div class=""><br class=""></div></div></div><span id="cid:DCC1DD61-38AA-4730-AF59-728997508F5D" class=""><matISLocalMat.cpp></span></div></blockquote></div><br class=""></blockquote><div class=""><br class=""></div></div></div></div></blockquote></div><br><br></blockquote><div><br></div></div></body></html>