<div dir="ltr">Stefano:<div>This patch </div><div><div>diff --git a/src/mat/impls/aij/seq/aij.c b/src/mat/impls/aij/seq/aij.c</div><div>index df16825..5c40de6 100644</div><div>--- a/src/mat/impls/aij/seq/aij.c</div><div>+++ b/src/mat/impls/aij/seq/aij.c</div><div>@@ -2542,11 +2542,14 @@ PetscErrorCode MatDestroySubMatrices_SeqAIJ(PetscInt n,Mat *mat[])</div><div> c = (Mat_SeqAIJ*)C->data;</div><div> submatj = c->submatis1;</div><div> if (submatj) {</div><div>- ierr = submatj->destroy(C);CHKERRQ(ierr);</div><div>- ierr = MatDestroySubMatrix_Private(submatj);CHKERRQ(ierr);</div><div>- ierr = PetscLayoutDestroy(&C->rmap);CHKERRQ(ierr);</div><div>- ierr = PetscLayoutDestroy(&C->cmap);CHKERRQ(ierr);</div><div>- ierr = PetscHeaderDestroy(&C);CHKERRQ(ierr);</div><div>+ if (--((PetscObject)C)->refct > 0) {</div><div>+ } else {</div><div>+ ierr = submatj->destroy(C);CHKERRQ(ierr);</div><div>+ ierr = MatDestroySubMatrix_Private(submatj);CHKERRQ(ierr);</div><div>+ ierr = PetscLayoutDestroy(&C->rmap);CHKERRQ(ierr);</div><div>+ ierr = PetscLayoutDestroy(&C->cmap);CHKERRQ(ierr);</div><div>+ ierr = PetscHeaderDestroy(&C);CHKERRQ(ierr);</div><div>+ }</div><div> } else {</div><div><br></div><div>should fix the problem. Let me know your thought.</div><div>Hong</div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Jan 14, 2018 at 9:09 PM, Hong <span dir="ltr"><<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">Stefano:</div><div class="gmail_quote">Sequential version works because it does not use common data structure.</div><div class="gmail_quote">The old version of MatCreateSubMatrices() does not reuse internal data structures, thus inefficient.</div><div class="gmail_quote">During its optimization, the internal data structures are introduced. </div><div class="gmail_quote"><br></div><div class="gmail_quote">I'll check it and see if I can provide these type of usage.<span class="gmail-HOEnZb"><font color="#888888"><br></font></span></div><span class="gmail-HOEnZb"><font color="#888888"><div class="gmail_quote"><br></div></font></span><div class="gmail_quote"><span class="gmail-HOEnZb"><font color="#888888">Hong</font></span><div><div class="gmail-h5"><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><br><div><span class="gmail-m_6876046937825509204gmail-"><blockquote type="cite"><div>On Jan 14, 2018, at 7:16 AM, Hong <<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>> wrote:</div><br class="gmail-m_6876046937825509204gmail-m_-1224139957275575380Apple-interchange-newline"><div><div dir="ltr"><div>Stefano,</div><div>MatCreateSubMatrices() must be destroyed by MatDestroySubMatrices()</div><div>because all submatrices in the array share common internal data structure for reuse. </div><div><br></div><div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(52,188,38)"><span style="font-variant-ligatures:no-common-ligatures">+ ierr = PetscObjectReference((PetscObj<wbr>ect)submat);CHKERRQ(ierr);</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(52,188,38)"><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(52,188,38)"><span style="font-variant-ligatures:no-common-ligatures">Only </span><span style="font-variant-ligatures:no-common-ligatures">submat = </span><span style="color:rgb(80,0,80);font-variant-ligatures:no-common-ligatures">subma</span><span style="color:rgb(80,0,80);font-variant-ligatures:no-common-ligatures">trices[0] adds reference count.</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(52,188,38)"><br></div><div style="color:rgb(80,0,80);margin:0px;font-size:14px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"> ierr = MatDestroySubMatrices(1,&subma<wbr>trices);CHKERRQ(ierr);</span></div><div style="color:rgb(80,0,80);margin:0px;font-size:14px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Now, the internal common data structure is destroyed.</span></div><div style="color:rgb(80,0,80);margin:0px;font-size:14px;line-height:normal;font-family:Menlo"><br></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(52,188,38)"><span style="font-variant-ligatures:no-common-ligatures">+ ierr = MatDestroy(&submat);CHKERRQ(ie<wbr>rr);</span></div></div><div><span style="font-variant-ligatures:no-common-ligatures">I guess submat becomes an orphan, it cannot be destroyed using previous ops->destroy routine.</span></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div><span style="font-variant-ligatures:no-common-ligatures">Shall we allow user to pick a single matrix out of submatrices array, and use/destroy it as a regular single matrix?</span></div><div><br></div></div></div></blockquote><div><br></div></span><div>I accept the fact that the array of submatrices obtained by MatCreateSubMatrices shall be destroyed by MatDestroySubMatrices.</div><div>However, not being able to take reference on one of the submats is not properly PETSc philosophy :-), as the Mats returned do not behave like proper Mat objects</div><div>How difficult is to allow this? and what is the common data structure the submats share?</div><div><div class="gmail-m_6876046937825509204gmail-h5"><br><blockquote type="cite"><div><div dir="ltr"><div>Hong<br><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Jan 13, 2018 at 1:09 PM, Stefano Zampini <span dir="ltr"><<a href="mailto:stefano.zampini@gmail.com" target="_blank">stefano.zampini@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div>Cc'ing petsc-dev<br><div class="gmail_quote">---------- Messaggio inoltrato ----------<br>Da: "Stefano Zampini" <<a href="mailto:stefano.zampini@gmail.com" target="_blank">stefano.zampini@gmail.com</a>><br>Data: 12 Gen 2018 9:32 PM<br>Oggetto: Re: MatDestroySubMatrices_SeqAIJ and reference counting<br>A: "Hong" <<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>><br>Cc: <br><div><div class="gmail-m_6876046937825509204gmail-m_-1224139957275575380gmail-h5"><br type="attribution"><blockquote class="gmail-m_6876046937825509204gmail-m_-1224139957275575380gmail-m_-5358193299216387954quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><br><div><div class="gmail-m_6876046937825509204gmail-m_-1224139957275575380gmail-m_-5358193299216387954quoted-text"><blockquote type="cite"><div>Sorry, I do not understand your question. Where comes matrix B?</div></blockquote><div><br></div></div><div>It’s not important where matrix B comes from. The error is reproducible by just increasing the reference count of any of the submats. See below the patch for ex4.c</div><div class="gmail-m_6876046937825509204gmail-m_-1224139957275575380gmail-m_-5358193299216387954quoted-text"><br><blockquote type="cite"><div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">Can you give me an example of this error?</div></div></div></div></blockquote><div><br></div></div>here you have<br><div><br></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"><b>diff --git a/src/mat/examples/tests/ex4.c b/src/mat/examples/tests/ex4.c</b></span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"><b>index 0555a54d9d..1d1961e4fa 100644</b></span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"><b>--- a/src/mat/examples/tests/ex4.c</b></span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"><b>+++ b/src/mat/examples/tests/ex4.c</b></span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(51,187,200)">@@ -68,7 +68,9 @@</span><span style="font-variant-ligatures:no-common-ligatures"> int main(int argc,char **argv)</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"> ierr = MatView(submat,sviewer);CHKERR<wbr>Q(ierr);</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"> ierr = PetscViewerRestoreSubViewer(PE<wbr>TSC_VIEWER_STDOUT_WORLD,PETSC_<wbr>COMM_SELF,&sviewer);CHKERRQ(ie<wbr>rr);</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"> ierr = PetscViewerFlush(PETSC_VIEWER_<wbr>STDOUT_WORLD);CHKERRQ(ierr);</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(52,188,38);background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures">+ ierr = PetscObjectReference((PetscObj<wbr>ect)submat);CHKERRQ(ierr);</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"> ierr = MatDestroySubMatrices(1,&subma<wbr>trices);CHKERRQ(ierr);</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;color:rgb(52,188,38);background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures">+ ierr = MatDestroy(&submat);CHKERRQ(ie<wbr>rr);</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255);min-height:16px"><span style="font-variant-ligatures:no-common-ligatures"> </span><br class="gmail-m_6876046937825509204gmail-m_-1224139957275575380gmail-m_-5358193299216387954m_326818260560541686webkit-block-placeholder"></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"> /* Form submatrix with rows 2-4 and all columns */</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures"> ierr = ISDestroy(&icol);CHKERRQ(ierr)<wbr>;</span></div><div class="gmail-m_6876046937825509204gmail-m_-1224139957275575380gmail-m_-5358193299216387954elided-text"><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><blockquote type="cite"><div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">Hong</div><div class="gmail_quote"><br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word">Hong,<div><br></div><div>can you explain what’s the rationale behind calling explicitly the Layout and header destroy for the submit case in the loop here <a href="https://bitbucket.org/petsc/petsc/src/ac3af1b492556bac9856a6aee1c73992bd0b1779/src/mat/impls/aij/seq/aij.c?at=master&fileviewer=file-view-default#aij.c-2531" target="_blank">https://bitbucket.org/pet<wbr>sc/petsc/src/ac3af1b492556bac9<wbr>856a6aee1c73992bd0b1779/src/ma<wbr>t/impls/aij/seq/aij.c?at=maste<wbr>r&fileviewer=file-view-default<wbr>#aij.c-2531</a></div><div><br></div><div>A code like this fails since you don’t take into account reference counting on the submatrices.</div><div><br></div><div>MatCreateSubMatrices(A,….&subm<wbr>ats)</div><div>PetscObjectCompose(B,”_XXXX”,s<wbr>ubmats[0);</div><div>…..</div><div>MatDestroySubMatrices(..,subma<wbr>ts);</div><div>MatDestroy(B); //Error, corrupt argument when trying to destroy composed objects</div><div><br></div><div>Here is a representative valgrind stack trace</div><div><br></div><div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures">==90133== Invalid read of size 4</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures">==90133== at 0x100368F63: PetscCheckPointer (/Users/szampini/software/pets<wbr>c/src/sys/error/checkptr.c:108<wbr>)</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures">==90133== by 0x1001ACA9F: PetscObjectDereference (/Users/szampini/software/pets<wbr>c/src/sys/objects/inherit.c:56<wbr>4)</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures">==90133== by 0x10019AC07: PetscObjectListDestroy (/Users/szampini/software/pets<wbr>c/src/sys/objects/olist.c:154)</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures">==90133== by 0x1001A8176: PetscHeaderDestroy_Private (/Users/szampini/software/pets<wbr>c/src/sys/objects/inherit.c:11<wbr>5)</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures">==90133== by 0x100626CAD: MatDestroy (/Users/szampini/software/pets<wbr>c/src/mat/interface/matrix.c:1<wbr>237)</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">==90133== Address 0x106c4c320 is 1,632 bytes inside a block of size 4,420 free'd</div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures">==90133== at 0x10014C9F3: free (in /usr/local/Cellar/valgrind/3.1<wbr>3.0/lib/valgrind/vgpreload_mem<wbr>check-amd64-darwin.so)</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures">==90133== by 0x100229B7A: PetscFreeAlign (/Users/szampini/software/pets<wbr>c/src/sys/memory/mal.c:88)</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures">==90133== by 0x10022D771: PetscTrFreeDefault (/Users/szampini/software/pets<wbr>c/src/sys/memory/mtr.c:309)</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures">==90133== by 0x100A7034D: MatDestroySubMatrices_SeqAIJ (/Users/szampini/software/pets<wbr>c/src/mat/impls/aij/seq/aij.c:<wbr>2549)</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures">==90133== by 0x1006260BD: MatDestroySubMatrices (/Users/szampini/software/pets<wbr>c/src/mat/interface/matrix.c:6<wbr>957)</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures">==90133== by 0x1006D82EC: _DestroyContainer</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures">==90133== by 0x1001AFA31: PetscContainerDestroy (/Users/szampini/software/pets<wbr>c/src/sys/objects/inherit.c:89<wbr>9)</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures">==90133== by 0x1001ACBE5: PetscObjectDereference (/Users/szampini/software/pets<wbr>c/src/sys/objects/inherit.c:56<wbr>6)</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures">==90133== by 0x10019AC07: PetscObjectListDestroy (/Users/szampini/software/pets<wbr>c/src/sys/objects/olist.c:154)</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures">==90133== by 0x1001A8176: PetscHeaderDestroy_Private (/Users/szampini/software/pets<wbr>c/src/sys/objects/inherit.c:11<wbr>5)</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures">==90133== by 0x100626CAD: MatDestroy (/Users/szampini/software/pets<wbr>c/src/mat/interface/matrix.c:1<wbr>237)</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)">==90133== Block was alloc'd at</div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures">==90133== at 0x10014C616: malloc (in /usr/local/Cellar/valgrind/3.1<wbr>3.0/lib/valgrind/vgpreload_mem<wbr>check-amd64-darwin.so)</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures">==90133== by 0x1002299AC: PetscMallocAlign (/Users/szampini/software/pets<wbr>c/src/sys/memory/mal.c:48)</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures">==90133== by 0x10022CA73: PetscTrMallocDefault (/Users/szampini/software/pets<wbr>c/src/sys/memory/mtr.c:183)</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures">==90133== by 0x10022B328: PetscMallocA (/Users/szampini/software/pets<wbr>c/src/sys/memory/mal.c:396)</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures">==90133== by 0x100C12E05: MatCreate (/Users/szampini/software/pets<wbr>c/src/mat/utils/gcreate.c:89)</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures">==90133== by 0x100B839F6: MatCreateSubMatrices_MPIAIJ_Lo<wbr>cal (/Users/szampini/software/pets<wbr>c/src/mat/impls/aij/mpi/mpiov.<wbr>c:2554)</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures">==90133== by 0x100B7DEE7: MatCreateSubMatrices_MPIAIJ (/Users/szampini/software/pets<wbr>c/src/mat/impls/aij/mpi/mpiov.<wbr>c:2038)</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><span style="font-variant-ligatures:no-common-ligatures">==90133== by 0x10066703A: MatCreateSubMatrices (/Users/szampini/software/pets<wbr>c/src/mat/interface/matrix.c:6<wbr>806)</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Menlo;background-color:rgb(255,255,255)"><br></div></div></div></blockquote></div><br></div></div>
</div></blockquote></div></div><br></div></blockquote></div></div></div><br></div></div>
</blockquote></div><br></div></div></div>
</div></blockquote></div></div></div><br></div></blockquote></div></div></div><br></div></div>
</blockquote></div><br></div></div></div>