<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Wed, Mar 16, 2016 at 1:21 AM, Tobin Isaac <span dir="ltr"><<a href="mailto:tisaac@uchicago.edu" target="_blank">tisaac@uchicago.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Tue, Mar 15, 2016 at 11:47:53PM -0500, Barry Smith wrote:<br>
><br>
> This is a really nasty problem. The example as previously written was completely reasonable, so your fix is a total hack :-). All the circular reference counting in PETSc is problematic because it is so dependent on exactly the details of how each particular object and its relationships are handled.<br>
<br>
</span>I agree that the need to call VecSetDM() in that case is bad, and it<br>
stems from assuming that the recycled vectors reference the dm: if<br>
we're going to count circular references, we should actually count<br>
them instead of assuming they exist.<br>
<br>
Where I added DMDestroy() in the Coarsen() routine, however, was in<br>
line with the kind of code we typically expect from users.<br>
<span class=""><br>
><br>
> Do we really need to even allow these nasty circular relationships to exist? What would we lose if we, for example, removed the two way relationships between the DMs and the Vecs? Just a little efficiency in not needing to create new Vecs because we can recycle them? But at the cost of very difficult to debug code that "should just work?" Similarly the nasty circular dependencies with dm->coarseMesh is done for "efficiency", is there a way to keep the efficiency but not the tricking dependencies?<br>
<br>
</span>I introduced dm->fineMesh, and I'll consider removing it, but having<br>
both dm->coarseMesh and dm->fineMesh references is about more than<br>
just efficiency. Particularly with the inverted multigrid that<br>
everyone's working on, there are workflows where it is more natural<br>
for the user to just maintain a handle on the coarsest mesh, not the<br>
finest mesh.</blockquote><div><br></div><div>I think all the references here are completely appropriate. I don't see another way of making</div><div>many things work than to have the DM know its pool of named vectors. I think it may be that </div><div>our simplistic reference counting scheme is at fault.</div><div><br></div><div>However, in this case, I think its clear that your function violated the implied contract for</div><div>DMCreateGloba/LocalVector(). This should be put in the documentation that the returned</div><div>vectors need to have the DM set to that DM.</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">
><br>
> I accept your "fix", thanks for figuring it out so quickly! but don't like it :-).<br>
><br>
> Barry<br>
><br>
><br>
><br>
> > On Mar 15, 2016, at 11:30 PM, Tobin Isaac <<a href="mailto:tisaac@uchicago.edu">tisaac@uchicago.edu</a>> wrote:<br>
> ><br>
> ><br>
> > I pushed a fix. There's a long explanation in the commit message:<br>
> > while this could be called user error, the cycle counting isn't very<br>
> > robust and should probably be changed.<br>
> ><br>
> > Toby<br>
> ><br>
> > On Tue, Mar 15, 2016 at 09:54:53PM -0500, Barry Smith wrote:<br>
> >><br>
> >> Dang, dang, dang, I can't believe I fell for that git trapdoor. Ok pushed now.<br>
> >><br>
> >> Barry<br>
> >><br>
> >>> On Mar 15, 2016, at 9:46 PM, Tobin Isaac <<a href="mailto:tisaac@uchicago.edu">tisaac@uchicago.edu</a>> wrote:<br>
> >>><br>
> >>><br>
> >>> Barry, please check in ex65.c<br>
> >>><br>
> >>> On Sun, Mar 13, 2016 at 04:20:06PM -0500, Barry Smith wrote:<br>
> >>>><br>
> >>>> Toby,<br>
> >>>><br>
> >>>> I'm trying to put together a very simple but complete DMSHELL example for <a href="mailto:popov@uni-mainz.de">popov@uni-mainz.de</a> and having some trouble which I think it might point to a bug or logical error in the code you wrote for maintaining dm->coarseMesh and dm->fineMesh and stuff.<br>
> >>>><br>
> >>>> $ petscmpiexec -valgrind -n 1 ./ex65 -pc_type mg -pc_mg_levels 2<br>
> >>>> ==80209== Invalid read of size 8<br>
> >>>> ==80209== at 0x100A9E2D5: DMCountNonCyclicReferences (dm.c:500)<br>
> >>>> ==80209== by 0x100A8F70A: DMDestroy (dm.c:573)<br>
> >>>> ==80209== by 0x101221BBE: KSPDestroy (itfunc.c:985)<br>
> >>>> ==80209== by 0x1010BCBFC: PCDestroy_MG (mg.c:302)<br>
> >>>> ==80209== by 0x1010E23F7: PCDestroy (precon.c:122)<br>
> >>>> ==80209== by 0x101221C3A: KSPDestroy (itfunc.c:986)<br>
> >>>> ==80209== by 0x100001C4C: main (in ./ex65)<br>
> >>>> ==80209== Address 0x10398fd68 is 5,864 bytes inside a block of size 6,196 free'd<br>
> >>>> ==80209== at 0x10001595D: free (vg_replace_malloc.c:480)<br>
> >>>> ==80209== by 0x1000FE393: PetscFreeAlign (mal.c:72)<br>
> >>>> ==80209== by 0x100100D1E: PetscTrFreeDefault (mtr.c:315)<br>
> >>>> ==80209== by 0x100A91C5A: DMDestroy (dm.c:716)<br>
> >>>> ==80209== by 0x1010E2478: PCDestroy (precon.c:123)<br>
> >>>> ==80209== by 0x101221C3A: KSPDestroy (itfunc.c:986)<br>
> >>>> ==80209== by 0x1010BCBFC: PCDestroy_MG (mg.c:302)<br>
> >>>> ==80209== by 0x1010E23F7: PCDestroy (precon.c:122)<br>
> >>>> ==80209== by 0x101221C3A: KSPDestroy (itfunc.c:986)<br>
> >>>> ==80209== by 0x100001C4C: main (in ./ex65)<br>
> >>>> ==80209==<br>
> >>>> ==80209== Invalid read of size 8<br>
> >>>> ==80209== at 0x100A9E2D5: DMCountNonCyclicReferences (dm.c:500)<br>
> >>>> ==80209== by 0x100A8F70A: DMDestroy (dm.c:573)<br>
> >>>> ==80209== by 0x1010E2478: PCDestroy (precon.c:123)<br>
> >>>> ==80209== by 0x101221C3A: KSPDestroy (itfunc.c:986)<br>
> >>>> ==80209== by 0x1010BCBFC: PCDestroy_MG (mg.c:302)<br>
> >>>> ==80209== by 0x1010E23F7: PCDestroy (precon.c:122)<br>
> >>>> ==80209== by 0x101221C3A: KSPDestroy (itfunc.c:986)<br>
> >>>> ==80209== by 0x100001C4C: main (in ./ex65)<br>
> >>>> ==80209== Address 0x10398fd68 is 5,864 bytes inside a block of size 6,196 free'd<br>
> >>>> ==80209== at 0x10001595D: free (vg_replace_malloc.c:480)<br>
> >>>> ==80209== by 0x1000FE393: PetscFreeAlign (mal.c:72)<br>
> >>>> ==80209== by 0x100100D1E: PetscTrFreeDefault (mtr.c:315)<br>
> >>>> ==80209== by 0x100A91C5A: DMDestroy (dm.c:716)<br>
> >>>> ==80209== by 0x1010E2478: PCDestroy (precon.c:123)<br>
> >>>> ==80209== by 0x101221C3A: KSPDestroy (itfunc.c:986)<br>
> >>>> ==80209== by 0x1010BCBFC: PCDestroy_MG (mg.c:302)<br>
> >>>> ==80209== by 0x1010E23F7: PCDestroy (precon.c:122)<br>
> >>>> ==80209== by 0x101221C3A: KSPDestroy (itfunc.c:986)<br>
> >>>> ==80209== by 0x100001C4C: main (in ./ex65)<br>
> >>>> ==80209==<br>
> >>>> ==80209== Invalid read of size 8<br>
> >>>> ==80209== at 0x100A9E2D5: DMCountNonCyclicReferences (dm.c:500)<br>
> >>>> ==80209== by 0x100A8F70A: DMDestroy (dm.c:573)<br>
> >>>> ==80209== by 0x100001CBC: main (in ./ex65)<br>
> >>>> ==80209== Address 0x10398fd68 is 5,864 bytes inside a block of size 6,196 free'd<br>
> >>>> ==80209== at 0x10001595D: free (vg_replace_malloc.c:480)<br>
> >>>> ==80209== by 0x1000FE393: PetscFreeAlign (mal.c:72)<br>
> >>>> ==80209== by 0x100100D1E: PetscTrFreeDefault (mtr.c:315)<br>
> >>>> ==80209== by 0x100A91C5A: DMDestroy (dm.c:716)<br>
> >>>> ==80209== by 0x1010E2478: PCDestroy (precon.c:123)<br>
> >>>> ==80209== by 0x101221C3A: KSPDestroy (itfunc.c:986)<br>
> >>>> ==80209== by 0x1010BCBFC: PCDestroy_MG (mg.c:302)<br>
> >>>> ==80209== by 0x1010E23F7: PCDestroy (precon.c:122)<br>
> >>>> ==80209== by 0x101221C3A: KSPDestroy (itfunc.c:986)<br>
> >>>> ==80209== by 0x100001C4C: main (in ./ex65)<br>
> >>>> ==80209==<br>
> >>>> ==80209== Invalid read of size 8<br>
> >>>> ==80209== at 0x100A914C4: DMDestroy (dm.c:696)<br>
> >>>> ==80209== by 0x100001CBC: main (in ./ex65)<br>
> >>>> ==80209== Address 0x10398fd68 is 5,864 bytes inside a block of size 6,196 free'd<br>
> >>>> ==80209== at 0x10001595D: free (vg_replace_malloc.c:480)<br>
> >>>> ==80209== by 0x1000FE393: PetscFreeAlign (mal.c:72)<br>
> >>>> ==80209== by 0x100100D1E: PetscTrFreeDefault (mtr.c:315)<br>
> >>>> ==80209== by 0x100A91C5A: DMDestroy (dm.c:716)<br>
> >>>> ==80209== by 0x1010E2478: PCDestroy (precon.c:123)<br>
> >>>> ==80209== by 0x101221C3A: KSPDestroy (itfunc.c:986)<br>
> >>>> ==80209== by 0x1010BCBFC: PCDestroy_MG (mg.c:302)<br>
> >>>> ==80209== by 0x1010E23F7: PCDestroy (precon.c:122)<br>
> >>>> ==80209== by 0x101221C3A: KSPDestroy (itfunc.c:986)<br>
> >>>> ==80209== by 0x100001C4C: main (in ./ex65)<br>
> >>>> ==80209==<br>
> >>>> ==80209== Invalid read of size 4<br>
> >>>> ==80209== at 0x1002319B4: PetscCheckPointer (checkptr.c:106)<br>
> >>>> ==80209== by 0x100A8F5C6: DMDestroy (dm.c:570)<br>
> >>>> ==80209== by 0x100A9156F: DMDestroy (dm.c:699)<br>
> >>>> ==80209== by 0x100001CBC: main (in ./ex65)<br>
> >>>> ==80209== Address 0x10398ece0 is 1,632 bytes inside a block of size 6,196 free'd<br>
> >>>> ==80209== at 0x10001595D: free (vg_replace_malloc.c:480)<br>
> >>>> ==80209== by 0x1000FE393: PetscFreeAlign (mal.c:72)<br>
> >>>> ==80209== by 0x100100D1E: PetscTrFreeDefault (mtr.c:315)<br>
> >>>> ==80209== by 0x100A91C5A: DMDestroy (dm.c:716)<br>
> >>>> ==80209== by 0x1010E2478: PCDestroy (precon.c:123)<br>
> >>>> ==80209== by 0x101221C3A: KSPDestroy (itfunc.c:986)<br>
> >>>> ==80209== by 0x1010BCBFC: PCDestroy_MG (mg.c:302)<br>
> >>>> ==80209== by 0x1010E23F7: PCDestroy (precon.c:122)<br>
> >>>> ==80209== by 0x101221C3A: KSPDestroy (itfunc.c:986)<br>
> >>>> ==80209== by 0x100001C4C: main (in ./ex65)<br>
> >>>> ==80209==<br>
> >>>> ==80209== Invalid read of size 4<br>
> >>>> ==80209== at 0x100A8F630: DMDestroy (dm.c:570)<br>
> >>>> ==80209== by 0x100A9156F: DMDestroy (dm.c:699)<br>
> >>>> ==80209== by 0x100001CBC: main (in ./ex65)<br>
> >>>> ==80209== Address 0x10398ece0 is 1,632 bytes inside a block of size 6,196 free'd<br>
> >>>> ==80209== at 0x10001595D: free (vg_replace_malloc.c:480)<br>
> >>>> ==80209== by 0x1000FE393: PetscFreeAlign (mal.c:72)<br>
> >>>> ==80209== by 0x100100D1E: PetscTrFreeDefault (mtr.c:315)<br>
> >>>> ==80209== by 0x100A91C5A: DMDestroy (dm.c:716)<br>
> >>>> ==80209== by 0x1010E2478: PCDestroy (precon.c:123)<br>
> >>>> ==80209== by 0x101221C3A: KSPDestroy (itfunc.c:986)<br>
> >>>> ==80209== by 0x1010BCBFC: PCDestroy_MG (mg.c:302)<br>
> >>>> ==80209== by 0x1010E23F7: PCDestroy (precon.c:122)<br>
> >>>> ==80209== by 0x101221C3A: KSPDestroy (itfunc.c:986)<br>
> >>>> ==80209== by 0x100001C4C: main (in ./ex65)<br>
> >>>> ==80209==<br>
> >>>> ==80209== Invalid read of size 4<br>
> >>>> ==80209== at 0x100A8F641: DMDestroy (dm.c:570)<br>
> >>>> ==80209== by 0x100A9156F: DMDestroy (dm.c:699)<br>
> >>>> ==80209== by 0x100001CBC: main (in ./ex65)<br>
> >>>> ==80209== Address 0x10398ece0 is 1,632 bytes inside a block of size 6,196 free'd<br>
> >>>> ==80209== at 0x10001595D: free (vg_replace_malloc.c:480)<br>
> >>>> ==80209== by 0x1000FE393: PetscFreeAlign (mal.c:72)<br>
> >>>> ==80209== by 0x100100D1E: PetscTrFreeDefault (mtr.c:315)<br>
> >>>> ==80209== by 0x100A91C5A: DMDestroy (dm.c:716)<br>
> >>>> ==80209== by 0x1010E2478: PCDestroy (precon.c:123)<br>
> >>>> ==80209== by 0x101221C3A: KSPDestroy (itfunc.c:986)<br>
> >>>> ==80209== by 0x1010BCBFC: PCDestroy_MG (mg.c:302)<br>
> >>>> ==80209== by 0x1010E23F7: PCDestroy (precon.c:122)<br>
> >>>> ==80209== by 0x101221C3A: KSPDestroy (itfunc.c:986)<br>
> >>>> ==80209== by 0x100001C4C: main (in ./ex65)<br>
> >>>> ==80209==<br>
> >>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>
> >>>> [0]PETSC ERROR: Invalid argument<br>
> >>>> [0]PETSC ERROR: Wrong type of object: Parameter # 1<br>
> >>>> [0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br>
> >>>> [0]PETSC ERROR: Petsc Development GIT revision: pre-tsfc-829-g3974c78 GIT Date: 2016-03-11 17:51:48 -0600<br>
> >>>> [0]PETSC ERROR: ./ex65 on a arch-basic named Barrys-MacBook-Pro.local by barrysmith Sun Mar 13 16:13:10 2016<br>
> >>>> [0]PETSC ERROR: Configure options --with-mpi-dir=/Users/barrysmith/PetscLibraries PETSC_ARCH=arch-basic<br>
> >>>> [0]PETSC ERROR: #1 DMDestroy() line 570 in /Users/barrysmith/Src/petsc/src/dm/interface/dm.c<br>
> >>>> [0]PETSC ERROR: #2 DMDestroy() line 699 in /Users/barrysmith/Src/petsc/src/dm/interface/dm.c<br>
> >>>> [0]PETSC ERROR: #3 main() line 67 in /Users/barrysmith/Src/petsc/src/ksp/ksp/examples/tutorials/ex65.c<br>
> >>>> [0]PETSC ERROR: PETSc Option Table entries:<br>
> >>>> [0]PETSC ERROR: -malloc_test<br>
> >>>> [0]PETSC ERROR: -pc_mg_levels 2<br>
> >>>> [0]PETSC ERROR: -pc_type mg<br>
> >>>> [0]PETSC ERROR: ----------------End of Error Message -------send entire error message to petsc-maint@mcs.anl.gov----------<br>
> >>>><br>
> >>>> The code is in the branch barry/add-dmshellcreaterestriction src/ksp/ksp/examples/tutorials/ex65.c which creates a DMSHELL that just uses an inner DMDA1 to create the objects. The code is virtually identical to ex25.c which just uses the DMDA1d directly but does not crash. It seems to me that having the DM objects be shells instead of DMDA should make absolutely no difference in your logic for tracking dm->coarseMesh etc but somehow something is fishy!!!! I could have a mistake in my example code but I do not think so.<br>
> >>>><br>
> >>>> Could you please take a look at the problem, feel free to add fixes directly to the branch.<br>
> >>>><br>
> >>>> Thanks<br>
> >>>><br>
> >>>> Barry<br>
> >>>><br>
> >>>><br>
> >><br>
><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</div></div>