Mark,<div><br></div><div>I pushed a fix for that specific bug, but from my reading of the GAMG code this won't solve the real problem:</div><div>PCSetUp_GAMG() reuses pc as a subpc in eksp, protecting subpc with an incref. That prevents the subpc</div>
<div>from being destroyed when eksp is destroyed, but the somewhat unfortunate logic of KSPDestroy(eksp) also calls </div><div>KSPReset(eksp), which cascades to PCReset(subpc), blowing out its internals -- PCReset() pays no mind to the refcount.</div>
<div>The bug was that GASM's guts weren't being destroyed very cleanly, allowing a later read into a freed array. After this fix that should cause a more obvious SEGV, but the real problem is that a pc that is supposed remain valid is being reset.</div>
<div><br></div><div>I'm not sure what the best way to address the general problem should be:</div><div>should KSPDestroy() has to be redesigned? should PCReset() be a noop for a positive refcount? or </div><div>should this use of a pc in two places be deemed illegal?</div>
<div><br></div><div>Dmitry.</div><div><br></div><div><br><br><div class="gmail_quote">On Wed, Mar 28, 2012 at 8:24 PM, Mark F. Adams <span dir="ltr"><<a href="mailto:mark.adams@columbia.edu">mark.adams@columbia.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Dmitry,<div><br></div><div>I seem to have a problem with GASM. I have code like this to get an eigen estimate:</div>
<div><br></div><div><div> ierr = PetscObjectReference( (PetscObject)subpc ); CHKERRQ(ierr);</div><div> ierr = KSPSetPC( eksp, subpc ); CHKERRQ( ierr );</div><div> < snip ></div><div> ierr = KSPDestroy( &eksp ); CHKERRQ(ierr);</div>
</div><div><br></div><div>This 'subpc' has GASM and is used in the real solver. And I get this error when the real solver gets setup, after the above code:</div><div><br></div><div><div><div><div>==2070== Invalid read of size 4</div>
<div>==2070== at 0x9F034C: PCSetUp_GASM (gasm.c:275)</div><div>==2070== by 0xA9620A: PCSetUp (precon.c:832)</div><div>==2070== by 0xB20EE5: KSPSetUp (itfunc.c:266)</div><div>==2070== by 0xA74975: PCSetUp_MG (mg.c:647)</div>
<div>==2070== by 0xA11A1B: PCSetUp_GAMG (gamg.c:803)</div><div>==2070== by 0xA9620A: PCSetUp (precon.c:832)</div><div>==2070== by 0xB20EE5: KSPSetUp (itfunc.c:266)</div><div>==2070== by 0xB22178: KSPSolve (itfunc.c:390)</div>
<div>==2070== by 0x95B8C0: kspsolve_ (itfuncf.c:219)</div><div>==2070== by 0x404E: MAIN__ (in ./ex54f)</div><div>==2070== by 0x4A6B: main (in ./ex54f)</div><div>==2070== Address 0x29e6ac0 is 32 bytes inside a block of size 108 free'd</div>
<div>==2070== at 0x19317D8: free (vg_replace_malloc.c:366)</div><div>==2070== by 0x865AD: PetscFreeAlign (mal.c:75)</div><div>==2070== by 0x9FAB7F: PCGASMDestroySubdomains (gasm.c:1309)</div><div>==2070== by 0x9F3AED: PCReset_GASM (gasm.c:536)</div>
<div>==2070== by 0xA8B322: PCReset (precon.c:80)</div><div>==2070== by 0xB270D0: KSPReset (itfunc.c:723)</div><div>==2070== by 0xB279BA: KSPDestroy (itfunc.c:760)</div><div>==2070== by 0xA1164D: PCSetUp_GAMG (gamg.c:779)</div>
<div>==2070== by 0xA9620A: PCSetUp (precon.c:832)</div><div>==2070== by 0xB20EE5: KSPSetUp (itfunc.c:266)</div><div>==2070== by 0xB22178: KSPSolve (itfunc.c:390)</div><div>==2070== by 0x95B8C0: kspsolve_ (itfuncf.c:219)</div>
<div>==2070== by 0x404E: MAIN__ (in ./ex54f)</div><div>==2070== by 0x4A6B: main (in ./ex54f)</div><div>==2070== </div><div>[0]PETSC ERROR: --------------------- Error Message ------------------------------------</div>
<div>[0]PETSC ERROR: Null argument, when expecting valid pointer!</div><div>[0]PETSC ERROR: Null Object: Parameter # 1!</div><div>[0]PETSC ERROR: ------------------------------------------------------------------------</div>
<div>[0]PETSC ERROR: Petsc Development HG revision: e4ca525c39fd35cca849d095040a2de0488f347b HG Date: Fri Mar 23 22:45:44 2012 -0500</div><div>[0]PETSC ERROR: See docs/changes/index.html for recent updates.</div><div>[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.</div>
<div>[0]PETSC ERROR: See docs/index.html for manual pages.</div><div>[0]PETSC ERROR: ------------------------------------------------------------------------</div><div>[0]PETSC ERROR: ./ex54f on a arch-maco named madams-macbk-3.home by markadams Wed Mar 28 21:20:16 2012</div>
<div>[0]PETSC ERROR: Libraries linked from /Users/markadams/Codes/petsc-dev/arch-macosx-gnu-O/lib</div><div>[0]PETSC ERROR: Configure run at Sat Mar 24 09:45:16 2012</div><div>[0]PETSC ERROR: Configure options CXX="/sw/lib/gcc4.6/bin/g++-4 -malign-double" CC="/sw/lib/gcc4.6/bin/gcc-4 -malign-double" FC=/sw/lib/gcc4.6/bin/gfortran --download-parmetis=1 --download-metis=1 --download-hdf5=1 --download-hypre=1 --download-ml=1 --download-triangle=1 --with-x=0 --with-debugging=1 --download-mpich=1 PETSC_ARCH=arch-macosx-gnu-O --download-f-blas-lapack=1</div>
<div>[0]PETSC ERROR: ------------------------------------------------------------------------</div><div>[0]PETSC ERROR: ISSort() line 761 in /Users/markadams/Codes/petsc-dev/src/vec/is/interface/index.c</div><div>[0]PETSC ERROR: PCSetUp_GASM() line 275 in /Users/markadams/Codes/petsc-dev/src/ksp/pc/impls/gasm/gasm.c</div>
</div><div><br></div><div>THis error goes away if I comment out the above call to KSPDestroy. So it looks like KSPDestroy is destroying GASM (or part of it) even though I take a reference to it, to prevent this destruction.</div>
<div><br></div><div>Any ideas?</div><span class="HOEnZb"><font color="#888888"><div><br></div><div>Mark</div></font></span><div><div class="h5"><div><br></div><br><blockquote type="cite">Mark,<div><br></div><div>You should pass your subdomain ISs through the first IS list argument, and leave the second as PETSC_NULL.</div>
<div>In that case GASM assumes that the user passed in a nonoverlapping decomposition through the first list.</div>
<div>This is the behavior that GASM inherited from ASM and it isn't very well (or at all) documented in either.</div><div>Let me know if that doesn't fix the problem.</div><div><br></div><div>Note also that you don't have to explicitly set overlap to 0: if the user sets not overlap PETSc won't try to increase it,</div>
<div>and if you set overlap to 0 it will not try to adjust the subdomains to make sure the overlap is 0. Both ASM and GASM simply assume that the user does the right thing and passes in a nonoverlapping decomposition.</div>
<div><br></div><div>Incidentally, it may be time to change this API. I'm trying to debug a bad convergence problem with GASM -- inexplicably, it converges slower than ASM with a single process and multiple subdomains -- so now is a good time to mess with the code, while I still remember what's what. Another thing I've considered doing is reordering the subdomains internally (on demand only), to ensure a deadlock-free ordering of the subcomms.</div>
<div><br></div><div>Dmitry.</div><div><br><div class="gmail_quote">On Tue, Mar 27, 2012 at 6:28 PM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
This is why petsc-dev is better than a random list of email addresses. No one on your random list uses GASM<br>
<br>
<br>
<br>
Barry<br>
<br>
On Mar 27, 2012, at 4:49 PM, Mark F. Adams wrote:<br>
<br>
> I'm trying to make general non-overlapping domains in GASM. I create a vector of IS, each with the global indices of the equations in each block.<br>
><br>
> I use code that looks like:<br>
><br>
> ierr = PCSetType( subpc, PCGASM ); CHKERRQ(ierr);<br>
> ierr = PCGASMSetLocalSubdomains( subpc, sz, PETSC_NULL, is ); CHKERRQ(ierr);<br>
> ierr = PetscFree( is ); CHKERRQ(ierr);<br>
> ierr = PCGASMSetOverlap( subpc, 0 ); CHKERRQ(ierr);<br>
><br>
> In parallel, I get errors like this:<br>
><br>
> [1]PETSC ERROR: gis_local contains -85 indices outside of gis!<br>
> <snip><br>
> [1]PETSC ERROR: PCSetUp_GASM() line 332 in /Users/markadams/Codes/petsc-dev/src/ksp/pc/impls/gasm/gasm.c<br>
><br>
> Anyone have any ideas on what I'm doing wrong here? I've checked the input and I think it is fine,<br>
><br>
> Mark<br>
<br>
</blockquote></div><br></div>
</blockquote></div></div></div><br></div></div></blockquote></div><br></div>