[petsc-dev] GASM
Dmitry Karpeev
karpeev at mcs.anl.gov
Thu Mar 29 07:44:30 CDT 2012
Mark,
I pushed a fix for that specific bug, but from my reading of the GAMG code
this won't solve the real problem:
PCSetUp_GAMG() reuses pc as a subpc in eksp, protecting subpc with an
incref. That prevents the subpc
from being destroyed when eksp is destroyed, but the somewhat unfortunate
logic of KSPDestroy(eksp) also calls
KSPReset(eksp), which cascades to PCReset(subpc), blowing out its internals
-- PCReset() pays no mind to the refcount.
The bug was that GASM's guts weren't being destroyed very cleanly, allowing
a later read into a freed array. After this fix that should cause a more
obvious SEGV, but the real problem is that a pc that is supposed remain
valid is being reset.
I'm not sure what the best way to address the general problem should be:
should KSPDestroy() has to be redesigned? should PCReset() be a noop for a
positive refcount? or
should this use of a pc in two places be deemed illegal?
Dmitry.
On Wed, Mar 28, 2012 at 8:24 PM, Mark F. Adams <mark.adams at columbia.edu>wrote:
> Dmitry,
>
> I seem to have a problem with GASM. I have code like this to get an eigen
> estimate:
>
> ierr = PetscObjectReference( (PetscObject)subpc );
> CHKERRQ(ierr);
> ierr = KSPSetPC( eksp, subpc ); CHKERRQ( ierr );
> < snip >
> ierr = KSPDestroy( &eksp ); CHKERRQ(ierr);
>
> This 'subpc' has GASM and is used in the real solver. And I get this
> error when the real solver gets setup, after the above code:
>
> ==2070== Invalid read of size 4
> ==2070== at 0x9F034C: PCSetUp_GASM (gasm.c:275)
> ==2070== by 0xA9620A: PCSetUp (precon.c:832)
> ==2070== by 0xB20EE5: KSPSetUp (itfunc.c:266)
> ==2070== by 0xA74975: PCSetUp_MG (mg.c:647)
> ==2070== by 0xA11A1B: PCSetUp_GAMG (gamg.c:803)
> ==2070== by 0xA9620A: PCSetUp (precon.c:832)
> ==2070== by 0xB20EE5: KSPSetUp (itfunc.c:266)
> ==2070== by 0xB22178: KSPSolve (itfunc.c:390)
> ==2070== by 0x95B8C0: kspsolve_ (itfuncf.c:219)
> ==2070== by 0x404E: MAIN__ (in ./ex54f)
> ==2070== by 0x4A6B: main (in ./ex54f)
> ==2070== Address 0x29e6ac0 is 32 bytes inside a block of size 108 free'd
> ==2070== at 0x19317D8: free (vg_replace_malloc.c:366)
> ==2070== by 0x865AD: PetscFreeAlign (mal.c:75)
> ==2070== by 0x9FAB7F: PCGASMDestroySubdomains (gasm.c:1309)
> ==2070== by 0x9F3AED: PCReset_GASM (gasm.c:536)
> ==2070== by 0xA8B322: PCReset (precon.c:80)
> ==2070== by 0xB270D0: KSPReset (itfunc.c:723)
> ==2070== by 0xB279BA: KSPDestroy (itfunc.c:760)
> ==2070== by 0xA1164D: PCSetUp_GAMG (gamg.c:779)
> ==2070== by 0xA9620A: PCSetUp (precon.c:832)
> ==2070== by 0xB20EE5: KSPSetUp (itfunc.c:266)
> ==2070== by 0xB22178: KSPSolve (itfunc.c:390)
> ==2070== by 0x95B8C0: kspsolve_ (itfuncf.c:219)
> ==2070== by 0x404E: MAIN__ (in ./ex54f)
> ==2070== by 0x4A6B: main (in ./ex54f)
> ==2070==
> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Null argument, when expecting valid pointer!
> [0]PETSC ERROR: Null Object: Parameter # 1!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Development HG revision:
> e4ca525c39fd35cca849d095040a2de0488f347b HG Date: Fri Mar 23 22:45:44 2012
> -0500
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: ./ex54f on a arch-maco named madams-macbk-3.home by
> markadams Wed Mar 28 21:20:16 2012
> [0]PETSC ERROR: Libraries linked from
> /Users/markadams/Codes/petsc-dev/arch-macosx-gnu-O/lib
> [0]PETSC ERROR: Configure run at Sat Mar 24 09:45:16 2012
> [0]PETSC ERROR: Configure options CXX="/sw/lib/gcc4.6/bin/g++-4
> -malign-double" CC="/sw/lib/gcc4.6/bin/gcc-4 -malign-double"
> FC=/sw/lib/gcc4.6/bin/gfortran --download-parmetis=1 --download-metis=1
> --download-hdf5=1 --download-hypre=1 --download-ml=1 --download-triangle=1
> --with-x=0 --with-debugging=1 --download-mpich=1
> PETSC_ARCH=arch-macosx-gnu-O --download-f-blas-lapack=1
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: ISSort() line 761 in
> /Users/markadams/Codes/petsc-dev/src/vec/is/interface/index.c
> [0]PETSC ERROR: PCSetUp_GASM() line 275 in
> /Users/markadams/Codes/petsc-dev/src/ksp/pc/impls/gasm/gasm.c
>
> THis error goes away if I comment out the above call to KSPDestroy. So it
> looks like KSPDestroy is destroying GASM (or part of it) even though I take
> a reference to it, to prevent this destruction.
>
> Any ideas?
>
> Mark
>
>
> Mark,
>
> You should pass your subdomain ISs through the first IS list argument, and
> leave the second as PETSC_NULL.
> In that case GASM assumes that the user passed in a nonoverlapping
> decomposition through the first list.
> This is the behavior that GASM inherited from ASM and it isn't very well
> (or at all) documented in either.
> Let me know if that doesn't fix the problem.
>
> Note also that you don't have to explicitly set overlap to 0: if the user
> sets not overlap PETSc won't try to increase it,
> and if you set overlap to 0 it will not try to adjust the subdomains to
> make sure the overlap is 0. Both ASM and GASM simply assume that the user
> does the right thing and passes in a nonoverlapping decomposition.
>
> Incidentally, it may be time to change this API. I'm trying to debug a
> bad convergence problem with GASM -- inexplicably, it converges slower than
> ASM with a single process and multiple subdomains -- so now is a good time
> to mess with the code, while I still remember what's what. Another thing
> I've considered doing is reordering the subdomains internally (on demand
> only), to ensure a deadlock-free ordering of the subcomms.
>
> Dmitry.
>
> On Tue, Mar 27, 2012 at 6:28 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>
>> This is why petsc-dev is better than a random list of email addresses.
>> No one on your random list uses GASM
>>
>>
>>
>> Barry
>>
>> On Mar 27, 2012, at 4:49 PM, Mark F. Adams wrote:
>>
>> > I'm trying to make general non-overlapping domains in GASM. I create a
>> vector of IS, each with the global indices of the equations in each block.
>> >
>> > I use code that looks like:
>> >
>> > ierr = PCSetType( subpc, PCGASM ); CHKERRQ(ierr);
>> > ierr = PCGASMSetLocalSubdomains( subpc, sz, PETSC_NULL, is );
>> CHKERRQ(ierr);
>> > ierr = PetscFree( is ); CHKERRQ(ierr);
>> > ierr = PCGASMSetOverlap( subpc, 0 ); CHKERRQ(ierr);
>> >
>> > In parallel, I get errors like this:
>> >
>> > [1]PETSC ERROR: gis_local contains -85 indices outside of gis!
>> > <snip>
>> > [1]PETSC ERROR: PCSetUp_GASM() line 332 in
>> /Users/markadams/Codes/petsc-dev/src/ksp/pc/impls/gasm/gasm.c
>> >
>> > Anyone have any ideas on what I'm doing wrong here? I've checked the
>> input and I think it is fine,
>> >
>> > Mark
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20120329/be40e9a9/attachment.html>
More information about the petsc-dev
mailing list