[petsc-dev] ASM for each field solve on GPUs

[Stefano Zampini]

You should swap fieldsplit and ASM

-pc_type fieldsplit
-fieldsplit_0_pc_type asm
….


> On Dec 31, 2020, at 10:19 PM, Mark Adams <mfadams at lbl.gov> wrote:
> 
> PCFieldSplitSetDefaults is called from PCSetUpOnBlocks and there is only one block. So it is called from a subpc, which does not have a DM and it's too late anyway.
> 
> #14 main (argc=<optimized out>, argv=<optimized out>) at /gpfs/alpine/csc314/scratch/adams/petsc/src/ts/utils/dmplexlandau/tutorials/ex2.c:774 (at 0x0000000010002a28)
> #13 TSSolve (ts=0x83638480, u=0x8338de50) at /gpfs/alpine/csc314/scratch/adams/petsc/src/ts/interface/ts.c:4168 (at 0x00002000015e60e4)
> #12 TSStep (ts=0x83638480) at /gpfs/alpine/csc314/scratch/adams/petsc/src/ts/interface/ts.c:3771 (at 0x00002000015e416c)
> #11 TSStep_ARKIMEX (ts=0x83638480) at /gpfs/alpine/csc314/scratch/adams/petsc/src/ts/impls/arkimex/arkimex.c:845 (at 0x000020000165e97c)
> #10 SNESSolve (snes=0x83639d20, b=0x0, x=0x8338f320) at /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/interface/snes.c:4563 (at 0x00002000014f2b98)
> #9 SNESSolve_NEWTONLS (snes=0x83639d20) at /gpfs/alpine/csc314/scratch/adams/petsc/src/snes/impls/ls/ls.c:225 (at 0x0000200001553d34)
> #8 KSPSolve (ksp=0x8363a610, b=0x8363db00, x=0x83642350) at /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/ksp/interface/itfunc.c:963 (at 0x000020000138987c)
> #7 KSPSolve_Private (ksp=0x8363a610, b=0x8363db00, x=0x83642350) at /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/ksp/interface/itfunc.c:727 (at 0x0000200001386d70)
> #6 KSPSetUpOnBlocks (ksp=0x8363a610) at /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/ksp/interface/itfunc.c:214 (at 0x0000200001381a6c)
> #5 PCSetUpOnBlocks (pc=0x8338afe0) at /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/pc/interface/precon.c:1046 (at 0x0000200001333aa0)
> #4 PCSetUpOnBlocks_ASM (pc=0x8338afe0) at /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/pc/impls/asm/asm.c:437 (at 0x00002000012cb87c)
> #3 KSPSetUp (ksp=0x836486c0) at /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/ksp/interface/itfunc.c:406 (at 0x0000200001383258)
> #2 PCSetUp (pc=0x83645200) at /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/pc/interface/precon.c:1015 (at 0x00002000013335f0)
> #1 PCSetUp_FieldSplit (pc=0x83645200) at /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:614 (at 0x00002000012a27f4)
> #0 PCFieldSplitSetDefaults (pc=0x83645200) at /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c:398 (at 0x000020000129f854)
> 
> On Thu, Dec 31, 2020 at 1:56 PM Matthew Knepley <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
> On Thu, Dec 31, 2020 at 1:50 PM Mark Adams <mfadams at lbl.gov <mailto:mfadams at lbl.gov>> wrote:
> On Thu, Dec 31, 2020 at 1:12 PM Matthew Knepley <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
> On Thu, Dec 31, 2020 at 12:46 PM Mark Adams <mfadams at lbl.gov <mailto:mfadams at lbl.gov>> wrote:
> Still missing something:
> 
> You can get in the debugger in PCFieldSplitSetDefaults(), but the most likely thing is that pc->dm == NULL here,
> 
> the setuponblocks PC does have a DM and it is a cusparse mattype.
> 
> Then go in the debugger and see why that option is not being used.
> 
>    Matt
>  
>  
> so the PC is not getting the DM.
> This does make some sense as you are doing ASM, which is ripping up the domain, so you might not want the same DM on the subproblem. I
> am not sure what should be done here.
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20201231/82544522/attachment.html>