[petsc-dev] making DA more light weight
Barry Smith
bsmith at mcs.anl.gov
Thu May 15 22:09:06 CDT 2014
Ok, added #define VecScatterOptimizedBS(mbs) (2 <= mbs && mbs <= 12) for Jed’s branch and now get
~/Src/petsc/src/dm/examples/tests jed/dmda-memusage $ petscmpiexec -n 2 ./ex20
DMDACreate3d : 86.041 MB + 86.041 MB
DMCreateGlobalVector: 169.980 MB + 83.939 MB
DMCreateGlobalVector: 253.870 MB + 83.890 MB
this is without the optimization of the ISLocalToGlobalMapping.
Still lots more to done, working away
Barry
On May 15, 2014, at 8:13 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
> Damn, damn, damn
>
> define VecScatterOptimizedBS(mbs) ((2 <= mbs && mbs <= 8) || mbs == 12)
>
> In the debugger I was always picking a block size that happened to handled nicely while you picked one that is not (but could easily be added). I figured you were in a different PETSc universe.
>
> I changed the bs to 8 and added a print inside the DMDACreate3d() at the appropriate place and got
>
> ~/Src/petsc/src/dm/examples/tests jed/dmda-memusage $ petscmpiexec -n 2 ./ex20
> DMDACreate3d before local to global : 43.442 MB
> DMDACreate3d after local to global : 77.562 MB
> DMDACreate3d : 77.578 MB + 77.578 MB
> DMCreateGlobalVector: 144.740 MB + 67.162 MB
> DMCreateGlobalVector: 211.857 MB + 67.117 MB
>
> Note that out of the 77.6 needed for the DMDA 34 is for the local to global. If that is 34 is divided by 8 then the total needed by the DMDA is 43.4 + 4.x < 50 which is less than ONE vector.
>
> Barry
>
>
>
> On May 15, 2014, at 8:00 PM, Jed Brown <jed at jedbrown.org> wrote:
>
>> Barry Smith <bsmith at mcs.anl.gov> writes:
>>
>>> Damn it Jed, I already told you the sequential case does NOT have the optimization for bs > 1. Run in the debugger on two processes and put a break point in ISGetIndices() and you will see that it is not used by the VecScatterCreate() inside the DMDACreate …
>>>
>>>> $ mpirun.hydra -n 2 ./ex20 -malloc_log -malloc_log_threshold 1e6
>>>>
>>>>
>>>> If you run as above in a debugger with a break point at ISGetIndices,
>>>> the first break is at
>>>>
>>>> $ mpirun.hydra -n 1 gdb -ex 'b ISGetIndices' -ex r --args ./ex20 : -n 1 ./ex20
>>>
>>> You are not running as above since you went from 2 to 1 process where completely different code is used in creating the scatter!
>>
>> Barry, this really is a two-process job, despite the fact that I'm only
>> attaching a debugger to rank 0. Same with the debugger traces below.
>> It's standard mpiexec syntax.
>>
>> mpiexec args executable pgmargs [ : args executable pgmargs ... ]
>>
>>>> [0] 3 12713984 DMSetUp_DA_3D()
>>>> [0] 1 4259840 ISBlockSetIndices_Block()
>>>> [0] 4 169738240 ISGetIndices_Block()
>>>
>>> This number above is NOT in the VecScatterCreate() it is in the creation of the local to global mapping
>>>
>>>> [0] 1 41943040 ISGetIndices_Stride()
>>>> [0] 1 4259840 ISLocalToGlobalMappingBlock()
>>>> [0] 2 167772160 VecCreate_MPI_Private()
>>>> [0] 10 339476480 VecScatterCreate_PtoS()
>>> ^^^^ I admit this number is larger than I would like to see but it is not caused by ISGetIndices
>>>
>>> Barry
>>>
>>> Likely we can just remove the sequential case from the checks in the VecScatter and always do the “parallel” and thus remove the unoptimized code that you keep tripping over.
>>>
>>>
>>> On May 15, 2014, at 5:38 PM, Jed Brown <jed at jedbrown.org> wrote:
>>>
>>>> Barry Smith <bsmith at mcs.anl.gov> writes:
>>>>
>>>>>> Note that ISGetIndices is still called in the parallel case.
>>>>>
>>>>> When bs is set > 1 it is not called in the VecScatterCreate()
>>>>> inside the DMDACreate 2d/3d ! Only ISBlockGetIndices() is
>>>>> called. The memory usage of VecScatterCreate() is O(dof*number of
>>>>> ghost points) + O(vector length/dof) in this case
>>>>>
>>>>> It is called MULTIPLY times in the ISLocalToGlobalMappingCreateIS()
>>>>> inside the DMDACreate 2d/3d thus currently this causes small integer
>>>>> * vector length usage. As I keep telling you this is the problem
>>>>> area for dof > 1. For dof == 1 I can live with 2.5 *sizeof (vector)
>>>>>
>>>>> You need a better test code to run with. Can you use
>>>>> ksp/ksp/examples/tests/ex42.c ? Or snes/examples/tutorials/ex19.c
>>>>> in parallel?
>>>>
>>>> I added a simple test for this purpose, src/dm/examples/tests/ex20.c
>>>> in 'jed/dmda-memusage'.
>>>>
>>>> $ mpirun.hydra -n 2 ./ex20 -malloc_log -malloc_log_threshold 1e6
>>>> DMDACreate3d : 225.866 MB + 225.866 MB
>>>> DMCreateGlobalVector: 309.666 MB + 83.800 MB
>>>> DMCreateGlobalVector: 393.552 MB + 83.886 MB
>>>> [0] Maximum memory PetscMalloc()ed 386401520 maximum size of entire process 309821440
>>>> [0] Memory usage sorted by function
>>>> [0] 3 12713984 DMSetUp_DA_3D()
>>>> [0] 1 4259840 ISBlockSetIndices_Block()
>>>> [0] 4 169738240 ISGetIndices_Block()
>>>> [0] 1 41943040 ISGetIndices_Stride()
>>>> [0] 1 4259840 ISLocalToGlobalMappingBlock()
>>>> [0] 2 167772160 VecCreate_MPI_Private()
>>>> [0] 10 339476480 VecScatterCreate_PtoS()
>>>>
>>>> If you run as above in a debugger with a break point at ISGetIndices,
>>>> the first break is at
>>>>
>>>> $ mpirun.hydra -n 1 gdb -ex 'b ISGetIndices' -ex r --args ./ex20 : -n 1 ./ex20
>>>> GNU gdb (GDB) 7.7.1
>>>> Copyright (C) 2014 Free Software Foundation, Inc.
>>>> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
>>>> This is free software: you are free to change and redistribute it.
>>>> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
>>>> and "show warranty" for details.
>>>> This GDB was configured as "x86_64-unknown-linux-gnu".
>>>> Type "show configuration" for configuration details.
>>>> For bug reporting instructions, please see:
>>>> <http://www.gnu.org/software/gdb/bugs/>.
>>>> Find the GDB manual and other documentation resources online at:
>>>> <http://www.gnu.org/software/gdb/documentation/>.
>>>> For help, type "help".
>>>> Type "apropos word" to search for commands related to "word"...
>>>> No symbol table is loaded. Use the "file" command.
>>>> Breakpoint 1 (err.c:343) pending.
>>>> Reading symbols from ./ex20...done.
>>>> Function "ISGetIndices" not defined.
>>>> Breakpoint 2 (ISGetIndices) pending.
>>>> Starting program: /home/jed/petsc/src/dm/examples/tests/ex20
>>>> warning: Could not load shared library symbols for linux-vdso.so.1.
>>>> Do you need "set solib-search-path" or "set sysroot"?
>>>> [Thread debugging using libthread_db enabled]
>>>> Using host libthread_db library "/usr/lib/libthread_db.so.1".
>>>>
>>>> Breakpoint 2, ISGetIndices (is=0x506840, ptr=0x7fffffffd328) at /home/jed/petsc/src/vec/is/is/interface/index.c:379
>>>> 379 PetscFunctionBegin;
>>>> (gdb) bt
>>>> #0 ISGetIndices (is=0x506840, ptr=0x7fffffffd328) at /home/jed/petsc/src/vec/is/is/interface/index.c:379
>>>> #1 0x00007ffff6a60cb9 in VecScatterCreate (xin=0x4fbfb0, ix=0x506840, yin=0x4f8d10, iy=0x503210, newctx=0x7fffffffd888) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1579
>>>> #2 0x00007ffff7129d9b in DMSetUp_DA_3D (da=0x4e3280) at /home/jed/petsc/src/dm/impls/da/da3.c:448
>>>> #3 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4e3280) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>> #4 0x00007ffff736fd9f in DMSetUp (dm=0x4e3280) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>> #5 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd58) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>> #6 0x0000000000400c96 in main (argc=1, argv=0x7fffffffde58) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>
>>>> That is this code, which dates from 1995 according to git annotate.
>>>>
>>>> /* general case */
>>>> {
>>>> PetscInt nx,ny;
>>>> const PetscInt *idx,*idy;
>>>> ierr = ISGetLocalSize(ix,&nx);CHKERRQ(ierr);
>>>> ierr = ISGetIndices(ix,&idx);CHKERRQ(ierr);
>>>> ierr = ISGetLocalSize(iy,&ny);CHKERRQ(ierr);
>>>> ierr = ISGetIndices(iy,&idy);CHKERRQ(ierr);
>>>> if (nx != ny) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_ARG_SIZ,"Local scatter sizes don't match");
>>>> ierr = VecScatterCreate_StoP(nx,idx,ny,idy,xin,yin,1,ctx);CHKERRQ(ierr);
>>>> ierr = ISRestoreIndices(ix,&idx);CHKERRQ(ierr);
>>>> ierr = ISRestoreIndices(iy,&idy);CHKERRQ(ierr);
>>>> ierr = PetscInfo(xin,"General case: Seq to MPI\n");CHKERRQ(ierr);
>>>> goto functionend;
>>>> }
>>>>
>>>>
>>>> Here are all the large allocations, found by stepping through printing
>>>> the stack trace for each case.
>>>>
>>>> $ mpirun.hydra -n 1 gdb -ex 'b mtr.c:228' -ex r --args ./ex20 -malloc_log -malloc_log_threshold 1e7 : -n 1 ./ex20
>>>> [...]
>>>>
>>>> (gdb) bt
>>>> #0 PetscTrMallocDefault (a=41943040, lineno=47, function=0x7ffff7afd5d4 <.L__func__.ISGetIndices_Block> "ISGetIndices_Block", filename=0x7ffff7afd53b <.L.str> "/home/jed/petsc/src/vec/is/is/impls/block/block.c", result=0x7fffffffcf88) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>> #1 0x00007ffff691fad2 in ISGetIndices_Block (in=0x4cbe90, idx=0x7fffffffd2f8) at /home/jed/petsc/src/vec/is/is/impls/block/block.c:47
>>>> #2 0x00007ffff6936468 in ISGetIndices (is=0x4cbe90, ptr=0x7fffffffd2f8) at /home/jed/petsc/src/vec/is/is/interface/index.c:382
>>>> #3 0x00007ffff6a60cb9 in VecScatterCreate (xin=0x4c1600, ix=0x4cbe90, yin=0x4be360, iy=0x4c8860, newctx=0x7fffffffd858) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1579
>>>> #4 0x00007ffff7129d9b in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:448
>>>> #5 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>> #6 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>> #7 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>> #8 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>
>>>>
>>>> #0 PetscTrMallocDefault (a=41943040, lineno=160, function=0x7ffff7afd32b <.L__func__.ISGetIndices_Stride> "ISGetIndices_Stride", filename=0x7ffff7afd279 <.L.str> "/home/jed/petsc/src/vec/is/is/impls/stride/stride.c", result=0x7fffffffd2f0) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>> #1 0x00007ffff691b1e9 in ISGetIndices_Stride (in=0x4c8860, idx=0x7fffffffd2f0) at /home/jed/petsc/src/vec/is/is/impls/stride/stride.c:160
>>>> #2 0x00007ffff6936468 in ISGetIndices (is=0x4c8860, ptr=0x7fffffffd2f0) at /home/jed/petsc/src/vec/is/is/interface/index.c:382
>>>> #3 0x00007ffff6a60da7 in VecScatterCreate (xin=0x4c1600, ix=0x4cbe90, yin=0x4be360, iy=0x4c8860, newctx=0x7fffffffd858) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1581
>>>> #4 0x00007ffff7129d9b in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:448
>>>> #5 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>> #6 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>> #7 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>> #8 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>
>>>>
>>>> #0 PetscTrMallocDefault (a=41943040, lineno=1745, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x7fffffffcee0) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>> #1 0x00007ffff6a9129f in VecScatterCreate_PtoS (nx=10485760, inidx=0x7fffe9086650, ny=10485760, inidy=0x7fffeb887650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1745
>>>> #2 0x00007ffff6a94c2d in VecScatterCreate_StoP (nx=10485760, inidx=0x7fffeb887650, ny=10485760, inidy=0x7fffe9086650, xin=0x4c1600, yin=0x4be360, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:2197
>>>> #3 0x00007ffff6a60ea0 in VecScatterCreate (xin=0x4c1600, ix=0x4cbe90, yin=0x4be360, iy=0x4c8860, newctx=0x7fffffffd858) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1583
>>>> #4 0x00007ffff7129d9b in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:448
>>>> #5 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>> #6 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>> #7 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>> #8 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>
>>>>
>>>> #0 PetscTrMallocDefault (a=41943040, lineno=1784, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x7fffffffcec0) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>> #1 0x00007ffff6a91c5f in VecScatterCreate_PtoS (nx=10485760, inidx=0x7fffe9086650, ny=10485760, inidy=0x7fffeb887650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1784
>>>> #2 0x00007ffff6a94c2d in VecScatterCreate_StoP (nx=10485760, inidx=0x7fffeb887650, ny=10485760, inidy=0x7fffe9086650, xin=0x4c1600, yin=0x4be360, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:2197
>>>> #3 0x00007ffff6a60ea0 in VecScatterCreate (xin=0x4c1600, ix=0x4cbe90, yin=0x4be360, iy=0x4c8860, newctx=0x7fffffffd858) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1583
>>>> #4 0x00007ffff7129d9b in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:448
>>>> #5 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>> #6 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>> #7 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>> #8 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>
>>>>
>>>> #0 PetscTrMallocDefault (a=41943040, lineno=1879, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x50fae0) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>> #1 0x00007ffff6a9423b in VecScatterCreate_PtoS (nx=10485760, inidx=0x7fffe9086650, ny=10485760, inidy=0x7fffeb887650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1879
>>>> #2 0x00007ffff6a94c2d in VecScatterCreate_StoP (nx=10485760, inidx=0x7fffeb887650, ny=10485760, inidy=0x7fffe9086650, xin=0x4c1600, yin=0x4be360, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:2197
>>>> #3 0x00007ffff6a60ea0 in VecScatterCreate (xin=0x4c1600, ix=0x4cbe90, yin=0x4be360, iy=0x4c8860, newctx=0x7fffffffd858) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1583
>>>> #4 0x00007ffff7129d9b in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:448
>>>> #5 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>> #6 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>> #7 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>> #8 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>
>>>>
>>>> #0 PetscTrMallocDefault (a=41943040, lineno=1880, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x510860) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>> #1 0x00007ffff6a9431f in VecScatterCreate_PtoS (nx=10485760, inidx=0x7fffe9086650, ny=10485760, inidy=0x7fffeb887650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1880
>>>> #2 0x00007ffff6a94c2d in VecScatterCreate_StoP (nx=10485760, inidx=0x7fffeb887650, ny=10485760, inidy=0x7fffe9086650, xin=0x4c1600, yin=0x4be360, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:2197
>>>> #3 0x00007ffff6a60ea0 in VecScatterCreate (xin=0x4c1600, ix=0x4cbe90, yin=0x4be360, iy=0x4c8860, newctx=0x7fffffffd858) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1583
>>>> #4 0x00007ffff7129d9b in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:448
>>>> #5 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>> #6 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>> #7 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>> #8 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>
>>>>
>>>> #0 PetscTrMallocDefault (a=42598400, lineno=47, function=0x7ffff7afd5d4 <.L__func__.ISGetIndices_Block> "ISGetIndices_Block", filename=0x7ffff7afd53b <.L.str> "/home/jed/petsc/src/vec/is/is/impls/block/block.c", result=0x7fffffffcf88) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>> #1 0x00007ffff691fad2 in ISGetIndices_Block (in=0x4cf6a0, idx=0x7fffffffd370) at /home/jed/petsc/src/vec/is/is/impls/block/block.c:47
>>>> #2 0x00007ffff6936468 in ISGetIndices (is=0x4cf6a0, ptr=0x7fffffffd370) at /home/jed/petsc/src/vec/is/is/interface/index.c:382
>>>> #3 0x00007ffff6a5f84a in VecScatterCreate (xin=0x4be360, ix=0x4cf6a0, yin=0x4c1600, iy=0x50ede0, newctx=0x7fffffffd850) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1493
>>>> #4 0x00007ffff712ed23 in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:1033
>>>> #5 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>> #6 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>> #7 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>> #8 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>
>>>>
>>>> #0 PetscTrMallocDefault (a=42598400, lineno=47, function=0x7ffff7afd5d4 <.L__func__.ISGetIndices_Block> "ISGetIndices_Block", filename=0x7ffff7afd53b <.L.str> "/home/jed/petsc/src/vec/is/is/impls/block/block.c", result=0x7fffffffcf88) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>> #1 0x00007ffff691fad2 in ISGetIndices_Block (in=0x50ede0, idx=0x7fffffffd368) at /home/jed/petsc/src/vec/is/is/impls/block/block.c:47
>>>> #2 0x00007ffff6936468 in ISGetIndices (is=0x50ede0, ptr=0x7fffffffd368) at /home/jed/petsc/src/vec/is/is/interface/index.c:382
>>>> #3 0x00007ffff6a5f938 in VecScatterCreate (xin=0x4be360, ix=0x4cf6a0, yin=0x4c1600, iy=0x50ede0, newctx=0x7fffffffd850) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1495
>>>> #4 0x00007ffff712ed23 in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:1033
>>>> #5 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>> #6 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>> #7 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>> #8 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>
>>>>
>>>> #0 PetscTrMallocDefault (a=42598400, lineno=1745, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x7fffffffcf80) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>> #1 0x00007ffff6a9129f in VecScatterCreate_PtoS (nx=10649600, inidx=0x7fffeafb5650, ny=10649600, inidy=0x7fffe17e3650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x514f70) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1745
>>>> #2 0x00007ffff6a5fa49 in VecScatterCreate (xin=0x4be360, ix=0x4cf6a0, yin=0x4c1600, iy=0x50ede0, newctx=0x7fffffffd850) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1497
>>>> #3 0x00007ffff712ed23 in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:1033
>>>> #4 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>> #5 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>> #6 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>> #7 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>
>>>>
>>>> #0 PetscTrMallocDefault (a=42598400, lineno=1784, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x7fffffffcf60) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>> #1 0x00007ffff6a91c5f in VecScatterCreate_PtoS (nx=10649600, inidx=0x7fffeafb5650, ny=10649600, inidy=0x7fffe17e3650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x514f70) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1784
>>>> #2 0x00007ffff6a5fa49 in VecScatterCreate (xin=0x4be360, ix=0x4cf6a0, yin=0x4c1600, iy=0x50ede0, newctx=0x7fffffffd850) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1497
>>>> #3 0x00007ffff712ed23 in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:1033
>>>> #4 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>> #5 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>> #6 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>> #7 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>
>>>>
>>>> #0 PetscTrMallocDefault (a=41943040, lineno=1879, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x5b9850) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>> #1 0x00007ffff6a9423b in VecScatterCreate_PtoS (nx=10649600, inidx=0x7fffeafb5650, ny=10649600, inidy=0x7fffe17e3650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x514f70) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1879
>>>> #2 0x00007ffff6a5fa49 in VecScatterCreate (xin=0x4be360, ix=0x4cf6a0, yin=0x4c1600, iy=0x50ede0, newctx=0x7fffffffd850) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1497
>>>> #3 0x00007ffff712ed23 in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:1033
>>>> #4 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>> #5 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>> #6 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>> #7 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>
>>>>
>>>> #0 PetscTrMallocDefault (a=41943040, lineno=1880, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x516ba0) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>> #1 0x00007ffff6a9431f in VecScatterCreate_PtoS (nx=10649600, inidx=0x7fffeafb5650, ny=10649600, inidy=0x7fffe17e3650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x514f70) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1880
>>>> #2 0x00007ffff6a5fa49 in VecScatterCreate (xin=0x4be360, ix=0x4cf6a0, yin=0x4c1600, iy=0x50ede0, newctx=0x7fffffffd850) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1497
>>>> #3 0x00007ffff712ed23 in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:1033
>>>> #4 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>> #5 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>> #6 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>> #7 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>
>>>>
>>>> #0 PetscTrMallocDefault (a=42598400, lineno=47, function=0x7ffff7afd5d4 <.L__func__.ISGetIndices_Block> "ISGetIndices_Block", filename=0x7ffff7afd53b <.L.str> "/home/jed/petsc/src/vec/is/is/impls/block/block.c", result=0x7fffffffd4b8) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>> #1 0x00007ffff691fad2 in ISGetIndices_Block (in=0x50ede0, idx=0x7fffffffd5e0) at /home/jed/petsc/src/vec/is/is/impls/block/block.c:47
>>>> #2 0x00007ffff6936468 in ISGetIndices (is=0x50ede0, ptr=0x7fffffffd5e0) at /home/jed/petsc/src/vec/is/is/interface/index.c:382
>>>> #3 0x00007ffff68eb66a in ISLocalToGlobalMappingCreateIS (is=0x50ede0, mapping=0x4a97a0) at /home/jed/petsc/src/vec/is/utils/isltog.c:139
>>>> #4 0x00007ffff713216f in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:1362
>>>> #5 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>> #6 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>> #7 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>> #8 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>
>>>>
>>>> #0 PetscTrMallocDefault (a=83886080, lineno=197, function=0x7ffff7b0171f <.L__func__.VecCreate_MPI_Private> "VecCreate_MPI_Private", filename=0x7ffff7b01735 <.L.str> "/home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c", result=0x513b60) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>> #1 0x00007ffff69e33d8 in VecCreate_MPI_Private (v=0x511c30, alloc=PETSC_TRUE, nghost=0, array=0x0) at /home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c:197
>>>> #2 0x00007ffff69e3f5b in VecCreate_MPI (vv=0x511c30) at /home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c:240
>>>> #3 0x00007ffff6a17cd1 in VecSetType (vec=0x511c30, method=0x7ffff7b01763 <.L.str6> "mpi") at /home/jed/petsc/src/vec/vec/interface/vecreg.c:53
>>>> #4 0x00007ffff69e4395 in VecCreate_Standard (v=0x511c30) at /home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c:267
>>>> #5 0x00007ffff6a17cd1 in VecSetType (vec=0x511c30, method=0x4b9b00 "standard") at /home/jed/petsc/src/vec/vec/interface/vecreg.c:53
>>>> #6 0x00007ffff715ad11 in DMCreateGlobalVector_DA (da=0x4a88d0, g=0x7fffffffdd20) at /home/jed/petsc/src/dm/impls/da/dadist.c:43
>>>> #7 0x00007ffff7370f54 in DMCreateGlobalVector (dm=0x4a88d0, vec=0x7fffffffdd20) at /home/jed/petsc/src/dm/interface/dm.c:680
>>>> #8 0x0000000000400e25 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:20
>>>>
>>>>
>>>> #0 PetscTrMallocDefault (a=83886080, lineno=197, function=0x7ffff7b0171f <.L__func__.VecCreate_MPI_Private> "VecCreate_MPI_Private", filename=0x7ffff7b01735 <.L.str> "/home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c", result=0x4c9550) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>> #1 0x00007ffff69e33d8 in VecCreate_MPI_Private (v=0x4d0ad0, alloc=PETSC_TRUE, nghost=0, array=0x0) at /home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c:197
>>>> #2 0x00007ffff69e3f5b in VecCreate_MPI (vv=0x4d0ad0) at /home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c:240
>>>> #3 0x00007ffff6a17cd1 in VecSetType (vec=0x4d0ad0, method=0x7ffff7b01763 <.L.str6> "mpi") at /home/jed/petsc/src/vec/vec/interface/vecreg.c:53
>>>> #4 0x00007ffff69e4395 in VecCreate_Standard (v=0x4d0ad0) at /home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c:267
>>>> #5 0x00007ffff6a17cd1 in VecSetType (vec=0x4d0ad0, method=0x4b9b00 "standard") at /home/jed/petsc/src/vec/vec/interface/vecreg.c:53
>>>> #6 0x00007ffff715ad11 in DMCreateGlobalVector_DA (da=0x4a88d0, g=0x7fffffffdd18) at /home/jed/petsc/src/dm/impls/da/dadist.c:43
>>>> #7 0x00007ffff7370f54 in DMCreateGlobalVector (dm=0x4a88d0, vec=0x7fffffffdd18) at /home/jed/petsc/src/dm/interface/dm.c:680
>>>> #8 0x0000000000400fb6 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:25
>>>>
>>>>
>
More information about the petsc-dev
mailing list