[petsc-dev] making DA more light weight
Barry Smith
bsmith at mcs.anl.gov
Thu May 15 22:21:33 CDT 2014
ok, adding new code for bs > 12 that uses inner loops or memcpys and clearer profiling
with a fixed width font this is easy to read. The scatter is not terrible in actual memory usage but does possibly produce too much unrecoverable memory? The local to global is bad news IMHO but is fixable. So my theory on memory usage in DMDA was fine but the practice was too shaky.
Haven’t pushed these changes till run complete test suite
Barry
~/Src/petsc/src/dm/examples/tests jed/dmda-memusage $ petscmpiexec -n 2 ./ex20 -dof 1
DMDACreate3d Scatter High water 26.079 MB increase 26.079 MB Current 12.713 MB increase 12.713 MB
DMDACreate3d High water 43.569 MB increase 17.490 MB Current 21.493 MB increase 8.779 MB
DMCreateGlobalVector High water 51.995 MB increase 8.425 MB Current 29.883 MB increase 8.391 MB
DMCreateGlobalVector High water 60.387 MB increase 8.393 MB Current 38.274 MB increase 8.391 MB
~/Src/petsc/src/dm/examples/tests jed/dmda-memusage $ petscmpiexec -n 2 ./ex20 -dof 2
DMDACreate3d Scatter High water 26.075 MB increase 26.075 MB Current 12.713 MB increase 12.713 MB
DMDACreate3d High water 51.991 MB increase 25.915 MB Current 30.275 MB increase 17.562 MB
DMCreateGlobalVector High water 68.817 MB increase 16.826 MB Current 47.054 MB increase 16.779 MB
DMCreateGlobalVector High water 85.619 MB increase 16.802 MB Current 63.833 MB increase 16.779 MB
~/Src/petsc/src/dm/examples/tests jed/dmda-memusage $ petscmpiexec -n 2 ./ex20 -dof 3
DMDACreate3d Scatter High water 26.063 MB increase 26.063 MB Current 12.713 MB increase 12.713 MB
DMDACreate3d High water 56.271 MB increase 30.208 MB Current 34.797 MB increase 22.084 MB
DMCreateGlobalVector High water 81.474 MB increase 25.203 MB Current 59.965 MB increase 25.168 MB
DMCreateGlobalVector High water 106.656 MB increase 25.182 MB Current 85.133 MB increase 25.168 MB
~/Src/petsc/src/dm/examples/tests jed/dmda-memusage $ petscmpiexec -n 2 ./ex20 -dof 4
DMDACreate3d Scatter High water 26.055 MB increase 26.055 MB Current 12.713 MB increase 12.713 MB
DMDACreate3d High water 60.568 MB increase 34.513 MB Current 39.319 MB increase 26.606 MB
DMCreateGlobalVector High water 94.163 MB increase 33.595 MB Current 72.875 MB increase 33.556 MB
DMCreateGlobalVector High water 127.721 MB increase 33.559 MB Current 106.432 MB increase 33.556 MB
~/Src/petsc/src/dm/examples/tests jed/dmda-memusage $
On May 15, 2014, at 10:09 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
> Ok, added #define VecScatterOptimizedBS(mbs) (2 <= mbs && mbs <= 12) for Jed’s branch and now get
>
> ~/Src/petsc/src/dm/examples/tests jed/dmda-memusage $ petscmpiexec -n 2 ./ex20
> DMDACreate3d : 86.041 MB + 86.041 MB
> DMCreateGlobalVector: 169.980 MB + 83.939 MB
> DMCreateGlobalVector: 253.870 MB + 83.890 MB
>
> this is without the optimization of the ISLocalToGlobalMapping.
>
> Still lots more to done, working away
>
>
> Barry
>
>
>
>
> On May 15, 2014, at 8:13 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>
>> Damn, damn, damn
>>
>> define VecScatterOptimizedBS(mbs) ((2 <= mbs && mbs <= 8) || mbs == 12)
>>
>> In the debugger I was always picking a block size that happened to handled nicely while you picked one that is not (but could easily be added). I figured you were in a different PETSc universe.
>>
>> I changed the bs to 8 and added a print inside the DMDACreate3d() at the appropriate place and got
>>
>> ~/Src/petsc/src/dm/examples/tests jed/dmda-memusage $ petscmpiexec -n 2 ./ex20
>> DMDACreate3d before local to global : 43.442 MB
>> DMDACreate3d after local to global : 77.562 MB
>> DMDACreate3d : 77.578 MB + 77.578 MB
>> DMCreateGlobalVector: 144.740 MB + 67.162 MB
>> DMCreateGlobalVector: 211.857 MB + 67.117 MB
>>
>> Note that out of the 77.6 needed for the DMDA 34 is for the local to global. If that is 34 is divided by 8 then the total needed by the DMDA is 43.4 + 4.x < 50 which is less than ONE vector.
>>
>> Barry
>>
>>
>>
>> On May 15, 2014, at 8:00 PM, Jed Brown <jed at jedbrown.org> wrote:
>>
>>> Barry Smith <bsmith at mcs.anl.gov> writes:
>>>
>>>> Damn it Jed, I already told you the sequential case does NOT have the optimization for bs > 1. Run in the debugger on two processes and put a break point in ISGetIndices() and you will see that it is not used by the VecScatterCreate() inside the DMDACreate …
>>>>
>>>>> $ mpirun.hydra -n 2 ./ex20 -malloc_log -malloc_log_threshold 1e6
>>>>>
>>>>>
>>>>> If you run as above in a debugger with a break point at ISGetIndices,
>>>>> the first break is at
>>>>>
>>>>> $ mpirun.hydra -n 1 gdb -ex 'b ISGetIndices' -ex r --args ./ex20 : -n 1 ./ex20
>>>>
>>>> You are not running as above since you went from 2 to 1 process where completely different code is used in creating the scatter!
>>>
>>> Barry, this really is a two-process job, despite the fact that I'm only
>>> attaching a debugger to rank 0. Same with the debugger traces below.
>>> It's standard mpiexec syntax.
>>>
>>> mpiexec args executable pgmargs [ : args executable pgmargs ... ]
>>>
>>>>> [0] 3 12713984 DMSetUp_DA_3D()
>>>>> [0] 1 4259840 ISBlockSetIndices_Block()
>>>>> [0] 4 169738240 ISGetIndices_Block()
>>>>
>>>> This number above is NOT in the VecScatterCreate() it is in the creation of the local to global mapping
>>>>
>>>>> [0] 1 41943040 ISGetIndices_Stride()
>>>>> [0] 1 4259840 ISLocalToGlobalMappingBlock()
>>>>> [0] 2 167772160 VecCreate_MPI_Private()
>>>>> [0] 10 339476480 VecScatterCreate_PtoS()
>>>> ^^^^ I admit this number is larger than I would like to see but it is not caused by ISGetIndices
>>>>
>>>> Barry
>>>>
>>>> Likely we can just remove the sequential case from the checks in the VecScatter and always do the “parallel” and thus remove the unoptimized code that you keep tripping over.
>>>>
>>>>
>>>> On May 15, 2014, at 5:38 PM, Jed Brown <jed at jedbrown.org> wrote:
>>>>
>>>>> Barry Smith <bsmith at mcs.anl.gov> writes:
>>>>>
>>>>>>> Note that ISGetIndices is still called in the parallel case.
>>>>>>
>>>>>> When bs is set > 1 it is not called in the VecScatterCreate()
>>>>>> inside the DMDACreate 2d/3d ! Only ISBlockGetIndices() is
>>>>>> called. The memory usage of VecScatterCreate() is O(dof*number of
>>>>>> ghost points) + O(vector length/dof) in this case
>>>>>>
>>>>>> It is called MULTIPLY times in the ISLocalToGlobalMappingCreateIS()
>>>>>> inside the DMDACreate 2d/3d thus currently this causes small integer
>>>>>> * vector length usage. As I keep telling you this is the problem
>>>>>> area for dof > 1. For dof == 1 I can live with 2.5 *sizeof (vector)
>>>>>>
>>>>>> You need a better test code to run with. Can you use
>>>>>> ksp/ksp/examples/tests/ex42.c ? Or snes/examples/tutorials/ex19.c
>>>>>> in parallel?
>>>>>
>>>>> I added a simple test for this purpose, src/dm/examples/tests/ex20.c
>>>>> in 'jed/dmda-memusage'.
>>>>>
>>>>> $ mpirun.hydra -n 2 ./ex20 -malloc_log -malloc_log_threshold 1e6
>>>>> DMDACreate3d : 225.866 MB + 225.866 MB
>>>>> DMCreateGlobalVector: 309.666 MB + 83.800 MB
>>>>> DMCreateGlobalVector: 393.552 MB + 83.886 MB
>>>>> [0] Maximum memory PetscMalloc()ed 386401520 maximum size of entire process 309821440
>>>>> [0] Memory usage sorted by function
>>>>> [0] 3 12713984 DMSetUp_DA_3D()
>>>>> [0] 1 4259840 ISBlockSetIndices_Block()
>>>>> [0] 4 169738240 ISGetIndices_Block()
>>>>> [0] 1 41943040 ISGetIndices_Stride()
>>>>> [0] 1 4259840 ISLocalToGlobalMappingBlock()
>>>>> [0] 2 167772160 VecCreate_MPI_Private()
>>>>> [0] 10 339476480 VecScatterCreate_PtoS()
>>>>>
>>>>> If you run as above in a debugger with a break point at ISGetIndices,
>>>>> the first break is at
>>>>>
>>>>> $ mpirun.hydra -n 1 gdb -ex 'b ISGetIndices' -ex r --args ./ex20 : -n 1 ./ex20
>>>>> GNU gdb (GDB) 7.7.1
>>>>> Copyright (C) 2014 Free Software Foundation, Inc.
>>>>> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
>>>>> This is free software: you are free to change and redistribute it.
>>>>> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
>>>>> and "show warranty" for details.
>>>>> This GDB was configured as "x86_64-unknown-linux-gnu".
>>>>> Type "show configuration" for configuration details.
>>>>> For bug reporting instructions, please see:
>>>>> <http://www.gnu.org/software/gdb/bugs/>.
>>>>> Find the GDB manual and other documentation resources online at:
>>>>> <http://www.gnu.org/software/gdb/documentation/>.
>>>>> For help, type "help".
>>>>> Type "apropos word" to search for commands related to "word"...
>>>>> No symbol table is loaded. Use the "file" command.
>>>>> Breakpoint 1 (err.c:343) pending.
>>>>> Reading symbols from ./ex20...done.
>>>>> Function "ISGetIndices" not defined.
>>>>> Breakpoint 2 (ISGetIndices) pending.
>>>>> Starting program: /home/jed/petsc/src/dm/examples/tests/ex20
>>>>> warning: Could not load shared library symbols for linux-vdso.so.1.
>>>>> Do you need "set solib-search-path" or "set sysroot"?
>>>>> [Thread debugging using libthread_db enabled]
>>>>> Using host libthread_db library "/usr/lib/libthread_db.so.1".
>>>>>
>>>>> Breakpoint 2, ISGetIndices (is=0x506840, ptr=0x7fffffffd328) at /home/jed/petsc/src/vec/is/is/interface/index.c:379
>>>>> 379 PetscFunctionBegin;
>>>>> (gdb) bt
>>>>> #0 ISGetIndices (is=0x506840, ptr=0x7fffffffd328) at /home/jed/petsc/src/vec/is/is/interface/index.c:379
>>>>> #1 0x00007ffff6a60cb9 in VecScatterCreate (xin=0x4fbfb0, ix=0x506840, yin=0x4f8d10, iy=0x503210, newctx=0x7fffffffd888) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1579
>>>>> #2 0x00007ffff7129d9b in DMSetUp_DA_3D (da=0x4e3280) at /home/jed/petsc/src/dm/impls/da/da3.c:448
>>>>> #3 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4e3280) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>> #4 0x00007ffff736fd9f in DMSetUp (dm=0x4e3280) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>> #5 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd58) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>> #6 0x0000000000400c96 in main (argc=1, argv=0x7fffffffde58) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>>
>>>>> That is this code, which dates from 1995 according to git annotate.
>>>>>
>>>>> /* general case */
>>>>> {
>>>>> PetscInt nx,ny;
>>>>> const PetscInt *idx,*idy;
>>>>> ierr = ISGetLocalSize(ix,&nx);CHKERRQ(ierr);
>>>>> ierr = ISGetIndices(ix,&idx);CHKERRQ(ierr);
>>>>> ierr = ISGetLocalSize(iy,&ny);CHKERRQ(ierr);
>>>>> ierr = ISGetIndices(iy,&idy);CHKERRQ(ierr);
>>>>> if (nx != ny) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_ARG_SIZ,"Local scatter sizes don't match");
>>>>> ierr = VecScatterCreate_StoP(nx,idx,ny,idy,xin,yin,1,ctx);CHKERRQ(ierr);
>>>>> ierr = ISRestoreIndices(ix,&idx);CHKERRQ(ierr);
>>>>> ierr = ISRestoreIndices(iy,&idy);CHKERRQ(ierr);
>>>>> ierr = PetscInfo(xin,"General case: Seq to MPI\n");CHKERRQ(ierr);
>>>>> goto functionend;
>>>>> }
>>>>>
>>>>>
>>>>> Here are all the large allocations, found by stepping through printing
>>>>> the stack trace for each case.
>>>>>
>>>>> $ mpirun.hydra -n 1 gdb -ex 'b mtr.c:228' -ex r --args ./ex20 -malloc_log -malloc_log_threshold 1e7 : -n 1 ./ex20
>>>>> [...]
>>>>>
>>>>> (gdb) bt
>>>>> #0 PetscTrMallocDefault (a=41943040, lineno=47, function=0x7ffff7afd5d4 <.L__func__.ISGetIndices_Block> "ISGetIndices_Block", filename=0x7ffff7afd53b <.L.str> "/home/jed/petsc/src/vec/is/is/impls/block/block.c", result=0x7fffffffcf88) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>> #1 0x00007ffff691fad2 in ISGetIndices_Block (in=0x4cbe90, idx=0x7fffffffd2f8) at /home/jed/petsc/src/vec/is/is/impls/block/block.c:47
>>>>> #2 0x00007ffff6936468 in ISGetIndices (is=0x4cbe90, ptr=0x7fffffffd2f8) at /home/jed/petsc/src/vec/is/is/interface/index.c:382
>>>>> #3 0x00007ffff6a60cb9 in VecScatterCreate (xin=0x4c1600, ix=0x4cbe90, yin=0x4be360, iy=0x4c8860, newctx=0x7fffffffd858) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1579
>>>>> #4 0x00007ffff7129d9b in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:448
>>>>> #5 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>> #6 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>> #7 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>> #8 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>>
>>>>>
>>>>> #0 PetscTrMallocDefault (a=41943040, lineno=160, function=0x7ffff7afd32b <.L__func__.ISGetIndices_Stride> "ISGetIndices_Stride", filename=0x7ffff7afd279 <.L.str> "/home/jed/petsc/src/vec/is/is/impls/stride/stride.c", result=0x7fffffffd2f0) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>> #1 0x00007ffff691b1e9 in ISGetIndices_Stride (in=0x4c8860, idx=0x7fffffffd2f0) at /home/jed/petsc/src/vec/is/is/impls/stride/stride.c:160
>>>>> #2 0x00007ffff6936468 in ISGetIndices (is=0x4c8860, ptr=0x7fffffffd2f0) at /home/jed/petsc/src/vec/is/is/interface/index.c:382
>>>>> #3 0x00007ffff6a60da7 in VecScatterCreate (xin=0x4c1600, ix=0x4cbe90, yin=0x4be360, iy=0x4c8860, newctx=0x7fffffffd858) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1581
>>>>> #4 0x00007ffff7129d9b in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:448
>>>>> #5 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>> #6 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>> #7 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>> #8 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>>
>>>>>
>>>>> #0 PetscTrMallocDefault (a=41943040, lineno=1745, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x7fffffffcee0) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>> #1 0x00007ffff6a9129f in VecScatterCreate_PtoS (nx=10485760, inidx=0x7fffe9086650, ny=10485760, inidy=0x7fffeb887650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1745
>>>>> #2 0x00007ffff6a94c2d in VecScatterCreate_StoP (nx=10485760, inidx=0x7fffeb887650, ny=10485760, inidy=0x7fffe9086650, xin=0x4c1600, yin=0x4be360, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:2197
>>>>> #3 0x00007ffff6a60ea0 in VecScatterCreate (xin=0x4c1600, ix=0x4cbe90, yin=0x4be360, iy=0x4c8860, newctx=0x7fffffffd858) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1583
>>>>> #4 0x00007ffff7129d9b in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:448
>>>>> #5 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>> #6 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>> #7 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>> #8 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>>
>>>>>
>>>>> #0 PetscTrMallocDefault (a=41943040, lineno=1784, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x7fffffffcec0) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>> #1 0x00007ffff6a91c5f in VecScatterCreate_PtoS (nx=10485760, inidx=0x7fffe9086650, ny=10485760, inidy=0x7fffeb887650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1784
>>>>> #2 0x00007ffff6a94c2d in VecScatterCreate_StoP (nx=10485760, inidx=0x7fffeb887650, ny=10485760, inidy=0x7fffe9086650, xin=0x4c1600, yin=0x4be360, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:2197
>>>>> #3 0x00007ffff6a60ea0 in VecScatterCreate (xin=0x4c1600, ix=0x4cbe90, yin=0x4be360, iy=0x4c8860, newctx=0x7fffffffd858) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1583
>>>>> #4 0x00007ffff7129d9b in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:448
>>>>> #5 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>> #6 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>> #7 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>> #8 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>>
>>>>>
>>>>> #0 PetscTrMallocDefault (a=41943040, lineno=1879, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x50fae0) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>> #1 0x00007ffff6a9423b in VecScatterCreate_PtoS (nx=10485760, inidx=0x7fffe9086650, ny=10485760, inidy=0x7fffeb887650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1879
>>>>> #2 0x00007ffff6a94c2d in VecScatterCreate_StoP (nx=10485760, inidx=0x7fffeb887650, ny=10485760, inidy=0x7fffe9086650, xin=0x4c1600, yin=0x4be360, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:2197
>>>>> #3 0x00007ffff6a60ea0 in VecScatterCreate (xin=0x4c1600, ix=0x4cbe90, yin=0x4be360, iy=0x4c8860, newctx=0x7fffffffd858) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1583
>>>>> #4 0x00007ffff7129d9b in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:448
>>>>> #5 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>> #6 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>> #7 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>> #8 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>>
>>>>>
>>>>> #0 PetscTrMallocDefault (a=41943040, lineno=1880, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x510860) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>> #1 0x00007ffff6a9431f in VecScatterCreate_PtoS (nx=10485760, inidx=0x7fffe9086650, ny=10485760, inidy=0x7fffeb887650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1880
>>>>> #2 0x00007ffff6a94c2d in VecScatterCreate_StoP (nx=10485760, inidx=0x7fffeb887650, ny=10485760, inidy=0x7fffe9086650, xin=0x4c1600, yin=0x4be360, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:2197
>>>>> #3 0x00007ffff6a60ea0 in VecScatterCreate (xin=0x4c1600, ix=0x4cbe90, yin=0x4be360, iy=0x4c8860, newctx=0x7fffffffd858) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1583
>>>>> #4 0x00007ffff7129d9b in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:448
>>>>> #5 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>> #6 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>> #7 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>> #8 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>>
>>>>>
>>>>> #0 PetscTrMallocDefault (a=42598400, lineno=47, function=0x7ffff7afd5d4 <.L__func__.ISGetIndices_Block> "ISGetIndices_Block", filename=0x7ffff7afd53b <.L.str> "/home/jed/petsc/src/vec/is/is/impls/block/block.c", result=0x7fffffffcf88) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>> #1 0x00007ffff691fad2 in ISGetIndices_Block (in=0x4cf6a0, idx=0x7fffffffd370) at /home/jed/petsc/src/vec/is/is/impls/block/block.c:47
>>>>> #2 0x00007ffff6936468 in ISGetIndices (is=0x4cf6a0, ptr=0x7fffffffd370) at /home/jed/petsc/src/vec/is/is/interface/index.c:382
>>>>> #3 0x00007ffff6a5f84a in VecScatterCreate (xin=0x4be360, ix=0x4cf6a0, yin=0x4c1600, iy=0x50ede0, newctx=0x7fffffffd850) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1493
>>>>> #4 0x00007ffff712ed23 in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:1033
>>>>> #5 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>> #6 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>> #7 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>> #8 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>>
>>>>>
>>>>> #0 PetscTrMallocDefault (a=42598400, lineno=47, function=0x7ffff7afd5d4 <.L__func__.ISGetIndices_Block> "ISGetIndices_Block", filename=0x7ffff7afd53b <.L.str> "/home/jed/petsc/src/vec/is/is/impls/block/block.c", result=0x7fffffffcf88) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>> #1 0x00007ffff691fad2 in ISGetIndices_Block (in=0x50ede0, idx=0x7fffffffd368) at /home/jed/petsc/src/vec/is/is/impls/block/block.c:47
>>>>> #2 0x00007ffff6936468 in ISGetIndices (is=0x50ede0, ptr=0x7fffffffd368) at /home/jed/petsc/src/vec/is/is/interface/index.c:382
>>>>> #3 0x00007ffff6a5f938 in VecScatterCreate (xin=0x4be360, ix=0x4cf6a0, yin=0x4c1600, iy=0x50ede0, newctx=0x7fffffffd850) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1495
>>>>> #4 0x00007ffff712ed23 in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:1033
>>>>> #5 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>> #6 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>> #7 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>> #8 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>>
>>>>>
>>>>> #0 PetscTrMallocDefault (a=42598400, lineno=1745, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x7fffffffcf80) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>> #1 0x00007ffff6a9129f in VecScatterCreate_PtoS (nx=10649600, inidx=0x7fffeafb5650, ny=10649600, inidy=0x7fffe17e3650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x514f70) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1745
>>>>> #2 0x00007ffff6a5fa49 in VecScatterCreate (xin=0x4be360, ix=0x4cf6a0, yin=0x4c1600, iy=0x50ede0, newctx=0x7fffffffd850) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1497
>>>>> #3 0x00007ffff712ed23 in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:1033
>>>>> #4 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>> #5 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>> #6 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>> #7 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>>
>>>>>
>>>>> #0 PetscTrMallocDefault (a=42598400, lineno=1784, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x7fffffffcf60) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>> #1 0x00007ffff6a91c5f in VecScatterCreate_PtoS (nx=10649600, inidx=0x7fffeafb5650, ny=10649600, inidy=0x7fffe17e3650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x514f70) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1784
>>>>> #2 0x00007ffff6a5fa49 in VecScatterCreate (xin=0x4be360, ix=0x4cf6a0, yin=0x4c1600, iy=0x50ede0, newctx=0x7fffffffd850) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1497
>>>>> #3 0x00007ffff712ed23 in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:1033
>>>>> #4 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>> #5 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>> #6 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>> #7 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>>
>>>>>
>>>>> #0 PetscTrMallocDefault (a=41943040, lineno=1879, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x5b9850) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>> #1 0x00007ffff6a9423b in VecScatterCreate_PtoS (nx=10649600, inidx=0x7fffeafb5650, ny=10649600, inidy=0x7fffe17e3650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x514f70) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1879
>>>>> #2 0x00007ffff6a5fa49 in VecScatterCreate (xin=0x4be360, ix=0x4cf6a0, yin=0x4c1600, iy=0x50ede0, newctx=0x7fffffffd850) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1497
>>>>> #3 0x00007ffff712ed23 in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:1033
>>>>> #4 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>> #5 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>> #6 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>> #7 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>>
>>>>>
>>>>> #0 PetscTrMallocDefault (a=41943040, lineno=1880, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x516ba0) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>> #1 0x00007ffff6a9431f in VecScatterCreate_PtoS (nx=10649600, inidx=0x7fffeafb5650, ny=10649600, inidy=0x7fffe17e3650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x514f70) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1880
>>>>> #2 0x00007ffff6a5fa49 in VecScatterCreate (xin=0x4be360, ix=0x4cf6a0, yin=0x4c1600, iy=0x50ede0, newctx=0x7fffffffd850) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1497
>>>>> #3 0x00007ffff712ed23 in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:1033
>>>>> #4 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>> #5 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>> #6 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>> #7 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>>
>>>>>
>>>>> #0 PetscTrMallocDefault (a=42598400, lineno=47, function=0x7ffff7afd5d4 <.L__func__.ISGetIndices_Block> "ISGetIndices_Block", filename=0x7ffff7afd53b <.L.str> "/home/jed/petsc/src/vec/is/is/impls/block/block.c", result=0x7fffffffd4b8) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>> #1 0x00007ffff691fad2 in ISGetIndices_Block (in=0x50ede0, idx=0x7fffffffd5e0) at /home/jed/petsc/src/vec/is/is/impls/block/block.c:47
>>>>> #2 0x00007ffff6936468 in ISGetIndices (is=0x50ede0, ptr=0x7fffffffd5e0) at /home/jed/petsc/src/vec/is/is/interface/index.c:382
>>>>> #3 0x00007ffff68eb66a in ISLocalToGlobalMappingCreateIS (is=0x50ede0, mapping=0x4a97a0) at /home/jed/petsc/src/vec/is/utils/isltog.c:139
>>>>> #4 0x00007ffff713216f in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:1362
>>>>> #5 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>> #6 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>> #7 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>> #8 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>>
>>>>>
>>>>> #0 PetscTrMallocDefault (a=83886080, lineno=197, function=0x7ffff7b0171f <.L__func__.VecCreate_MPI_Private> "VecCreate_MPI_Private", filename=0x7ffff7b01735 <.L.str> "/home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c", result=0x513b60) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>> #1 0x00007ffff69e33d8 in VecCreate_MPI_Private (v=0x511c30, alloc=PETSC_TRUE, nghost=0, array=0x0) at /home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c:197
>>>>> #2 0x00007ffff69e3f5b in VecCreate_MPI (vv=0x511c30) at /home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c:240
>>>>> #3 0x00007ffff6a17cd1 in VecSetType (vec=0x511c30, method=0x7ffff7b01763 <.L.str6> "mpi") at /home/jed/petsc/src/vec/vec/interface/vecreg.c:53
>>>>> #4 0x00007ffff69e4395 in VecCreate_Standard (v=0x511c30) at /home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c:267
>>>>> #5 0x00007ffff6a17cd1 in VecSetType (vec=0x511c30, method=0x4b9b00 "standard") at /home/jed/petsc/src/vec/vec/interface/vecreg.c:53
>>>>> #6 0x00007ffff715ad11 in DMCreateGlobalVector_DA (da=0x4a88d0, g=0x7fffffffdd20) at /home/jed/petsc/src/dm/impls/da/dadist.c:43
>>>>> #7 0x00007ffff7370f54 in DMCreateGlobalVector (dm=0x4a88d0, vec=0x7fffffffdd20) at /home/jed/petsc/src/dm/interface/dm.c:680
>>>>> #8 0x0000000000400e25 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:20
>>>>>
>>>>>
>>>>> #0 PetscTrMallocDefault (a=83886080, lineno=197, function=0x7ffff7b0171f <.L__func__.VecCreate_MPI_Private> "VecCreate_MPI_Private", filename=0x7ffff7b01735 <.L.str> "/home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c", result=0x4c9550) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>> #1 0x00007ffff69e33d8 in VecCreate_MPI_Private (v=0x4d0ad0, alloc=PETSC_TRUE, nghost=0, array=0x0) at /home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c:197
>>>>> #2 0x00007ffff69e3f5b in VecCreate_MPI (vv=0x4d0ad0) at /home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c:240
>>>>> #3 0x00007ffff6a17cd1 in VecSetType (vec=0x4d0ad0, method=0x7ffff7b01763 <.L.str6> "mpi") at /home/jed/petsc/src/vec/vec/interface/vecreg.c:53
>>>>> #4 0x00007ffff69e4395 in VecCreate_Standard (v=0x4d0ad0) at /home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c:267
>>>>> #5 0x00007ffff6a17cd1 in VecSetType (vec=0x4d0ad0, method=0x4b9b00 "standard") at /home/jed/petsc/src/vec/vec/interface/vecreg.c:53
>>>>> #6 0x00007ffff715ad11 in DMCreateGlobalVector_DA (da=0x4a88d0, g=0x7fffffffdd18) at /home/jed/petsc/src/dm/impls/da/dadist.c:43
>>>>> #7 0x00007ffff7370f54 in DMCreateGlobalVector (dm=0x4a88d0, vec=0x7fffffffdd18) at /home/jed/petsc/src/dm/interface/dm.c:680
>>>>> #8 0x0000000000400fb6 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:25
>>>>>
>>>>>
>>
>
More information about the petsc-dev
mailing list