[petsc-dev] making DA more light weight
Jed Brown
jed at jedbrown.org
Thu May 15 22:30:34 CDT 2014
Barry Smith <bsmith at mcs.anl.gov> writes:
> ok, adding new code for bs > 12 that uses inner loops or memcpys and clearer profiling
>
> with a fixed width font this is easy to read. The scatter is not
> terrible in actual memory usage but does possibly produce too much
> unrecoverable memory? The local to global is bad news IMHO but is
> fixable. So my theory on memory usage in DMDA was fine but the
> practice was too shaky.
Great! This looks much better and now I think we'll be in good shape
after the relatively straightforward local-to-global optimization.
> Haven’t pushed these changes till run complete test suite
>
> Barry
>
>
> ~/Src/petsc/src/dm/examples/tests jed/dmda-memusage $ petscmpiexec -n 2 ./ex20 -dof 1
> DMDACreate3d Scatter High water 26.079 MB increase 26.079 MB Current 12.713 MB increase 12.713 MB
> DMDACreate3d High water 43.569 MB increase 17.490 MB Current 21.493 MB increase 8.779 MB
> DMCreateGlobalVector High water 51.995 MB increase 8.425 MB Current 29.883 MB increase 8.391 MB
> DMCreateGlobalVector High water 60.387 MB increase 8.393 MB Current 38.274 MB increase 8.391 MB
> ~/Src/petsc/src/dm/examples/tests jed/dmda-memusage $ petscmpiexec -n 2 ./ex20 -dof 2
> DMDACreate3d Scatter High water 26.075 MB increase 26.075 MB Current 12.713 MB increase 12.713 MB
> DMDACreate3d High water 51.991 MB increase 25.915 MB Current 30.275 MB increase 17.562 MB
> DMCreateGlobalVector High water 68.817 MB increase 16.826 MB Current 47.054 MB increase 16.779 MB
> DMCreateGlobalVector High water 85.619 MB increase 16.802 MB Current 63.833 MB increase 16.779 MB
> ~/Src/petsc/src/dm/examples/tests jed/dmda-memusage $ petscmpiexec -n 2 ./ex20 -dof 3
> DMDACreate3d Scatter High water 26.063 MB increase 26.063 MB Current 12.713 MB increase 12.713 MB
> DMDACreate3d High water 56.271 MB increase 30.208 MB Current 34.797 MB increase 22.084 MB
> DMCreateGlobalVector High water 81.474 MB increase 25.203 MB Current 59.965 MB increase 25.168 MB
> DMCreateGlobalVector High water 106.656 MB increase 25.182 MB Current 85.133 MB increase 25.168 MB
> ~/Src/petsc/src/dm/examples/tests jed/dmda-memusage $ petscmpiexec -n 2 ./ex20 -dof 4
> DMDACreate3d Scatter High water 26.055 MB increase 26.055 MB Current 12.713 MB increase 12.713 MB
> DMDACreate3d High water 60.568 MB increase 34.513 MB Current 39.319 MB increase 26.606 MB
> DMCreateGlobalVector High water 94.163 MB increase 33.595 MB Current 72.875 MB increase 33.556 MB
> DMCreateGlobalVector High water 127.721 MB increase 33.559 MB Current 106.432 MB increase 33.556 MB
> ~/Src/petsc/src/dm/examples/tests jed/dmda-memusage $
>
>
> On May 15, 2014, at 10:09 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>
>> Ok, added #define VecScatterOptimizedBS(mbs) (2 <= mbs && mbs <= 12) for Jed’s branch and now get
>>
>> ~/Src/petsc/src/dm/examples/tests jed/dmda-memusage $ petscmpiexec -n 2 ./ex20
>> DMDACreate3d : 86.041 MB + 86.041 MB
>> DMCreateGlobalVector: 169.980 MB + 83.939 MB
>> DMCreateGlobalVector: 253.870 MB + 83.890 MB
>>
>> this is without the optimization of the ISLocalToGlobalMapping.
>>
>> Still lots more to done, working away
>>
>>
>> Barry
>>
>>
>>
>>
>> On May 15, 2014, at 8:13 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>>
>>> Damn, damn, damn
>>>
>>> define VecScatterOptimizedBS(mbs) ((2 <= mbs && mbs <= 8) || mbs == 12)
>>>
>>> In the debugger I was always picking a block size that happened to handled nicely while you picked one that is not (but could easily be added). I figured you were in a different PETSc universe.
>>>
>>> I changed the bs to 8 and added a print inside the DMDACreate3d() at the appropriate place and got
>>>
>>> ~/Src/petsc/src/dm/examples/tests jed/dmda-memusage $ petscmpiexec -n 2 ./ex20
>>> DMDACreate3d before local to global : 43.442 MB
>>> DMDACreate3d after local to global : 77.562 MB
>>> DMDACreate3d : 77.578 MB + 77.578 MB
>>> DMCreateGlobalVector: 144.740 MB + 67.162 MB
>>> DMCreateGlobalVector: 211.857 MB + 67.117 MB
>>>
>>> Note that out of the 77.6 needed for the DMDA 34 is for the local to global. If that is 34 is divided by 8 then the total needed by the DMDA is 43.4 + 4.x < 50 which is less than ONE vector.
>>>
>>> Barry
>>>
>>>
>>>
>>> On May 15, 2014, at 8:00 PM, Jed Brown <jed at jedbrown.org> wrote:
>>>
>>>> Barry Smith <bsmith at mcs.anl.gov> writes:
>>>>
>>>>> Damn it Jed, I already told you the sequential case does NOT have the optimization for bs > 1. Run in the debugger on two processes and put a break point in ISGetIndices() and you will see that it is not used by the VecScatterCreate() inside the DMDACreate …
>>>>>
>>>>>> $ mpirun.hydra -n 2 ./ex20 -malloc_log -malloc_log_threshold 1e6
>>>>>>
>>>>>>
>>>>>> If you run as above in a debugger with a break point at ISGetIndices,
>>>>>> the first break is at
>>>>>>
>>>>>> $ mpirun.hydra -n 1 gdb -ex 'b ISGetIndices' -ex r --args ./ex20 : -n 1 ./ex20
>>>>>
>>>>> You are not running as above since you went from 2 to 1 process where completely different code is used in creating the scatter!
>>>>
>>>> Barry, this really is a two-process job, despite the fact that I'm only
>>>> attaching a debugger to rank 0. Same with the debugger traces below.
>>>> It's standard mpiexec syntax.
>>>>
>>>> mpiexec args executable pgmargs [ : args executable pgmargs ... ]
>>>>
>>>>>> [0] 3 12713984 DMSetUp_DA_3D()
>>>>>> [0] 1 4259840 ISBlockSetIndices_Block()
>>>>>> [0] 4 169738240 ISGetIndices_Block()
>>>>>
>>>>> This number above is NOT in the VecScatterCreate() it is in the creation of the local to global mapping
>>>>>
>>>>>> [0] 1 41943040 ISGetIndices_Stride()
>>>>>> [0] 1 4259840 ISLocalToGlobalMappingBlock()
>>>>>> [0] 2 167772160 VecCreate_MPI_Private()
>>>>>> [0] 10 339476480 VecScatterCreate_PtoS()
>>>>> ^^^^ I admit this number is larger than I would like to see but it is not caused by ISGetIndices
>>>>>
>>>>> Barry
>>>>>
>>>>> Likely we can just remove the sequential case from the checks in the VecScatter and always do the “parallel” and thus remove the unoptimized code that you keep tripping over.
>>>>>
>>>>>
>>>>> On May 15, 2014, at 5:38 PM, Jed Brown <jed at jedbrown.org> wrote:
>>>>>
>>>>>> Barry Smith <bsmith at mcs.anl.gov> writes:
>>>>>>
>>>>>>>> Note that ISGetIndices is still called in the parallel case.
>>>>>>>
>>>>>>> When bs is set > 1 it is not called in the VecScatterCreate()
>>>>>>> inside the DMDACreate 2d/3d ! Only ISBlockGetIndices() is
>>>>>>> called. The memory usage of VecScatterCreate() is O(dof*number of
>>>>>>> ghost points) + O(vector length/dof) in this case
>>>>>>>
>>>>>>> It is called MULTIPLY times in the ISLocalToGlobalMappingCreateIS()
>>>>>>> inside the DMDACreate 2d/3d thus currently this causes small integer
>>>>>>> * vector length usage. As I keep telling you this is the problem
>>>>>>> area for dof > 1. For dof == 1 I can live with 2.5 *sizeof (vector)
>>>>>>>
>>>>>>> You need a better test code to run with. Can you use
>>>>>>> ksp/ksp/examples/tests/ex42.c ? Or snes/examples/tutorials/ex19.c
>>>>>>> in parallel?
>>>>>>
>>>>>> I added a simple test for this purpose, src/dm/examples/tests/ex20.c
>>>>>> in 'jed/dmda-memusage'.
>>>>>>
>>>>>> $ mpirun.hydra -n 2 ./ex20 -malloc_log -malloc_log_threshold 1e6
>>>>>> DMDACreate3d : 225.866 MB + 225.866 MB
>>>>>> DMCreateGlobalVector: 309.666 MB + 83.800 MB
>>>>>> DMCreateGlobalVector: 393.552 MB + 83.886 MB
>>>>>> [0] Maximum memory PetscMalloc()ed 386401520 maximum size of entire process 309821440
>>>>>> [0] Memory usage sorted by function
>>>>>> [0] 3 12713984 DMSetUp_DA_3D()
>>>>>> [0] 1 4259840 ISBlockSetIndices_Block()
>>>>>> [0] 4 169738240 ISGetIndices_Block()
>>>>>> [0] 1 41943040 ISGetIndices_Stride()
>>>>>> [0] 1 4259840 ISLocalToGlobalMappingBlock()
>>>>>> [0] 2 167772160 VecCreate_MPI_Private()
>>>>>> [0] 10 339476480 VecScatterCreate_PtoS()
>>>>>>
>>>>>> If you run as above in a debugger with a break point at ISGetIndices,
>>>>>> the first break is at
>>>>>>
>>>>>> $ mpirun.hydra -n 1 gdb -ex 'b ISGetIndices' -ex r --args ./ex20 : -n 1 ./ex20
>>>>>> GNU gdb (GDB) 7.7.1
>>>>>> Copyright (C) 2014 Free Software Foundation, Inc.
>>>>>> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
>>>>>> This is free software: you are free to change and redistribute it.
>>>>>> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
>>>>>> and "show warranty" for details.
>>>>>> This GDB was configured as "x86_64-unknown-linux-gnu".
>>>>>> Type "show configuration" for configuration details.
>>>>>> For bug reporting instructions, please see:
>>>>>> <http://www.gnu.org/software/gdb/bugs/>.
>>>>>> Find the GDB manual and other documentation resources online at:
>>>>>> <http://www.gnu.org/software/gdb/documentation/>.
>>>>>> For help, type "help".
>>>>>> Type "apropos word" to search for commands related to "word"...
>>>>>> No symbol table is loaded. Use the "file" command.
>>>>>> Breakpoint 1 (err.c:343) pending.
>>>>>> Reading symbols from ./ex20...done.
>>>>>> Function "ISGetIndices" not defined.
>>>>>> Breakpoint 2 (ISGetIndices) pending.
>>>>>> Starting program: /home/jed/petsc/src/dm/examples/tests/ex20
>>>>>> warning: Could not load shared library symbols for linux-vdso.so.1.
>>>>>> Do you need "set solib-search-path" or "set sysroot"?
>>>>>> [Thread debugging using libthread_db enabled]
>>>>>> Using host libthread_db library "/usr/lib/libthread_db.so.1".
>>>>>>
>>>>>> Breakpoint 2, ISGetIndices (is=0x506840, ptr=0x7fffffffd328) at /home/jed/petsc/src/vec/is/is/interface/index.c:379
>>>>>> 379 PetscFunctionBegin;
>>>>>> (gdb) bt
>>>>>> #0 ISGetIndices (is=0x506840, ptr=0x7fffffffd328) at /home/jed/petsc/src/vec/is/is/interface/index.c:379
>>>>>> #1 0x00007ffff6a60cb9 in VecScatterCreate (xin=0x4fbfb0, ix=0x506840, yin=0x4f8d10, iy=0x503210, newctx=0x7fffffffd888) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1579
>>>>>> #2 0x00007ffff7129d9b in DMSetUp_DA_3D (da=0x4e3280) at /home/jed/petsc/src/dm/impls/da/da3.c:448
>>>>>> #3 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4e3280) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>>> #4 0x00007ffff736fd9f in DMSetUp (dm=0x4e3280) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>>> #5 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd58) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>>> #6 0x0000000000400c96 in main (argc=1, argv=0x7fffffffde58) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>>>
>>>>>> That is this code, which dates from 1995 according to git annotate.
>>>>>>
>>>>>> /* general case */
>>>>>> {
>>>>>> PetscInt nx,ny;
>>>>>> const PetscInt *idx,*idy;
>>>>>> ierr = ISGetLocalSize(ix,&nx);CHKERRQ(ierr);
>>>>>> ierr = ISGetIndices(ix,&idx);CHKERRQ(ierr);
>>>>>> ierr = ISGetLocalSize(iy,&ny);CHKERRQ(ierr);
>>>>>> ierr = ISGetIndices(iy,&idy);CHKERRQ(ierr);
>>>>>> if (nx != ny) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_ARG_SIZ,"Local scatter sizes don't match");
>>>>>> ierr = VecScatterCreate_StoP(nx,idx,ny,idy,xin,yin,1,ctx);CHKERRQ(ierr);
>>>>>> ierr = ISRestoreIndices(ix,&idx);CHKERRQ(ierr);
>>>>>> ierr = ISRestoreIndices(iy,&idy);CHKERRQ(ierr);
>>>>>> ierr = PetscInfo(xin,"General case: Seq to MPI\n");CHKERRQ(ierr);
>>>>>> goto functionend;
>>>>>> }
>>>>>>
>>>>>>
>>>>>> Here are all the large allocations, found by stepping through printing
>>>>>> the stack trace for each case.
>>>>>>
>>>>>> $ mpirun.hydra -n 1 gdb -ex 'b mtr.c:228' -ex r --args ./ex20 -malloc_log -malloc_log_threshold 1e7 : -n 1 ./ex20
>>>>>> [...]
>>>>>>
>>>>>> (gdb) bt
>>>>>> #0 PetscTrMallocDefault (a=41943040, lineno=47, function=0x7ffff7afd5d4 <.L__func__.ISGetIndices_Block> "ISGetIndices_Block", filename=0x7ffff7afd53b <.L.str> "/home/jed/petsc/src/vec/is/is/impls/block/block.c", result=0x7fffffffcf88) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>>> #1 0x00007ffff691fad2 in ISGetIndices_Block (in=0x4cbe90, idx=0x7fffffffd2f8) at /home/jed/petsc/src/vec/is/is/impls/block/block.c:47
>>>>>> #2 0x00007ffff6936468 in ISGetIndices (is=0x4cbe90, ptr=0x7fffffffd2f8) at /home/jed/petsc/src/vec/is/is/interface/index.c:382
>>>>>> #3 0x00007ffff6a60cb9 in VecScatterCreate (xin=0x4c1600, ix=0x4cbe90, yin=0x4be360, iy=0x4c8860, newctx=0x7fffffffd858) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1579
>>>>>> #4 0x00007ffff7129d9b in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:448
>>>>>> #5 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>>> #6 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>>> #7 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>>> #8 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>>>
>>>>>>
>>>>>> #0 PetscTrMallocDefault (a=41943040, lineno=160, function=0x7ffff7afd32b <.L__func__.ISGetIndices_Stride> "ISGetIndices_Stride", filename=0x7ffff7afd279 <.L.str> "/home/jed/petsc/src/vec/is/is/impls/stride/stride.c", result=0x7fffffffd2f0) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>>> #1 0x00007ffff691b1e9 in ISGetIndices_Stride (in=0x4c8860, idx=0x7fffffffd2f0) at /home/jed/petsc/src/vec/is/is/impls/stride/stride.c:160
>>>>>> #2 0x00007ffff6936468 in ISGetIndices (is=0x4c8860, ptr=0x7fffffffd2f0) at /home/jed/petsc/src/vec/is/is/interface/index.c:382
>>>>>> #3 0x00007ffff6a60da7 in VecScatterCreate (xin=0x4c1600, ix=0x4cbe90, yin=0x4be360, iy=0x4c8860, newctx=0x7fffffffd858) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1581
>>>>>> #4 0x00007ffff7129d9b in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:448
>>>>>> #5 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>>> #6 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>>> #7 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>>> #8 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>>>
>>>>>>
>>>>>> #0 PetscTrMallocDefault (a=41943040, lineno=1745, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x7fffffffcee0) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>>> #1 0x00007ffff6a9129f in VecScatterCreate_PtoS (nx=10485760, inidx=0x7fffe9086650, ny=10485760, inidy=0x7fffeb887650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1745
>>>>>> #2 0x00007ffff6a94c2d in VecScatterCreate_StoP (nx=10485760, inidx=0x7fffeb887650, ny=10485760, inidy=0x7fffe9086650, xin=0x4c1600, yin=0x4be360, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:2197
>>>>>> #3 0x00007ffff6a60ea0 in VecScatterCreate (xin=0x4c1600, ix=0x4cbe90, yin=0x4be360, iy=0x4c8860, newctx=0x7fffffffd858) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1583
>>>>>> #4 0x00007ffff7129d9b in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:448
>>>>>> #5 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>>> #6 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>>> #7 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>>> #8 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>>>
>>>>>>
>>>>>> #0 PetscTrMallocDefault (a=41943040, lineno=1784, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x7fffffffcec0) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>>> #1 0x00007ffff6a91c5f in VecScatterCreate_PtoS (nx=10485760, inidx=0x7fffe9086650, ny=10485760, inidy=0x7fffeb887650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1784
>>>>>> #2 0x00007ffff6a94c2d in VecScatterCreate_StoP (nx=10485760, inidx=0x7fffeb887650, ny=10485760, inidy=0x7fffe9086650, xin=0x4c1600, yin=0x4be360, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:2197
>>>>>> #3 0x00007ffff6a60ea0 in VecScatterCreate (xin=0x4c1600, ix=0x4cbe90, yin=0x4be360, iy=0x4c8860, newctx=0x7fffffffd858) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1583
>>>>>> #4 0x00007ffff7129d9b in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:448
>>>>>> #5 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>>> #6 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>>> #7 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>>> #8 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>>>
>>>>>>
>>>>>> #0 PetscTrMallocDefault (a=41943040, lineno=1879, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x50fae0) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>>> #1 0x00007ffff6a9423b in VecScatterCreate_PtoS (nx=10485760, inidx=0x7fffe9086650, ny=10485760, inidy=0x7fffeb887650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1879
>>>>>> #2 0x00007ffff6a94c2d in VecScatterCreate_StoP (nx=10485760, inidx=0x7fffeb887650, ny=10485760, inidy=0x7fffe9086650, xin=0x4c1600, yin=0x4be360, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:2197
>>>>>> #3 0x00007ffff6a60ea0 in VecScatterCreate (xin=0x4c1600, ix=0x4cbe90, yin=0x4be360, iy=0x4c8860, newctx=0x7fffffffd858) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1583
>>>>>> #4 0x00007ffff7129d9b in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:448
>>>>>> #5 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>>> #6 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>>> #7 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>>> #8 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>>>
>>>>>>
>>>>>> #0 PetscTrMallocDefault (a=41943040, lineno=1880, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x510860) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>>> #1 0x00007ffff6a9431f in VecScatterCreate_PtoS (nx=10485760, inidx=0x7fffe9086650, ny=10485760, inidy=0x7fffeb887650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1880
>>>>>> #2 0x00007ffff6a94c2d in VecScatterCreate_StoP (nx=10485760, inidx=0x7fffeb887650, ny=10485760, inidy=0x7fffe9086650, xin=0x4c1600, yin=0x4be360, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:2197
>>>>>> #3 0x00007ffff6a60ea0 in VecScatterCreate (xin=0x4c1600, ix=0x4cbe90, yin=0x4be360, iy=0x4c8860, newctx=0x7fffffffd858) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1583
>>>>>> #4 0x00007ffff7129d9b in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:448
>>>>>> #5 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>>> #6 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>>> #7 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>>> #8 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>>>
>>>>>>
>>>>>> #0 PetscTrMallocDefault (a=42598400, lineno=47, function=0x7ffff7afd5d4 <.L__func__.ISGetIndices_Block> "ISGetIndices_Block", filename=0x7ffff7afd53b <.L.str> "/home/jed/petsc/src/vec/is/is/impls/block/block.c", result=0x7fffffffcf88) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>>> #1 0x00007ffff691fad2 in ISGetIndices_Block (in=0x4cf6a0, idx=0x7fffffffd370) at /home/jed/petsc/src/vec/is/is/impls/block/block.c:47
>>>>>> #2 0x00007ffff6936468 in ISGetIndices (is=0x4cf6a0, ptr=0x7fffffffd370) at /home/jed/petsc/src/vec/is/is/interface/index.c:382
>>>>>> #3 0x00007ffff6a5f84a in VecScatterCreate (xin=0x4be360, ix=0x4cf6a0, yin=0x4c1600, iy=0x50ede0, newctx=0x7fffffffd850) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1493
>>>>>> #4 0x00007ffff712ed23 in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:1033
>>>>>> #5 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>>> #6 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>>> #7 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>>> #8 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>>>
>>>>>>
>>>>>> #0 PetscTrMallocDefault (a=42598400, lineno=47, function=0x7ffff7afd5d4 <.L__func__.ISGetIndices_Block> "ISGetIndices_Block", filename=0x7ffff7afd53b <.L.str> "/home/jed/petsc/src/vec/is/is/impls/block/block.c", result=0x7fffffffcf88) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>>> #1 0x00007ffff691fad2 in ISGetIndices_Block (in=0x50ede0, idx=0x7fffffffd368) at /home/jed/petsc/src/vec/is/is/impls/block/block.c:47
>>>>>> #2 0x00007ffff6936468 in ISGetIndices (is=0x50ede0, ptr=0x7fffffffd368) at /home/jed/petsc/src/vec/is/is/interface/index.c:382
>>>>>> #3 0x00007ffff6a5f938 in VecScatterCreate (xin=0x4be360, ix=0x4cf6a0, yin=0x4c1600, iy=0x50ede0, newctx=0x7fffffffd850) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1495
>>>>>> #4 0x00007ffff712ed23 in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:1033
>>>>>> #5 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>>> #6 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>>> #7 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>>> #8 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>>>
>>>>>>
>>>>>> #0 PetscTrMallocDefault (a=42598400, lineno=1745, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x7fffffffcf80) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>>> #1 0x00007ffff6a9129f in VecScatterCreate_PtoS (nx=10649600, inidx=0x7fffeafb5650, ny=10649600, inidy=0x7fffe17e3650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x514f70) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1745
>>>>>> #2 0x00007ffff6a5fa49 in VecScatterCreate (xin=0x4be360, ix=0x4cf6a0, yin=0x4c1600, iy=0x50ede0, newctx=0x7fffffffd850) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1497
>>>>>> #3 0x00007ffff712ed23 in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:1033
>>>>>> #4 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>>> #5 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>>> #6 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>>> #7 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>>>
>>>>>>
>>>>>> #0 PetscTrMallocDefault (a=42598400, lineno=1784, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x7fffffffcf60) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>>> #1 0x00007ffff6a91c5f in VecScatterCreate_PtoS (nx=10649600, inidx=0x7fffeafb5650, ny=10649600, inidy=0x7fffe17e3650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x514f70) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1784
>>>>>> #2 0x00007ffff6a5fa49 in VecScatterCreate (xin=0x4be360, ix=0x4cf6a0, yin=0x4c1600, iy=0x50ede0, newctx=0x7fffffffd850) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1497
>>>>>> #3 0x00007ffff712ed23 in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:1033
>>>>>> #4 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>>> #5 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>>> #6 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>>> #7 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>>>
>>>>>>
>>>>>> #0 PetscTrMallocDefault (a=41943040, lineno=1879, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x5b9850) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>>> #1 0x00007ffff6a9423b in VecScatterCreate_PtoS (nx=10649600, inidx=0x7fffeafb5650, ny=10649600, inidy=0x7fffe17e3650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x514f70) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1879
>>>>>> #2 0x00007ffff6a5fa49 in VecScatterCreate (xin=0x4be360, ix=0x4cf6a0, yin=0x4c1600, iy=0x50ede0, newctx=0x7fffffffd850) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1497
>>>>>> #3 0x00007ffff712ed23 in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:1033
>>>>>> #4 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>>> #5 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>>> #6 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>>> #7 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>>>
>>>>>>
>>>>>> #0 PetscTrMallocDefault (a=41943040, lineno=1880, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x516ba0) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>>> #1 0x00007ffff6a9431f in VecScatterCreate_PtoS (nx=10649600, inidx=0x7fffeafb5650, ny=10649600, inidy=0x7fffe17e3650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x514f70) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1880
>>>>>> #2 0x00007ffff6a5fa49 in VecScatterCreate (xin=0x4be360, ix=0x4cf6a0, yin=0x4c1600, iy=0x50ede0, newctx=0x7fffffffd850) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1497
>>>>>> #3 0x00007ffff712ed23 in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:1033
>>>>>> #4 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>>> #5 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>>> #6 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>>> #7 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>>>
>>>>>>
>>>>>> #0 PetscTrMallocDefault (a=42598400, lineno=47, function=0x7ffff7afd5d4 <.L__func__.ISGetIndices_Block> "ISGetIndices_Block", filename=0x7ffff7afd53b <.L.str> "/home/jed/petsc/src/vec/is/is/impls/block/block.c", result=0x7fffffffd4b8) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>>> #1 0x00007ffff691fad2 in ISGetIndices_Block (in=0x50ede0, idx=0x7fffffffd5e0) at /home/jed/petsc/src/vec/is/is/impls/block/block.c:47
>>>>>> #2 0x00007ffff6936468 in ISGetIndices (is=0x50ede0, ptr=0x7fffffffd5e0) at /home/jed/petsc/src/vec/is/is/interface/index.c:382
>>>>>> #3 0x00007ffff68eb66a in ISLocalToGlobalMappingCreateIS (is=0x50ede0, mapping=0x4a97a0) at /home/jed/petsc/src/vec/is/utils/isltog.c:139
>>>>>> #4 0x00007ffff713216f in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:1362
>>>>>> #5 0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>>> #6 0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>>> #7 0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>>> #8 0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>>>
>>>>>>
>>>>>> #0 PetscTrMallocDefault (a=83886080, lineno=197, function=0x7ffff7b0171f <.L__func__.VecCreate_MPI_Private> "VecCreate_MPI_Private", filename=0x7ffff7b01735 <.L.str> "/home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c", result=0x513b60) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>>> #1 0x00007ffff69e33d8 in VecCreate_MPI_Private (v=0x511c30, alloc=PETSC_TRUE, nghost=0, array=0x0) at /home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c:197
>>>>>> #2 0x00007ffff69e3f5b in VecCreate_MPI (vv=0x511c30) at /home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c:240
>>>>>> #3 0x00007ffff6a17cd1 in VecSetType (vec=0x511c30, method=0x7ffff7b01763 <.L.str6> "mpi") at /home/jed/petsc/src/vec/vec/interface/vecreg.c:53
>>>>>> #4 0x00007ffff69e4395 in VecCreate_Standard (v=0x511c30) at /home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c:267
>>>>>> #5 0x00007ffff6a17cd1 in VecSetType (vec=0x511c30, method=0x4b9b00 "standard") at /home/jed/petsc/src/vec/vec/interface/vecreg.c:53
>>>>>> #6 0x00007ffff715ad11 in DMCreateGlobalVector_DA (da=0x4a88d0, g=0x7fffffffdd20) at /home/jed/petsc/src/dm/impls/da/dadist.c:43
>>>>>> #7 0x00007ffff7370f54 in DMCreateGlobalVector (dm=0x4a88d0, vec=0x7fffffffdd20) at /home/jed/petsc/src/dm/interface/dm.c:680
>>>>>> #8 0x0000000000400e25 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:20
>>>>>>
>>>>>>
>>>>>> #0 PetscTrMallocDefault (a=83886080, lineno=197, function=0x7ffff7b0171f <.L__func__.VecCreate_MPI_Private> "VecCreate_MPI_Private", filename=0x7ffff7b01735 <.L.str> "/home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c", result=0x4c9550) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>>> #1 0x00007ffff69e33d8 in VecCreate_MPI_Private (v=0x4d0ad0, alloc=PETSC_TRUE, nghost=0, array=0x0) at /home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c:197
>>>>>> #2 0x00007ffff69e3f5b in VecCreate_MPI (vv=0x4d0ad0) at /home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c:240
>>>>>> #3 0x00007ffff6a17cd1 in VecSetType (vec=0x4d0ad0, method=0x7ffff7b01763 <.L.str6> "mpi") at /home/jed/petsc/src/vec/vec/interface/vecreg.c:53
>>>>>> #4 0x00007ffff69e4395 in VecCreate_Standard (v=0x4d0ad0) at /home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c:267
>>>>>> #5 0x00007ffff6a17cd1 in VecSetType (vec=0x4d0ad0, method=0x4b9b00 "standard") at /home/jed/petsc/src/vec/vec/interface/vecreg.c:53
>>>>>> #6 0x00007ffff715ad11 in DMCreateGlobalVector_DA (da=0x4a88d0, g=0x7fffffffdd18) at /home/jed/petsc/src/dm/impls/da/dadist.c:43
>>>>>> #7 0x00007ffff7370f54 in DMCreateGlobalVector (dm=0x4a88d0, vec=0x7fffffffdd18) at /home/jed/petsc/src/dm/interface/dm.c:680
>>>>>> #8 0x0000000000400fb6 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:25
>>>>>>
>>>>>>
>>>
>>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20140515/9eddd1d6/attachment.sig>
More information about the petsc-dev
mailing list