[petsc-dev] making DA more light weight

Barry Smith bsmith at mcs.anl.gov
Thu May 15 22:21:33 CDT 2014


   ok, adding new code for bs > 12 that uses inner loops or memcpys and clearer profiling 

with a fixed width font this is easy to read. The scatter is not terrible in actual memory usage but does possibly produce too much unrecoverable memory? The local to global is bad news IMHO but is fixable. So my theory on memory usage in DMDA was fine but the practice was too shaky.

  Haven’t pushed these changes till run complete test suite

  Barry


~/Src/petsc/src/dm/examples/tests  jed/dmda-memusage $ petscmpiexec -n 2 ./ex20 -dof 1
DMDACreate3d Scatter High water    26.079 MB increase   26.079 MB Current   12.713 MB increase   12.713 MB
DMDACreate3d         High water    43.569 MB increase   17.490 MB Current   21.493 MB increase    8.779 MB
DMCreateGlobalVector High water    51.995 MB increase    8.425 MB Current   29.883 MB increase    8.391 MB
DMCreateGlobalVector High water    60.387 MB increase    8.393 MB Current   38.274 MB increase    8.391 MB
~/Src/petsc/src/dm/examples/tests  jed/dmda-memusage $ petscmpiexec -n 2 ./ex20 -dof 2
DMDACreate3d Scatter High water    26.075 MB increase   26.075 MB Current   12.713 MB increase   12.713 MB
DMDACreate3d         High water    51.991 MB increase   25.915 MB Current   30.275 MB increase   17.562 MB
DMCreateGlobalVector High water    68.817 MB increase   16.826 MB Current   47.054 MB increase   16.779 MB
DMCreateGlobalVector High water    85.619 MB increase   16.802 MB Current   63.833 MB increase   16.779 MB
~/Src/petsc/src/dm/examples/tests  jed/dmda-memusage $ petscmpiexec -n 2 ./ex20 -dof 3
DMDACreate3d Scatter High water    26.063 MB increase   26.063 MB Current   12.713 MB increase   12.713 MB
DMDACreate3d         High water    56.271 MB increase   30.208 MB Current   34.797 MB increase   22.084 MB
DMCreateGlobalVector High water    81.474 MB increase   25.203 MB Current   59.965 MB increase   25.168 MB
DMCreateGlobalVector High water   106.656 MB increase   25.182 MB Current   85.133 MB increase   25.168 MB
~/Src/petsc/src/dm/examples/tests  jed/dmda-memusage $ petscmpiexec -n 2 ./ex20 -dof 4
DMDACreate3d Scatter High water    26.055 MB increase   26.055 MB Current   12.713 MB increase   12.713 MB
DMDACreate3d         High water    60.568 MB increase   34.513 MB Current   39.319 MB increase   26.606 MB
DMCreateGlobalVector High water    94.163 MB increase   33.595 MB Current   72.875 MB increase   33.556 MB
DMCreateGlobalVector High water   127.721 MB increase   33.559 MB Current  106.432 MB increase   33.556 MB
~/Src/petsc/src/dm/examples/tests  jed/dmda-memusage $ 


On May 15, 2014, at 10:09 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

> 
>   Ok, added #define VecScatterOptimizedBS(mbs) (2 <= mbs && mbs <= 12) for Jed’s branch and now get 
> 
> ~/Src/petsc/src/dm/examples/tests  jed/dmda-memusage $ petscmpiexec -n 2 ./ex20 
> DMDACreate3d        :   86.041 MB +   86.041 MB
> DMCreateGlobalVector:  169.980 MB +   83.939 MB
> DMCreateGlobalVector:  253.870 MB +   83.890 MB
> 
> this is without the optimization of the ISLocalToGlobalMapping.
> 
>  Still lots more to done, working away
> 
> 
>   Barry
> 
> 
> 
> 
> On May 15, 2014, at 8:13 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>> 
>> Damn, damn, damn
>> 
>> define VecScatterOptimizedBS(mbs) ((2 <= mbs && mbs <= 8) || mbs == 12)
>> 
>> In the debugger I was always picking a block size that happened to handled nicely while you picked one that is not (but could easily be added).  I figured you were in a different PETSc universe.
>> 
>> I changed the bs to 8 and added a print inside the DMDACreate3d() at the appropriate place and got
>> 
>> ~/Src/petsc/src/dm/examples/tests  jed/dmda-memusage $ petscmpiexec -n 2 ./ex20 
>> DMDACreate3d   before local to global     :   43.442 MB
>> DMDACreate3d   after local to global     :   77.562 MB
>> DMDACreate3d        :   77.578 MB +   77.578 MB
>> DMCreateGlobalVector:  144.740 MB +   67.162 MB
>> DMCreateGlobalVector:  211.857 MB +   67.117 MB
>> 
>> Note that out of the 77.6 needed for the DMDA   34 is for the local to global. If that is 34 is divided by 8 then the total needed by the DMDA is 43.4 + 4.x < 50 which is less than ONE vector. 
>> 
>>  Barry
>> 
>> 
>> 
>> On May 15, 2014, at 8:00 PM, Jed Brown <jed at jedbrown.org> wrote:
>> 
>>> Barry Smith <bsmith at mcs.anl.gov> writes:
>>> 
>>>> Damn it Jed, I already told you the sequential case does NOT have the optimization for bs > 1. Run in the debugger on two processes and put a break point in ISGetIndices() and you will see that it is not used by the VecScatterCreate() inside the DMDACreate …
>>>> 
>>>>> $ mpirun.hydra -n 2 ./ex20 -malloc_log -malloc_log_threshold 1e6
>>>>> 
>>>>> 
>>>>> If you run as above in a debugger with a break point at ISGetIndices,
>>>>> the first break is at
>>>>> 
>>>>> $ mpirun.hydra -n 1 gdb -ex 'b ISGetIndices' -ex r --args ./ex20 : -n 1  ./ex20
>>>> 
>>>> You are not running as above since you went from 2 to 1 process where completely different code is used in creating the scatter!
>>> 
>>> Barry, this really is a two-process job, despite the fact that I'm only
>>> attaching a debugger to rank 0.  Same with the debugger traces below.
>>> It's standard mpiexec syntax.
>>> 
>>> mpiexec args executable pgmargs [ : args executable pgmargs ... ]
>>> 
>>>>> [0] 3 12713984 DMSetUp_DA_3D()
>>>>> [0] 1 4259840 ISBlockSetIndices_Block()
>>>>> [0] 4 169738240 ISGetIndices_Block()
>>>> 
>>>>          This number above is NOT in the VecScatterCreate() it is in the creation of the local to global mapping
>>>> 
>>>>> [0] 1 41943040 ISGetIndices_Stride()
>>>>> [0] 1 4259840 ISLocalToGlobalMappingBlock()
>>>>> [0] 2 167772160 VecCreate_MPI_Private()
>>>>> [0] 10 339476480 VecScatterCreate_PtoS()
>>>>             ^^^^ I admit this number is larger than I would like to see but it is not caused by ISGetIndices
>>>> 
>>>> Barry
>>>> 
>>>> Likely we can just remove the sequential case from the checks in the VecScatter and always do the “parallel” and thus remove the unoptimized code that you keep tripping over.
>>>> 
>>>> 
>>>> On May 15, 2014, at 5:38 PM, Jed Brown <jed at jedbrown.org> wrote:
>>>> 
>>>>> Barry Smith <bsmith at mcs.anl.gov> writes:
>>>>> 
>>>>>>> Note that ISGetIndices is still called in the parallel case.
>>>>>> 
>>>>>> When bs is set > 1 it is not called in the VecScatterCreate()
>>>>>> inside the DMDACreate 2d/3d ! Only ISBlockGetIndices() is
>>>>>> called. The memory usage of VecScatterCreate() is O(dof*number of
>>>>>> ghost points) + O(vector length/dof) in this case
>>>>>> 
>>>>>> It is called MULTIPLY times in the ISLocalToGlobalMappingCreateIS()
>>>>>> inside the DMDACreate 2d/3d thus currently this causes small integer
>>>>>> * vector length usage. As I keep telling you this is the problem
>>>>>> area for dof > 1.  For dof == 1 I can live with 2.5 *sizeof (vector)
>>>>>> 
>>>>>> You need a better test code to run with. Can you use
>>>>>> ksp/ksp/examples/tests/ex42.c ? Or snes/examples/tutorials/ex19.c
>>>>>> in parallel?
>>>>> 
>>>>> I added a simple test for this purpose, src/dm/examples/tests/ex20.c
>>>>> in 'jed/dmda-memusage'.
>>>>> 
>>>>> $ mpirun.hydra -n 2 ./ex20 -malloc_log -malloc_log_threshold 1e6
>>>>> DMDACreate3d        :  225.866 MB +  225.866 MB
>>>>> DMCreateGlobalVector:  309.666 MB +   83.800 MB
>>>>> DMCreateGlobalVector:  393.552 MB +   83.886 MB
>>>>> [0] Maximum memory PetscMalloc()ed 386401520 maximum size of entire process 309821440
>>>>> [0] Memory usage sorted by function
>>>>> [0] 3 12713984 DMSetUp_DA_3D()
>>>>> [0] 1 4259840 ISBlockSetIndices_Block()
>>>>> [0] 4 169738240 ISGetIndices_Block()
>>>>> [0] 1 41943040 ISGetIndices_Stride()
>>>>> [0] 1 4259840 ISLocalToGlobalMappingBlock()
>>>>> [0] 2 167772160 VecCreate_MPI_Private()
>>>>> [0] 10 339476480 VecScatterCreate_PtoS()
>>>>> 
>>>>> If you run as above in a debugger with a break point at ISGetIndices,
>>>>> the first break is at
>>>>> 
>>>>> $ mpirun.hydra -n 1 gdb -ex 'b ISGetIndices' -ex r --args ./ex20 : -n 1  ./ex20
>>>>> GNU gdb (GDB) 7.7.1
>>>>> Copyright (C) 2014 Free Software Foundation, Inc.
>>>>> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
>>>>> This is free software: you are free to change and redistribute it.
>>>>> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>>>>> and "show warranty" for details.
>>>>> This GDB was configured as "x86_64-unknown-linux-gnu".
>>>>> Type "show configuration" for configuration details.
>>>>> For bug reporting instructions, please see:
>>>>> <http://www.gnu.org/software/gdb/bugs/>.
>>>>> Find the GDB manual and other documentation resources online at:
>>>>> <http://www.gnu.org/software/gdb/documentation/>.
>>>>> For help, type "help".
>>>>> Type "apropos word" to search for commands related to "word"...
>>>>> No symbol table is loaded.  Use the "file" command.
>>>>> Breakpoint 1 (err.c:343) pending.
>>>>> Reading symbols from ./ex20...done.
>>>>> Function "ISGetIndices" not defined.
>>>>> Breakpoint 2 (ISGetIndices) pending.
>>>>> Starting program: /home/jed/petsc/src/dm/examples/tests/ex20 
>>>>> warning: Could not load shared library symbols for linux-vdso.so.1.
>>>>> Do you need "set solib-search-path" or "set sysroot"?
>>>>> [Thread debugging using libthread_db enabled]
>>>>> Using host libthread_db library "/usr/lib/libthread_db.so.1".
>>>>> 
>>>>> Breakpoint 2, ISGetIndices (is=0x506840, ptr=0x7fffffffd328) at /home/jed/petsc/src/vec/is/is/interface/index.c:379
>>>>> 379       PetscFunctionBegin;
>>>>> (gdb) bt
>>>>> #0  ISGetIndices (is=0x506840, ptr=0x7fffffffd328) at /home/jed/petsc/src/vec/is/is/interface/index.c:379
>>>>> #1  0x00007ffff6a60cb9 in VecScatterCreate (xin=0x4fbfb0, ix=0x506840, yin=0x4f8d10, iy=0x503210, newctx=0x7fffffffd888) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1579
>>>>> #2  0x00007ffff7129d9b in DMSetUp_DA_3D (da=0x4e3280) at /home/jed/petsc/src/dm/impls/da/da3.c:448
>>>>> #3  0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4e3280) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>> #4  0x00007ffff736fd9f in DMSetUp (dm=0x4e3280) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>> #5  0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd58) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>> #6  0x0000000000400c96 in main (argc=1, argv=0x7fffffffde58) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>> 
>>>>> That is this code, which dates from 1995 according to git annotate.
>>>>> 
>>>>> /* general case */
>>>>> {
>>>>>   PetscInt       nx,ny;
>>>>>   const PetscInt *idx,*idy;
>>>>>   ierr = ISGetLocalSize(ix,&nx);CHKERRQ(ierr);
>>>>>   ierr = ISGetIndices(ix,&idx);CHKERRQ(ierr);
>>>>>   ierr = ISGetLocalSize(iy,&ny);CHKERRQ(ierr);
>>>>>   ierr = ISGetIndices(iy,&idy);CHKERRQ(ierr);
>>>>>   if (nx != ny) SETERRQ(PETSC_COMM_SELF,PETSC_ERR_ARG_SIZ,"Local scatter sizes don't match");
>>>>>   ierr = VecScatterCreate_StoP(nx,idx,ny,idy,xin,yin,1,ctx);CHKERRQ(ierr);
>>>>>   ierr = ISRestoreIndices(ix,&idx);CHKERRQ(ierr);
>>>>>   ierr = ISRestoreIndices(iy,&idy);CHKERRQ(ierr);
>>>>>   ierr = PetscInfo(xin,"General case: Seq to MPI\n");CHKERRQ(ierr);
>>>>>   goto functionend;
>>>>> }
>>>>> 
>>>>> 
>>>>> Here are all the large allocations, found by stepping through printing
>>>>> the stack trace for each case.
>>>>> 
>>>>> $ mpirun.hydra -n 1 gdb -ex 'b mtr.c:228' -ex r --args ./ex20 -malloc_log -malloc_log_threshold 1e7 : -n 1  ./ex20
>>>>> [...]
>>>>> 
>>>>> (gdb) bt
>>>>> #0  PetscTrMallocDefault (a=41943040, lineno=47, function=0x7ffff7afd5d4 <.L__func__.ISGetIndices_Block> "ISGetIndices_Block", filename=0x7ffff7afd53b <.L.str> "/home/jed/petsc/src/vec/is/is/impls/block/block.c", result=0x7fffffffcf88) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>> #1  0x00007ffff691fad2 in ISGetIndices_Block (in=0x4cbe90, idx=0x7fffffffd2f8) at /home/jed/petsc/src/vec/is/is/impls/block/block.c:47
>>>>> #2  0x00007ffff6936468 in ISGetIndices (is=0x4cbe90, ptr=0x7fffffffd2f8) at /home/jed/petsc/src/vec/is/is/interface/index.c:382
>>>>> #3  0x00007ffff6a60cb9 in VecScatterCreate (xin=0x4c1600, ix=0x4cbe90, yin=0x4be360, iy=0x4c8860, newctx=0x7fffffffd858) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1579
>>>>> #4  0x00007ffff7129d9b in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:448
>>>>> #5  0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>> #6  0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>> #7  0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>> #8  0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>> 
>>>>> 
>>>>> #0  PetscTrMallocDefault (a=41943040, lineno=160, function=0x7ffff7afd32b <.L__func__.ISGetIndices_Stride> "ISGetIndices_Stride", filename=0x7ffff7afd279 <.L.str> "/home/jed/petsc/src/vec/is/is/impls/stride/stride.c", result=0x7fffffffd2f0) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>> #1  0x00007ffff691b1e9 in ISGetIndices_Stride (in=0x4c8860, idx=0x7fffffffd2f0) at /home/jed/petsc/src/vec/is/is/impls/stride/stride.c:160
>>>>> #2  0x00007ffff6936468 in ISGetIndices (is=0x4c8860, ptr=0x7fffffffd2f0) at /home/jed/petsc/src/vec/is/is/interface/index.c:382
>>>>> #3  0x00007ffff6a60da7 in VecScatterCreate (xin=0x4c1600, ix=0x4cbe90, yin=0x4be360, iy=0x4c8860, newctx=0x7fffffffd858) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1581
>>>>> #4  0x00007ffff7129d9b in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:448
>>>>> #5  0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>> #6  0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>> #7  0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>> #8  0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>> 
>>>>> 
>>>>> #0  PetscTrMallocDefault (a=41943040, lineno=1745, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x7fffffffcee0) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>> #1  0x00007ffff6a9129f in VecScatterCreate_PtoS (nx=10485760, inidx=0x7fffe9086650, ny=10485760, inidy=0x7fffeb887650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1745
>>>>> #2  0x00007ffff6a94c2d in VecScatterCreate_StoP (nx=10485760, inidx=0x7fffeb887650, ny=10485760, inidy=0x7fffe9086650, xin=0x4c1600, yin=0x4be360, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:2197
>>>>> #3  0x00007ffff6a60ea0 in VecScatterCreate (xin=0x4c1600, ix=0x4cbe90, yin=0x4be360, iy=0x4c8860, newctx=0x7fffffffd858) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1583
>>>>> #4  0x00007ffff7129d9b in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:448
>>>>> #5  0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>> #6  0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>> #7  0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>> #8  0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>> 
>>>>> 
>>>>> #0  PetscTrMallocDefault (a=41943040, lineno=1784, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x7fffffffcec0) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>> #1  0x00007ffff6a91c5f in VecScatterCreate_PtoS (nx=10485760, inidx=0x7fffe9086650, ny=10485760, inidy=0x7fffeb887650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1784
>>>>> #2  0x00007ffff6a94c2d in VecScatterCreate_StoP (nx=10485760, inidx=0x7fffeb887650, ny=10485760, inidy=0x7fffe9086650, xin=0x4c1600, yin=0x4be360, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:2197
>>>>> #3  0x00007ffff6a60ea0 in VecScatterCreate (xin=0x4c1600, ix=0x4cbe90, yin=0x4be360, iy=0x4c8860, newctx=0x7fffffffd858) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1583
>>>>> #4  0x00007ffff7129d9b in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:448
>>>>> #5  0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>> #6  0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>> #7  0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>> #8  0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>> 
>>>>> 
>>>>> #0  PetscTrMallocDefault (a=41943040, lineno=1879, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x50fae0) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>> #1  0x00007ffff6a9423b in VecScatterCreate_PtoS (nx=10485760, inidx=0x7fffe9086650, ny=10485760, inidy=0x7fffeb887650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1879
>>>>> #2  0x00007ffff6a94c2d in VecScatterCreate_StoP (nx=10485760, inidx=0x7fffeb887650, ny=10485760, inidy=0x7fffe9086650, xin=0x4c1600, yin=0x4be360, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:2197
>>>>> #3  0x00007ffff6a60ea0 in VecScatterCreate (xin=0x4c1600, ix=0x4cbe90, yin=0x4be360, iy=0x4c8860, newctx=0x7fffffffd858) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1583
>>>>> #4  0x00007ffff7129d9b in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:448
>>>>> #5  0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>> #6  0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>> #7  0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>> #8  0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>> 
>>>>> 
>>>>> #0  PetscTrMallocDefault (a=41943040, lineno=1880, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x510860) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>> #1  0x00007ffff6a9431f in VecScatterCreate_PtoS (nx=10485760, inidx=0x7fffe9086650, ny=10485760, inidy=0x7fffeb887650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1880
>>>>> #2  0x00007ffff6a94c2d in VecScatterCreate_StoP (nx=10485760, inidx=0x7fffeb887650, ny=10485760, inidy=0x7fffe9086650, xin=0x4c1600, yin=0x4be360, bs=1, ctx=0x4d2ea0) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:2197
>>>>> #3  0x00007ffff6a60ea0 in VecScatterCreate (xin=0x4c1600, ix=0x4cbe90, yin=0x4be360, iy=0x4c8860, newctx=0x7fffffffd858) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1583
>>>>> #4  0x00007ffff7129d9b in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:448
>>>>> #5  0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>> #6  0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>> #7  0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>> #8  0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>> 
>>>>> 
>>>>> #0  PetscTrMallocDefault (a=42598400, lineno=47, function=0x7ffff7afd5d4 <.L__func__.ISGetIndices_Block> "ISGetIndices_Block", filename=0x7ffff7afd53b <.L.str> "/home/jed/petsc/src/vec/is/is/impls/block/block.c", result=0x7fffffffcf88) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>> #1  0x00007ffff691fad2 in ISGetIndices_Block (in=0x4cf6a0, idx=0x7fffffffd370) at /home/jed/petsc/src/vec/is/is/impls/block/block.c:47
>>>>> #2  0x00007ffff6936468 in ISGetIndices (is=0x4cf6a0, ptr=0x7fffffffd370) at /home/jed/petsc/src/vec/is/is/interface/index.c:382
>>>>> #3  0x00007ffff6a5f84a in VecScatterCreate (xin=0x4be360, ix=0x4cf6a0, yin=0x4c1600, iy=0x50ede0, newctx=0x7fffffffd850) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1493
>>>>> #4  0x00007ffff712ed23 in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:1033
>>>>> #5  0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>> #6  0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>> #7  0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>> #8  0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>> 
>>>>> 
>>>>> #0  PetscTrMallocDefault (a=42598400, lineno=47, function=0x7ffff7afd5d4 <.L__func__.ISGetIndices_Block> "ISGetIndices_Block", filename=0x7ffff7afd53b <.L.str> "/home/jed/petsc/src/vec/is/is/impls/block/block.c", result=0x7fffffffcf88) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>> #1  0x00007ffff691fad2 in ISGetIndices_Block (in=0x50ede0, idx=0x7fffffffd368) at /home/jed/petsc/src/vec/is/is/impls/block/block.c:47
>>>>> #2  0x00007ffff6936468 in ISGetIndices (is=0x50ede0, ptr=0x7fffffffd368) at /home/jed/petsc/src/vec/is/is/interface/index.c:382
>>>>> #3  0x00007ffff6a5f938 in VecScatterCreate (xin=0x4be360, ix=0x4cf6a0, yin=0x4c1600, iy=0x50ede0, newctx=0x7fffffffd850) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1495
>>>>> #4  0x00007ffff712ed23 in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:1033
>>>>> #5  0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>> #6  0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>> #7  0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>> #8  0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>> 
>>>>> 
>>>>> #0  PetscTrMallocDefault (a=42598400, lineno=1745, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x7fffffffcf80) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>> #1  0x00007ffff6a9129f in VecScatterCreate_PtoS (nx=10649600, inidx=0x7fffeafb5650, ny=10649600, inidy=0x7fffe17e3650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x514f70) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1745
>>>>> #2  0x00007ffff6a5fa49 in VecScatterCreate (xin=0x4be360, ix=0x4cf6a0, yin=0x4c1600, iy=0x50ede0, newctx=0x7fffffffd850) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1497
>>>>> #3  0x00007ffff712ed23 in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:1033
>>>>> #4  0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>> #5  0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>> #6  0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>> #7  0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>> 
>>>>> 
>>>>> #0  PetscTrMallocDefault (a=42598400, lineno=1784, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x7fffffffcf60) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>> #1  0x00007ffff6a91c5f in VecScatterCreate_PtoS (nx=10649600, inidx=0x7fffeafb5650, ny=10649600, inidy=0x7fffe17e3650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x514f70) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1784
>>>>> #2  0x00007ffff6a5fa49 in VecScatterCreate (xin=0x4be360, ix=0x4cf6a0, yin=0x4c1600, iy=0x50ede0, newctx=0x7fffffffd850) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1497
>>>>> #3  0x00007ffff712ed23 in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:1033
>>>>> #4  0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>> #5  0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>> #6  0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>> #7  0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>> 
>>>>> 
>>>>> #0  PetscTrMallocDefault (a=41943040, lineno=1879, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x5b9850) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>> #1  0x00007ffff6a9423b in VecScatterCreate_PtoS (nx=10649600, inidx=0x7fffeafb5650, ny=10649600, inidy=0x7fffe17e3650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x514f70) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1879
>>>>> #2  0x00007ffff6a5fa49 in VecScatterCreate (xin=0x4be360, ix=0x4cf6a0, yin=0x4c1600, iy=0x50ede0, newctx=0x7fffffffd850) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1497
>>>>> #3  0x00007ffff712ed23 in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:1033
>>>>> #4  0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>> #5  0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>> #6  0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>> #7  0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>> 
>>>>> 
>>>>> #0  PetscTrMallocDefault (a=41943040, lineno=1880, function=0x7ffff7b0449a <.L__func__.VecScatterCreate_PtoS> "VecScatterCreate_PtoS", filename=0x7ffff7b03f29 <.L.str> "/home/jed/petsc/src/vec/vec/utils/vpscat.c", result=0x516ba0) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>> #1  0x00007ffff6a9431f in VecScatterCreate_PtoS (nx=10649600, inidx=0x7fffeafb5650, ny=10649600, inidy=0x7fffe17e3650, xin=0x4be360, yin=0x4c1600, bs=1, ctx=0x514f70) at /home/jed/petsc/src/vec/vec/utils/vpscat.c:1880
>>>>> #2  0x00007ffff6a5fa49 in VecScatterCreate (xin=0x4be360, ix=0x4cf6a0, yin=0x4c1600, iy=0x50ede0, newctx=0x7fffffffd850) at /home/jed/petsc/src/vec/vec/utils/vscat.c:1497
>>>>> #3  0x00007ffff712ed23 in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:1033
>>>>> #4  0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>> #5  0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>> #6  0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>> #7  0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>> 
>>>>> 
>>>>> #0  PetscTrMallocDefault (a=42598400, lineno=47, function=0x7ffff7afd5d4 <.L__func__.ISGetIndices_Block> "ISGetIndices_Block", filename=0x7ffff7afd53b <.L.str> "/home/jed/petsc/src/vec/is/is/impls/block/block.c", result=0x7fffffffd4b8) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>> #1  0x00007ffff691fad2 in ISGetIndices_Block (in=0x50ede0, idx=0x7fffffffd5e0) at /home/jed/petsc/src/vec/is/is/impls/block/block.c:47
>>>>> #2  0x00007ffff6936468 in ISGetIndices (is=0x50ede0, ptr=0x7fffffffd5e0) at /home/jed/petsc/src/vec/is/is/interface/index.c:382
>>>>> #3  0x00007ffff68eb66a in ISLocalToGlobalMappingCreateIS (is=0x50ede0, mapping=0x4a97a0) at /home/jed/petsc/src/vec/is/utils/isltog.c:139
>>>>> #4  0x00007ffff713216f in DMSetUp_DA_3D (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/da3.c:1362
>>>>> #5  0x00007ffff71a3ef5 in DMSetUp_DA (da=0x4a88d0) at /home/jed/petsc/src/dm/impls/da/dareg.c:27
>>>>> #6  0x00007ffff736fd9f in DMSetUp (dm=0x4a88d0) at /home/jed/petsc/src/dm/interface/dm.c:559
>>>>> #7  0x00007ffff71330b1 in DMDACreate3d (comm=1140850688, bx=DM_BOUNDARY_NONE, by=DM_BOUNDARY_NONE, bz=DM_BOUNDARY_NONE, stencil_type=DMDA_STENCIL_STAR, M=-128, N=-128, P=-128, m=-1, n=-1, p=-1, dof=10, s=1, lx=0x0, ly=0x0, lz=0x0, da=0x7fffffffdd28) at /home/jed/petsc/src/dm/impls/da/da3.c:1463
>>>>> #8  0x0000000000400c96 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:15
>>>>> 
>>>>> 
>>>>> #0  PetscTrMallocDefault (a=83886080, lineno=197, function=0x7ffff7b0171f <.L__func__.VecCreate_MPI_Private> "VecCreate_MPI_Private", filename=0x7ffff7b01735 <.L.str> "/home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c", result=0x513b60) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>> #1  0x00007ffff69e33d8 in VecCreate_MPI_Private (v=0x511c30, alloc=PETSC_TRUE, nghost=0, array=0x0) at /home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c:197
>>>>> #2  0x00007ffff69e3f5b in VecCreate_MPI (vv=0x511c30) at /home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c:240
>>>>> #3  0x00007ffff6a17cd1 in VecSetType (vec=0x511c30, method=0x7ffff7b01763 <.L.str6> "mpi") at /home/jed/petsc/src/vec/vec/interface/vecreg.c:53
>>>>> #4  0x00007ffff69e4395 in VecCreate_Standard (v=0x511c30) at /home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c:267
>>>>> #5  0x00007ffff6a17cd1 in VecSetType (vec=0x511c30, method=0x4b9b00 "standard") at /home/jed/petsc/src/vec/vec/interface/vecreg.c:53
>>>>> #6  0x00007ffff715ad11 in DMCreateGlobalVector_DA (da=0x4a88d0, g=0x7fffffffdd20) at /home/jed/petsc/src/dm/impls/da/dadist.c:43
>>>>> #7  0x00007ffff7370f54 in DMCreateGlobalVector (dm=0x4a88d0, vec=0x7fffffffdd20) at /home/jed/petsc/src/dm/interface/dm.c:680
>>>>> #8  0x0000000000400e25 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:20
>>>>> 
>>>>> 
>>>>> #0  PetscTrMallocDefault (a=83886080, lineno=197, function=0x7ffff7b0171f <.L__func__.VecCreate_MPI_Private> "VecCreate_MPI_Private", filename=0x7ffff7b01735 <.L.str> "/home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c", result=0x4c9550) at /home/jed/petsc/src/sys/memory/mtr.c:228
>>>>> #1  0x00007ffff69e33d8 in VecCreate_MPI_Private (v=0x4d0ad0, alloc=PETSC_TRUE, nghost=0, array=0x0) at /home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c:197
>>>>> #2  0x00007ffff69e3f5b in VecCreate_MPI (vv=0x4d0ad0) at /home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c:240
>>>>> #3  0x00007ffff6a17cd1 in VecSetType (vec=0x4d0ad0, method=0x7ffff7b01763 <.L.str6> "mpi") at /home/jed/petsc/src/vec/vec/interface/vecreg.c:53
>>>>> #4  0x00007ffff69e4395 in VecCreate_Standard (v=0x4d0ad0) at /home/jed/petsc/src/vec/vec/impls/mpi/pbvec.c:267
>>>>> #5  0x00007ffff6a17cd1 in VecSetType (vec=0x4d0ad0, method=0x4b9b00 "standard") at /home/jed/petsc/src/vec/vec/interface/vecreg.c:53
>>>>> #6  0x00007ffff715ad11 in DMCreateGlobalVector_DA (da=0x4a88d0, g=0x7fffffffdd18) at /home/jed/petsc/src/dm/impls/da/dadist.c:43
>>>>> #7  0x00007ffff7370f54 in DMCreateGlobalVector (dm=0x4a88d0, vec=0x7fffffffdd18) at /home/jed/petsc/src/dm/interface/dm.c:680
>>>>> #8  0x0000000000400fb6 in main (argc=4, argv=0x7fffffffde28) at /home/jed/petsc/src/dm/examples/tests/ex20.c:25
>>>>> 
>>>>> 
>> 
> 




More information about the petsc-dev mailing list