[petsc-users] SNESComputeJacobianDefaultColor use too much memory

Tue Feb 16 23:49:13 CST 2016

Hi Barry,

I run the code with valgrind on a workstation for a smaller case and it 
produces some ASCII information (see attached).  Is this helpful?

Best,
Rongliang

On 02/17/2016 01:30 PM, Barry Smith wrote:
>    Hmm, something didn't work right with massif. I should give a bunch of ASCII information about how much memory is used at different times in the code. You may need to play around with the massif options and google documentation on how to get it to provide useful information. Once it produces the useful information it will be very helpful.
>
> time=0
> mem_heap_B=0
> mem_heap_extra_B=0
> mem_stacks_B=0
> heap_tree=empty
>
>
>> On Feb 16, 2016, at 9:44 PM, Rongliang Chen <rongliang.chan at gmail.com> wrote:
>>
>> Dear Barry,
>>
>> Many thanks for your reply.
>>
>> I checked with the valgrind and did not obtain any outputs (massif.out.<pid>) because the job was killed before it reached the end.
>>
>> Then I switch to a smaller case, it works well and one of the output is attached (I did not find any useful information in it). The output with the option -mat_coloring_view is followed, which shows that the number of colors is 65. Any ideas for this?
>>
>> MatColoring Object: 480 MPI processes
>>    type: sl
>>    Weight type: RANDOM
>>    Distance 2, Max. Colors 65535
>>    Number of colors 65
>>    Number of total columns 1637350
>>
>> Best regards,
>> Rongliang
>>
>> On 02/17/2016 01:13 AM, Barry Smith wrote:
>>>    How many colors are needed?
>>>
>>>    You need to produce a breakdown of where all the memory is being used. For example valgrind with the
>>> http://valgrind.org/docs/manual/ms-manual.html
>>>
>>>
>>>    Barry
>>>
>>>
>>>> On Feb 16, 2016, at 6:45 AM, Rongliang Chen <rongliang.chan at gmail.com>
>>>>   wrote:
>>>>
>>>> Dear all,
>>>>
>>>> I am using the DMPlex to solve a PDE on a unstructured mesh and I use the SNESComputeJacobianDefaultColor to compute the Jacobian matrix.
>>>>
>>>> My code works well for small problems (such as problem with 3.3x10^5 cells using 120 cores) but when I increase the number of cells (2.6x10^6 cells using 1920 cores), I go out of memory in the function MatColoringApply. It shows that one of the cores uses over 11G memory. I think this is unreasonable. Do you have any suggestions for debugging this problem?
>>>>
>>>> Best regards,
>>>> Rongliang
>>>>
>> <massif.out.12562>

-------------- next part --------------
desc: (none)
cmd: ./Nwtun -malloc off -f ../meshes/2DNACA0012AIRFOIL_Hex50x30x10x2.exo -overlap 1 -options_file options.txt -time_integral_method 1 -initial_time 0.0 -final_time 0.5 -dt 1.e-3 -timestep 1 -max_time_its 0 -includeenergy -reconstruct -PressureFlux -testcase 2DNACA -symmetricbc -output_solution -solutionfile results/2DNACAImp -steps_output 2 -inflow_u 349.02 -viscosity 0.01 -k 0.026 -R 287.02 -adiabatic 1.4 -Euler -Attack 10 -Mach 0.8 -fd_jacobian_color -monitor_step_gap -monitormass -parallel_csr -parallel_output -limiter zero -RiemannSolver Rusanov -CFL 0.9 -reconstruct -second_order -DiffFuncForJac -mat_coloring_view
time_unit: i
#-----------
snapshot=0
#-----------
time=0
mem_heap_B=0
mem_heap_extra_B=0
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=1
#-----------
time=466351390
mem_heap_B=5106022
mem_heap_extra_B=13618
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=2
#-----------
time=746348011
mem_heap_B=5897454
mem_heap_extra_B=14138
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=3
#-----------
time=1208542856
mem_heap_B=9098558
mem_heap_extra_B=14114
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=4
#-----------
time=1585087744
mem_heap_B=6260499
mem_heap_extra_B=14581
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=5
#-----------
time=1923773721
mem_heap_B=6299991
mem_heap_extra_B=14641
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=6
#-----------
time=2304638724
mem_heap_B=30905886
mem_heap_extra_B=12810
mem_stacks_B=0
heap_tree=peak
n2: 30905886 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n8: 30364925 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 24893576 0x9832FA: DMPlexGetAdjacency_Internal (plexdistribute.c:220)
   n1: 24893576 0x9839A6: DMPlexGetAdjacency (plexdistribute.c:263)
    n1: 24893576 0x554357: DMPlexEnlargePartitionNwtun (plexpartition.c:507)
     n1: 24893576 0x555C98: DMPlexCreatePartitionNwtun (plexpartition.c:669)
      n1: 24893576 0x556B5F: DMPlexDistributeNwtun (plexdistribute.c:68)
       n1: 24893576 0x4154E6: CreateMesh (SetupMeshes.c:34)
        n0: 24893576 0x4076DF: main (Nwtun.c:55)
  n0: 1033045 in 104 places, all below massif's threshold (01.00%)
  n1: 855200 0x462A1D: DMSetUp_Plex (plex.c:1582)
   n2: 855200 0x50B88B: DMSetUp (dm.c:560)
    n1: 855200 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229)
     n1: 855200 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 855200 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 855200 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 855200 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 855200 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
  n1: 855200 0x462AB1: DMSetUp_Plex (plex.c:1583)
   n2: 855200 0x50B88B: DMSetUp (dm.c:560)
    n1: 855200 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229)
     n1: 855200 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 855200 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 855200 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 855200 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 855200 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
  n2: 855200 0x46371F: DMPlexSymmetrize (plex.c:1655)
   n1: 855200 0x9E4D06: DMPlexInterpolateFaces_Internal (plexinterpolate.c:303)
    n1: 855200 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
     n1: 855200 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
      n1: 855200 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 855200 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 855200 0x4076DF: main (Nwtun.c:55)
   n0: 0 in 1 place, below massif's threshold (01.00%)
  n1: 710752 0x52EBD7: PetscSectionSetChart (vsectionis.c:389)
   n0: 710752 in 6 places, all below massif's threshold (01.00%)
  n1: 710752 0x52EC32: PetscSectionSetChart (vsectionis.c:389)
   n0: 710752 in 6 places, all below massif's threshold (01.00%)
  n1: 451200 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132)
   n1: 451200 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
    n1: 451200 0x4152C9: CreateMesh (SetupMeshes.c:23)
     n0: 451200 0x4076DF: main (Nwtun.c:55)
 n0: 540961 in 134 places, all below massif's threshold (01.00%)
#-----------
snapshot=7
#-----------
time=2801678661
mem_heap_B=6355101
mem_heap_extra_B=11195
mem_stacks_B=0
heap_tree=detailed
n5: 6355101 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n11: 5823604 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 855200 0x462A1D: DMSetUp_Plex (plex.c:1582)
   n2: 855200 0x50B88B: DMSetUp (dm.c:560)
    n1: 855200 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229)
     n1: 855200 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 855200 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 855200 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 855200 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 855200 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
  n1: 855200 0x462AB1: DMSetUp_Plex (plex.c:1583)
   n2: 855200 0x50B88B: DMSetUp (dm.c:560)
    n1: 855200 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229)
     n1: 855200 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 855200 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 855200 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 855200 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 855200 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
  n2: 855200 0x46371F: DMPlexSymmetrize (plex.c:1655)
   n1: 855200 0x9E4D06: DMPlexInterpolateFaces_Internal (plexinterpolate.c:303)
    n1: 855200 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
     n1: 855200 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
      n1: 855200 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 855200 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 855200 0x4076DF: main (Nwtun.c:55)
   n0: 0 in 1 place, below massif's threshold (01.00%)
  n3: 710848 0x52EBD7: PetscSectionSetChart (vsectionis.c:389)
   n2: 308000 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 308000 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 308000 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 308000 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 308000 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 308000 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 308000 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
   n2: 308000 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 308000 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 308000 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 308000 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 308000 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 308000 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 308000 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
   n0: 94848 in 5 places, all below massif's threshold (01.00%)
  n3: 710848 0x52EC32: PetscSectionSetChart (vsectionis.c:389)
   n2: 308000 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 308000 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 308000 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 308000 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 308000 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 308000 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 308000 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
   n2: 308000 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 308000 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 308000 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 308000 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 308000 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 308000 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 308000 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
   n0: 94848 in 5 places, all below massif's threshold (01.00%)
  n1: 473944 0x56B8A6: PetscSegBufferAlloc_Private (segbuffer.c:33)
   n1: 473944 0x56BFCC: PetscSegBufferGet (segbuffer.c:102)
    n2: 473944 0x97A8D6: PetscSegBufferGetInts (petscsys.h:2626)
     n1: 409728 0x98071B: DMPlexCreatePartitionClosure (plexpartition.c:580)
      n1: 409728 0x557178: DMPlexDistributeNwtun (plexdistribute.c:92)
       n1: 409728 0x4154E6: CreateMesh (SetupMeshes.c:34)
        n0: 409728 0x4076DF: main (Nwtun.c:55)
     n1: 64216 0x9805AF: DMPlexCreatePartitionClosure (plexpartition.c:573)
      n1: 64216 0x557178: DMPlexDistributeNwtun (plexdistribute.c:92)
       n1: 64216 0x4154E6: CreateMesh (SetupMeshes.c:34)
        n0: 64216 0x4076DF: main (Nwtun.c:55)
  n1: 451200 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132)
   n1: 451200 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
    n1: 451200 0x4152C9: CreateMesh (SetupMeshes.c:23)
     n0: 451200 0x4076DF: main (Nwtun.c:55)
  n2: 308000 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46)
   n2: 308000 0x4E2DE5: DMLabelGetStratumIS (plexlabel.c:571)
    n1: 308000 0x46481D: DMPlexStratify (plex.c:1768)
     n1: 308000 0x9E4D5E: DMPlexInterpolateFaces_Internal (plexinterpolate.c:304)
      n1: 308000 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
       n1: 308000 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
        n1: 308000 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
         n1: 308000 0x4152C9: CreateMesh (SetupMeshes.c:23)
          n0: 308000 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
   n0: 0 in 1 place, below massif's threshold (01.00%)
  n3: 288120 0x50FE4C: DMGetWorkArray (dm.c:1005)
   n2: 192080 0x45FFD3: DMPlexGetTransitiveClosure (plex.c:1291)
    n1: 153664 0x466336: DMPlexGetFullJoin (plex.c:1946)
     n1: 153664 0x445DE9: DMPlexCreateExodusNwtun (plexexodusii.c:372)
      n1: 153664 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 153664 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 153664 0x4076DF: main (Nwtun.c:55)
    n0: 38416 in 1 place, below massif's threshold (01.00%)
   n1: 76832 0x45FF3D: DMPlexGetTransitiveClosure (plex.c:1287)
    n0: 76832 in 2 places, all below massif's threshold (01.00%)
   n0: 19208 in 4 places, all below massif's threshold (01.00%)
  n0: 173204 in 114 places, all below massif's threshold (01.00%)
  n1: 141840 0xC99701: VecCreate_MPI_Private (pbvec.c:201)
   n1: 141840 0xC99C3D: VecCreate_MPI (pbvec.c:244)
    n1: 141840 0x6BF560: VecSetType (vecreg.c:53)
     n1: 141840 0xC99FA5: VecCreate_Standard (pbvec.c:271)
      n1: 141840 0x6BF560: VecSetType (vecreg.c:53)
       n1: 141840 0x44533A: DMPlexCreateExodusNwtun (plexexodusii.c:314)
        n1: 141840 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
         n1: 141840 0x4152C9: CreateMesh (SetupMeshes.c:23)
          n0: 141840 0x4076DF: main (Nwtun.c:55)
 n1: 180480 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202)
  n1: 180480 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
   n1: 180480 0x4152C9: CreateMesh (SetupMeshes.c:23)
    n0: 180480 0x4076DF: main (Nwtun.c:55)
 n1: 161240 0x4DBEB6: kh_resize_HASHI (hash.h:510)
  n2: 161240 0x4DC2CD: kh_put_HASHI (hash.h:510)
   n2: 161240 0x4E14DD: DMLabelSetValue (plexlabel.c:450)
    n1: 161240 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909)
     n0: 161240 in 3 places, all below massif's threshold (01.00%)
    n0: 0 in 3 places, all below massif's threshold (01.00%)
   n0: 0 in 1 place, below massif's threshold (01.00%)
 n0: 109157 in 131 places, all below massif's threshold (01.00%)
 n1: 80620 0x4DBE90: kh_resize_HASHI (hash.h:510)
  n2: 80620 0x4DC2CD: kh_put_HASHI (hash.h:510)
   n2: 80620 0x4E14DD: DMLabelSetValue (plexlabel.c:450)
    n1: 80620 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909)
     n0: 80620 in 3 places, all below massif's threshold (01.00%)
    n0: 0 in 3 places, all below massif's threshold (01.00%)
   n0: 0 in 1 place, below massif's threshold (01.00%)
#-----------
snapshot=8
#-----------
time=3336978286
mem_heap_B=8295339
mem_heap_extra_B=12405
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=9
#-----------
time=3820713760
mem_heap_B=1630326
mem_heap_extra_B=10986
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=10
#-----------
time=4287539798
mem_heap_B=1686266
mem_heap_extra_B=13262
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=11
#-----------
time=4618200211
mem_heap_B=1884316
mem_heap_extra_B=31492
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=12
#-----------
time=5199848578
mem_heap_B=1893829
mem_heap_extra_B=33355
mem_stacks_B=0
heap_tree=detailed
n6: 1893829 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n12: 1292450 0x57877E: PetscMallocAlign (mal.c:27)
  n0: 232010 in 224 places, all below massif's threshold (01.00%)
  n1: 155712 0xC9786A: VecCreate_Seq (bvec3.c:38)
   n1: 155712 0x6BF560: VecSetType (vecreg.c:53)
    n1: 155712 0xC99F47: VecCreate_Standard (pbvec.c:269)
     n1: 155712 0x6BF560: VecSetType (vecreg.c:53)
      n1: 155712 0xA0773A: DMCreateLocalVector_Section_Private (dmi.c:65)
       n1: 155712 0x976C9B: DMCreateLocalVector_Plex (plexcreate.c:931)
        n2: 155712 0x50D220: DMCreateLocalVector (dm.c:709)
         n1: 130752 0x417EA6: ConstructGeometryFVM (SetupMeshes.c:306)
          n0: 130752 0x407898: main (Nwtun.c:60)
         n1: 24960 0x4178BF: ConstructGeometryFVM (SetupMeshes.c:273)
          n0: 24960 0x407898: main (Nwtun.c:60)
  n3: 153664 0x50FE4C: DMGetWorkArray (dm.c:1005)
   n2: 76832 0x45FF3D: DMPlexGetTransitiveClosure (plex.c:1287)
    n2: 76832 0x47B16A: DMPlexVecGetClosure (plex.c:4614)
     n1: 38416 0x44F2E0: DMPlexComputeGeometryFVM_3D_Internal (plexgeometry.c:872)
      n1: 38416 0x44FF85: DMPlexComputeCellGeometryFVM (plexgeometry.c:977)
       n1: 38416 0x417A90: ConstructGeometryFVM (SetupMeshes.c:281)
        n0: 38416 0x407898: main (Nwtun.c:60)
     n1: 38416 0x44E4E6: DMPlexComputeGeometryFVM_2D_Internal (plexgeometry.c:793)
      n1: 38416 0x44FF11: DMPlexComputeCellGeometryFVM (plexgeometry.c:974)
       n1: 38416 0x4180BC: ConstructGeometryFVM (SetupMeshes.c:324)
        n0: 38416 0x407898: main (Nwtun.c:60)
    n0: 0 in 2 places, all below massif's threshold (01.00%)
   n2: 76832 0x45FFD3: DMPlexGetTransitiveClosure (plex.c:1291)
    n2: 76832 0x47B16A: DMPlexVecGetClosure (plex.c:4614)
     n1: 38416 0x44F2E0: DMPlexComputeGeometryFVM_3D_Internal (plexgeometry.c:872)
      n1: 38416 0x44FF85: DMPlexComputeCellGeometryFVM (plexgeometry.c:977)
       n1: 38416 0x417A90: ConstructGeometryFVM (SetupMeshes.c:281)
        n0: 38416 0x407898: main (Nwtun.c:60)
     n1: 38416 0x44E4E6: DMPlexComputeGeometryFVM_2D_Internal (plexgeometry.c:793)
      n1: 38416 0x44FF11: DMPlexComputeCellGeometryFVM (plexgeometry.c:974)
       n1: 38416 0x4180BC: ConstructGeometryFVM (SetupMeshes.c:324)
        n0: 38416 0x407898: main (Nwtun.c:60)
    n0: 0 in 2 places, all below massif's threshold (01.00%)
   n0: 0 in 4 places, all below massif's threshold (01.00%)
  n4: 135920 0x52EC32: PetscSectionSetChart (vsectionis.c:389)
   n0: 41064 in 12 places, all below massif's threshold (01.00%)
   n2: 35464 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 35464 0x4B39F0: DMPlexShiftSizes_Internal (plexsubmesh.c:189)
     n1: 35464 0x4B8A41: DMPlexConstructGhostCells_Internal (plexsubmesh.c:507)
      n1: 35464 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
       n1: 35464 0x415800: CreateMesh (SetupMeshes.c:53)
        n0: 35464 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 2 places, all below massif's threshold (01.00%)
   n2: 35464 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 35464 0x4B39F0: DMPlexShiftSizes_Internal (plexsubmesh.c:189)
     n1: 35464 0x4B8A41: DMPlexConstructGhostCells_Internal (plexsubmesh.c:507)
      n1: 35464 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
       n1: 35464 0x415800: CreateMesh (SetupMeshes.c:53)
        n0: 35464 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 2 places, all below massif's threshold (01.00%)
   n1: 23928 0x52ED49: PetscSectionSetChart (vsectionis.c:392)
    n0: 23928 in 4 places, all below massif's threshold (01.00%)
  n4: 135920 0x52EBD7: PetscSectionSetChart (vsectionis.c:389)
   n0: 41064 in 12 places, all below massif's threshold (01.00%)
   n2: 35464 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 35464 0x4B39F0: DMPlexShiftSizes_Internal (plexsubmesh.c:189)
     n1: 35464 0x4B8A41: DMPlexConstructGhostCells_Internal (plexsubmesh.c:507)
      n1: 35464 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
       n1: 35464 0x415800: CreateMesh (SetupMeshes.c:53)
        n0: 35464 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 2 places, all below massif's threshold (01.00%)
   n2: 35464 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 35464 0x4B39F0: DMPlexShiftSizes_Internal (plexsubmesh.c:189)
     n1: 35464 0x4B8A41: DMPlexConstructGhostCells_Internal (plexsubmesh.c:507)
      n1: 35464 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
       n1: 35464 0x415800: CreateMesh (SetupMeshes.c:53)
        n0: 35464 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 2 places, all below massif's threshold (01.00%)
   n1: 23928 0x52ED49: PetscSectionSetChart (vsectionis.c:392)
    n0: 23928 in 4 places, all below massif's threshold (01.00%)
  n2: 116064 0x659AFA: ISCreate (isreg.c:41)
   n2: 116064 0x64D670: ISCreateGeneral (general.c:632)
    n2: 116064 0x4E222E: DMLabelGetValueIS (plexlabel.c:518)
     n2: 116064 0x4E8069: DMPlexGetLabelIdIS (plexlabel.c:1013)
      n1: 116064 0x41D333: SetUpSymetricBoundaryMaching (SetupMeshes.c:1030)
       n0: 116064 0x40796B: main (Nwtun.c:65)
      n0: 0 in 1 place, below massif's threshold (01.00%)
     n0: 0 in 2 places, all below massif's threshold (01.00%)
    n0: 0 in 4 places, all below massif's threshold (01.00%)
   n0: 0 in 1 place, below massif's threshold (01.00%)
  n1: 90944 0x462A1D: DMSetUp_Plex (plex.c:1582)
   n2: 90944 0x50B88B: DMSetUp (dm.c:560)
    n1: 90944 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534)
     n1: 90944 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
      n1: 90944 0x415800: CreateMesh (SetupMeshes.c:53)
       n0: 90944 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 3 places, all below massif's threshold (01.00%)
  n1: 90944 0x462C3D: DMSetUp_Plex (plex.c:1587)
   n1: 90944 0x50B88B: DMSetUp (dm.c:560)
    n1: 90944 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534)
     n1: 90944 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
      n1: 90944 0x415800: CreateMesh (SetupMeshes.c:53)
       n0: 90944 0x4076DF: main (Nwtun.c:55)
  n1: 90944 0x462AB1: DMSetUp_Plex (plex.c:1583)
   n2: 90944 0x50B88B: DMSetUp (dm.c:560)
    n1: 90944 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534)
     n1: 90944 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
      n1: 90944 0x415800: CreateMesh (SetupMeshes.c:53)
       n0: 90944 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 3 places, all below massif's threshold (01.00%)
  n2: 38728 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46)
   n2: 35464 0x4E2DE5: DMLabelGetStratumIS (plexlabel.c:571)
    n2: 35464 0x46481D: DMPlexStratify (plex.c:1768)
     n1: 35464 0x4B955D: DMPlexConstructGhostCells_Internal (plexsubmesh.c:560)
      n1: 35464 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
       n1: 35464 0x415800: CreateMesh (SetupMeshes.c:53)
        n0: 35464 0x4076DF: main (Nwtun.c:55)
     n0: 0 in 2 places, all below massif's threshold (01.00%)
    n0: 0 in 1 place, below massif's threshold (01.00%)
   n0: 3264 in 3 places, all below massif's threshold (01.00%)
  n2: 26784 0x659BA0: ISCreate (isreg.c:41)
   n2: 26784 0x64D670: ISCreateGeneral (general.c:632)
    n2: 26784 0x4E222E: DMLabelGetValueIS (plexlabel.c:518)
     n2: 26784 0x4E8069: DMPlexGetLabelIdIS (plexlabel.c:1013)
      n1: 26784 0x41D333: SetUpSymetricBoundaryMaching (SetupMeshes.c:1030)
       n0: 26784 0x40796B: main (Nwtun.c:65)
      n0: 0 in 1 place, below massif's threshold (01.00%)
     n0: 0 in 2 places, all below massif's threshold (01.00%)
    n0: 0 in 4 places, all below massif's threshold (01.00%)
   n0: 0 in 1 place, below massif's threshold (01.00%)
  n2: 24816 0x505DF4: DMCreate (dm.c:43)
   n1: 20680 0x5066C4: DMClone (dm.c:99)
    n0: 20680 in 4 places, all below massif's threshold (01.00%)
   n0: 4136 in 4 places, all below massif's threshold (01.00%)
 n2: 266240 0x5E920BC: MPIU_Handle_obj_alloc_unsafe (in /home/rlchen/mpich2/lib/libmpich.so.3.0)
  n1: 192512 0x5EA339B: MPID_Isend (in /home/rlchen/mpich2/lib/libmpich.so.3.0)
   n1: 192512 0x5E9CBA9: PMPI_Isend (in /home/rlchen/mpich2/lib/libmpich.so.3.0)
    n1: 192512 0x11532F6: PetscCommBuildTwoSided_Allreduce (mpits.c:174)
     n1: 192512 0x11538F7: PetscCommBuildTwoSided (mpits.c:237)
      n1: 192512 0xC8603E: PetscSFSetUp_Basic (sfbasic.c:552)
       n1: 192512 0x67F6D7: PetscSFSetUp (sf.c:192)
        n1: 192512 0x689A32: PetscSFFetchAndOpBegin (sf.c:1138)
         n1: 192512 0x53DD1C: PetscSFConvertPartition (vsectionis.c:1916)
          n1: 192512 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101)
           n1: 192512 0x4154E6: CreateMesh (SetupMeshes.c:34)
            n0: 192512 0x4076DF: main (Nwtun.c:55)
  n1: 73728 0x5EAFDD0: MPID_Type_dup (in /home/rlchen/mpich2/lib/libmpich.so.3.0)
   n1: 73728 0x5EFFF19: PMPI_Type_dup (in /home/rlchen/mpich2/lib/libmpich.so.3.0)
    n1: 73728 0xC8750A: PetscSFBasicPackTypeSetup (sfbasic.c:654)
     n1: 73728 0xC88D96: PetscSFBasicGetPack (sfbasic.c:778)
      n1: 73728 0xC8B6AF: PetscSFReduceBegin_Basic (sfbasic.c:973)
       n1: 73728 0x68814F: PetscSFReduceBegin (sf.c:993)
        n1: 73728 0x682011: PetscSFCreateInverseSF (sf.c:459)
         n1: 73728 0x53E180: PetscSFConvertPartition (vsectionis.c:1937)
          n1: 73728 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101)
           n1: 73728 0x4154E6: CreateMesh (SetupMeshes.c:34)
            n0: 73728 0x4076DF: main (Nwtun.c:55)
 n1: 180480 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202)
  n1: 180480 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
   n1: 180480 0x4152C9: CreateMesh (SetupMeshes.c:23)
    n0: 180480 0x4076DF: main (Nwtun.c:55)
 n0: 102947 in 141 places, all below massif's threshold (01.00%)
 n1: 30088 0x443EEF: DMPlexCreateExodusNwtun (plexexodusii.c:201)
  n1: 30088 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
   n1: 30088 0x4152C9: CreateMesh (SetupMeshes.c:23)
    n0: 30088 0x4076DF: main (Nwtun.c:55)
 n1: 21624 0x4DBEB6: kh_resize_HASHI (hash.h:510)
  n2: 21624 0x4DC2CD: kh_put_HASHI (hash.h:510)
   n1: 21624 0x4E14DD: DMLabelSetValue (plexlabel.c:450)
    n0: 21624 in 8 places, all below massif's threshold (01.00%)
   n0: 0 in 1 place, below massif's threshold (01.00%)
#-----------
snapshot=13
#-----------
time=5707788994
mem_heap_B=3019566
mem_heap_extra_B=38722
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=14
#-----------
time=6349483701
mem_heap_B=4209659
mem_heap_extra_B=40661
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=15
#-----------
time=6766057848
mem_heap_B=5309276
mem_heap_extra_B=42020
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=16
#-----------
time=7358988205
mem_heap_B=5309276
mem_heap_extra_B=42020
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=17
#-----------
time=7800734843
mem_heap_B=5309252
mem_heap_extra_B=42028
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=18
#-----------
time=8095696846
mem_heap_B=5309252
mem_heap_extra_B=42028
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=19
#-----------
time=8536311900
mem_heap_B=5309276
mem_heap_extra_B=42020
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=20
#-----------
time=9120156697
mem_heap_B=5309276
mem_heap_extra_B=42020
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=21
#-----------
time=9685257975
mem_heap_B=5309276
mem_heap_extra_B=42020
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=22
#-----------
time=10085942687
mem_heap_B=5309284
mem_heap_extra_B=42044
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=23
#-----------
time=10490188517
mem_heap_B=5309276
mem_heap_extra_B=42020
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=24
#-----------
time=10889161638
mem_heap_B=5309252
mem_heap_extra_B=42028
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=25
#-----------
time=11158478616
mem_heap_B=5309252
mem_heap_extra_B=42028
mem_stacks_B=0
heap_tree=detailed
n4: 5309252 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n21: 4707353 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 712000 0x8518E1: MatFDColoringSetUpBlocked_AIJ_Private (fdaij.c:121)
   n1: 712000 0x919037: MatFDColoringSetUp_MPIXAIJ (fdmpiaij.c:570)
    n1: 712000 0x74CD34: MatFDColoringSetUp (fdmatrix.c:250)
     n1: 712000 0x43E33A: SNESComputeJacobianDefaultColorNwtun (SetupJacobian.c:907)
      n1: 712000 0x434942: FormJacobian (SetupJacobian.c:56)
       n1: 712000 0xA8F850: SNESComputeJacobian (snes.c:2193)
        n1: 712000 0xAF7C15: SNESSolve_NEWTONLS (ls.c:230)
         n1: 712000 0xA9E3CC: SNESSolve (snes.c:3743)
          n1: 712000 0x410469: SolveTimeDependent (Nwtun.c:806)
           n0: 712000 0x40BC62: main (Nwtun.c:417)
  n0: 520457 in 389 places, all below massif's threshold (01.00%)
  n1: 408112 0xC9786A: VecCreate_Seq (bvec3.c:38)
   n2: 408112 0x6BF560: VecSetType (vecreg.c:53)
    n1: 405312 0xC99F47: VecCreate_Standard (pbvec.c:269)
     n1: 405312 0x6BF560: VecSetType (vecreg.c:53)
      n1: 405312 0xA0773A: DMCreateLocalVector_Section_Private (dmi.c:65)
       n1: 405312 0x976C9B: DMCreateLocalVector_Plex (plexcreate.c:931)
        n3: 405312 0x50D220: DMCreateLocalVector (dm.c:709)
         n3: 249600 0xA035E0: DMGetLocalVector (dmget.c:49)
          n1: 93600 0x425614: CaculateLocalFunction_LS (SetupFunctions.c:742)
           n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
            n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52)
             n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033)
              n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
               n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743)
                n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806)
                 n0: 93600 0x40BC62: main (Nwtun.c:417)
          n1: 93600 0x4257D8: CaculateLocalFunction_LS (SetupFunctions.c:748)
           n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
            n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52)
             n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033)
              n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
               n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743)
                n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806)
                 n0: 93600 0x40BC62: main (Nwtun.c:417)
          n0: 62400 in 2 places, all below massif's threshold (01.00%)
         n1: 130752 0x417EA6: ConstructGeometryFVM (SetupMeshes.c:306)
          n0: 130752 0x407898: main (Nwtun.c:60)
         n0: 24960 in 1 place, below massif's threshold (01.00%)
    n0: 2800 in 1 place, below massif's threshold (01.00%)
  n1: 362400 0x8512B1: MatFDColoringSetUpBlocked_AIJ_Private (fdaij.c:70)
   n1: 362400 0x919037: MatFDColoringSetUp_MPIXAIJ (fdmpiaij.c:570)
    n1: 362400 0x74CD34: MatFDColoringSetUp (fdmatrix.c:250)
     n1: 362400 0x43E33A: SNESComputeJacobianDefaultColorNwtun (SetupJacobian.c:907)
      n1: 362400 0x434942: FormJacobian (SetupJacobian.c:56)
       n1: 362400 0xA8F850: SNESComputeJacobian (snes.c:2193)
        n1: 362400 0xAF7C15: SNESSolve_NEWTONLS (ls.c:230)
         n1: 362400 0xA9E3CC: SNESSolve (snes.c:3743)
          n1: 362400 0x410469: SolveTimeDependent (Nwtun.c:806)
           n0: 362400 0x40BC62: main (Nwtun.c:417)
  n1: 356000 0x81EE1C: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719)
   n2: 356000 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664)
    n1: 340000 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255)
     n1: 340000 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959)
      n1: 340000 0x955310: MatXAIJSetPreallocation (gcreate.c:252)
       n1: 340000 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435)
        n1: 340000 0x459619: DMCreateMatrix_Plex (plex.c:615)
         n1: 340000 0x50F42F: DMCreateMatrix (dm.c:941)
          n0: 340000 0x40B906: main (Nwtun.c:396)
    n0: 16000 in 2 places, all below massif's threshold (01.00%)
  n1: 356000 0x81EDCA: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719)
   n2: 356000 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664)
    n1: 340000 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255)
     n1: 340000 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959)
      n1: 340000 0x955310: MatXAIJSetPreallocation (gcreate.c:252)
       n1: 340000 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435)
        n1: 340000 0x459619: DMCreateMatrix_Plex (plex.c:615)
         n1: 340000 0x50F42F: DMCreateMatrix (dm.c:941)
          n0: 340000 0x40B906: main (Nwtun.c:396)
    n0: 16000 in 2 places, all below massif's threshold (01.00%)
  n2: 233704 0xC99701: VecCreate_MPI_Private (pbvec.c:201)
   n1: 137064 0xC99C3D: VecCreate_MPI (pbvec.c:244)
    n1: 137064 0x6BF560: VecSetType (vecreg.c:53)
     n1: 137064 0xC99FA5: VecCreate_Standard (pbvec.c:271)
      n2: 137064 0x6BF560: VecSetType (vecreg.c:53)
       n1: 109360 0xA07131: DMCreateGlobalVector_Section_Private (dmi.c:34)
        n1: 109360 0x976932: DMCreateGlobalVector_Plex (plexcreate.c:917)
         n2: 109360 0x50CE38: DMCreateGlobalVector (dm.c:681)
          n2: 97280 0xA04005: DMGetGlobalVector (dmget.c:154)
           n1: 85200 0x426EC6: ConstructCellCentriodGradient (SetupFunctions.c:855)
            n1: 85200 0x42054B: FormTimeStepFunction (SetupFunctions.c:233)
             n1: 85200 0x41E3AB: FormFunction (SetupFunctions.c:52)
              n1: 85200 0xA8D3FC: SNESComputeFunction (snes.c:2033)
               n1: 85200 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
                n1: 85200 0xA9E3CC: SNESSolve (snes.c:3743)
                 n1: 85200 0x410469: SolveTimeDependent (Nwtun.c:806)
                  n0: 85200 0x40BC62: main (Nwtun.c:417)
           n0: 12080 in 1 place, below massif's threshold (01.00%)
          n0: 12080 in 1 place, below massif's threshold (01.00%)
       n0: 27704 in 4 places, all below massif's threshold (01.00%)
   n1: 96640 0xC98DCF: VecDuplicate_MPI (pbvec.c:68)
    n1: 96640 0x6B04CF: VecDuplicate (vector.c:399)
     n0: 96640 in 8 places, all below massif's threshold (01.00%)
  n1: 216960 0x521E45: DMCreateDefaultSF (dm.c:3075)
   n1: 216960 0x520D8A: DMGetDefaultSF (dm.c:2991)
    n2: 216960 0x5153DB: DMGlobalToLocalBegin (dm.c:1649)
     n1: 187200 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
      n1: 187200 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
       n1: 187200 0x41E3AB: FormFunction (SetupFunctions.c:52)
        n1: 187200 0xA8D3FC: SNESComputeFunction (snes.c:2033)
         n1: 187200 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
          n1: 187200 0xA9E3CC: SNESSolve (snes.c:3743)
           n1: 187200 0x410469: SolveTimeDependent (Nwtun.c:806)
            n0: 187200 0x40BC62: main (Nwtun.c:417)
     n0: 29760 in 1 place, below massif's threshold (01.00%)
  n1: 154640 0x52EBD7: PetscSectionSetChart (vsectionis.c:389)
   n0: 154640 in 19 places, all below massif's threshold (01.00%)
  n1: 154640 0x52EC32: PetscSectionSetChart (vsectionis.c:389)
   n0: 154640 in 19 places, all below massif's threshold (01.00%)
  n3: 153664 0x50FE4C: DMGetWorkArray (dm.c:1005)
   n2: 76832 0x45FF3D: DMPlexGetTransitiveClosure (plex.c:1287)
    n1: 76832 0x47B16A: DMPlexVecGetClosure (plex.c:4614)
     n0: 76832 in 2 places, all below massif's threshold (01.00%)
    n0: 0 in 2 places, all below massif's threshold (01.00%)
   n2: 76832 0x45FFD3: DMPlexGetTransitiveClosure (plex.c:1291)
    n1: 76832 0x47B16A: DMPlexVecGetClosure (plex.c:4614)
     n0: 76832 in 2 places, all below massif's threshold (01.00%)
    n0: 0 in 2 places, all below massif's threshold (01.00%)
   n0: 0 in 4 places, all below massif's threshold (01.00%)
  n2: 117480 0x6812ED: PetscSFSetGraph (sf.c:349)
   n1: 108480 0x522622: DMCreateDefaultSF (dm.c:3116)
    n1: 108480 0x520D8A: DMGetDefaultSF (dm.c:2991)
     n2: 108480 0x5153DB: DMGlobalToLocalBegin (dm.c:1649)
      n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
       n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
        n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52)
         n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033)
          n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
           n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743)
            n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806)
             n0: 93600 0x40BC62: main (Nwtun.c:417)
      n0: 14880 in 1 place, below massif's threshold (01.00%)
   n0: 9000 in 9 places, all below massif's threshold (01.00%)
  n2: 117480 0x681299: PetscSFSetGraph (sf.c:349)
   n1: 108480 0x522622: DMCreateDefaultSF (dm.c:3116)
    n1: 108480 0x520D8A: DMGetDefaultSF (dm.c:2991)
     n2: 108480 0x5153DB: DMGlobalToLocalBegin (dm.c:1649)
      n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
       n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
        n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52)
         n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033)
          n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
           n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743)
            n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806)
             n0: 93600 0x40BC62: main (Nwtun.c:417)
      n0: 14880 in 1 place, below massif's threshold (01.00%)
   n0: 9000 in 9 places, all below massif's threshold (01.00%)
  n2: 117480 0xC88EE6: PetscSFBasicGetPack (sfbasic.c:779)
   n1: 117480 0xC8AA7F: PetscSFBcastBegin_Basic (sfbasic.c:914)
    n2: 117480 0x687407: PetscSFBcastBegin (sf.c:926)
     n2: 108480 0x51554F: DMGlobalToLocalBegin (dm.c:1656)
      n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
       n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
        n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52)
         n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033)
          n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
           n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743)
            n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806)
             n0: 93600 0x40BC62: main (Nwtun.c:417)
      n0: 14880 in 1 place, below massif's threshold (01.00%)
     n0: 9000 in 8 places, all below massif's threshold (01.00%)
   n0: 0 in 1 place, below massif's threshold (01.00%)
  n2: 116064 0x659AFA: ISCreate (isreg.c:41)
   n2: 116064 0x64D670: ISCreateGeneral (general.c:632)
    n2: 116064 0x4E222E: DMLabelGetValueIS (plexlabel.c:518)
     n2: 116064 0x4E8069: DMPlexGetLabelIdIS (plexlabel.c:1013)
      n1: 116064 0x41D333: SetUpSymetricBoundaryMaching (SetupMeshes.c:1030)
       n0: 116064 0x40796B: main (Nwtun.c:65)
      n0: 0 in 2 places, all below massif's threshold (01.00%)
     n0: 0 in 2 places, all below massif's threshold (01.00%)
    n0: 0 in 6 places, all below massif's threshold (01.00%)
   n0: 0 in 1 place, below massif's threshold (01.00%)
  n2: 114480 0xC88E56: PetscSFBasicGetPack (sfbasic.c:779)
   n1: 114480 0xC8AA7F: PetscSFBcastBegin_Basic (sfbasic.c:914)
    n2: 114480 0x687407: PetscSFBcastBegin (sf.c:926)
     n2: 108800 0x51554F: DMGlobalToLocalBegin (dm.c:1656)
      n1: 93840 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
       n1: 93840 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
        n1: 93840 0x41E3AB: FormFunction (SetupFunctions.c:52)
         n1: 93840 0xA8D3FC: SNESComputeFunction (snes.c:2033)
          n1: 93840 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
           n1: 93840 0xA9E3CC: SNESSolve (snes.c:3743)
            n1: 93840 0x410469: SolveTimeDependent (Nwtun.c:806)
             n0: 93840 0x40BC62: main (Nwtun.c:417)
      n0: 14960 in 1 place, below massif's threshold (01.00%)
     n0: 5680 in 8 places, all below massif's threshold (01.00%)
   n0: 0 in 1 place, below massif's threshold (01.00%)
  n1: 114480 0xC861ED: PetscSFSetUp_Basic (sfbasic.c:557)
   n2: 114480 0x67F6D7: PetscSFSetUp (sf.c:192)
    n2: 114480 0x68739A: PetscSFBcastBegin (sf.c:925)
     n2: 108800 0x51554F: DMGlobalToLocalBegin (dm.c:1656)
      n1: 93840 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
       n1: 93840 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
        n1: 93840 0x41E3AB: FormFunction (SetupFunctions.c:52)
         n1: 93840 0xA8D3FC: SNESComputeFunction (snes.c:2033)
          n1: 93840 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
           n1: 93840 0xA9E3CC: SNESSolve (snes.c:3743)
            n1: 93840 0x410469: SolveTimeDependent (Nwtun.c:806)
             n0: 93840 0x40BC62: main (Nwtun.c:417)
      n0: 14960 in 1 place, below massif's threshold (01.00%)
     n0: 5680 in 7 places, all below massif's threshold (01.00%)
    n0: 0 in 2 places, all below massif's threshold (01.00%)
  n1: 108480 0x521DAD: DMCreateDefaultSF (dm.c:3074)
   n1: 108480 0x520D8A: DMGetDefaultSF (dm.c:2991)
    n2: 108480 0x5153DB: DMGlobalToLocalBegin (dm.c:1649)
     n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
      n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
       n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52)
        n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033)
         n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
          n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743)
           n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806)
            n0: 93600 0x40BC62: main (Nwtun.c:417)
     n0: 14880 in 1 place, below massif's threshold (01.00%)
  n1: 90944 0x462C3D: DMSetUp_Plex (plex.c:1587)
   n1: 90944 0x50B88B: DMSetUp (dm.c:560)
    n1: 90944 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534)
     n1: 90944 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
      n1: 90944 0x415800: CreateMesh (SetupMeshes.c:53)
       n0: 90944 0x4076DF: main (Nwtun.c:55)
  n1: 90944 0x462AB1: DMSetUp_Plex (plex.c:1583)
   n2: 90944 0x50B88B: DMSetUp (dm.c:560)
    n1: 90944 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534)
     n1: 90944 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
      n1: 90944 0x415800: CreateMesh (SetupMeshes.c:53)
       n0: 90944 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 3 places, all below massif's threshold (01.00%)
  n1: 90944 0x462A1D: DMSetUp_Plex (plex.c:1582)
   n2: 90944 0x50B88B: DMSetUp (dm.c:560)
    n1: 90944 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534)
     n1: 90944 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
      n1: 90944 0x415800: CreateMesh (SetupMeshes.c:53)
       n0: 90944 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 3 places, all below massif's threshold (01.00%)
 n2: 266240 0x5E920BC: MPIU_Handle_obj_alloc_unsafe (in /home/rlchen/mpich2/lib/libmpich.so.3.0)
  n1: 192512 0x5EA339B: MPID_Isend (in /home/rlchen/mpich2/lib/libmpich.so.3.0)
   n1: 192512 0x5E9CBA9: PMPI_Isend (in /home/rlchen/mpich2/lib/libmpich.so.3.0)
    n1: 192512 0x11532F6: PetscCommBuildTwoSided_Allreduce (mpits.c:174)
     n1: 192512 0x11538F7: PetscCommBuildTwoSided (mpits.c:237)
      n1: 192512 0xC8603E: PetscSFSetUp_Basic (sfbasic.c:552)
       n1: 192512 0x67F6D7: PetscSFSetUp (sf.c:192)
        n1: 192512 0x689A32: PetscSFFetchAndOpBegin (sf.c:1138)
         n1: 192512 0x53DD1C: PetscSFConvertPartition (vsectionis.c:1916)
          n1: 192512 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101)
           n1: 192512 0x4154E6: CreateMesh (SetupMeshes.c:34)
            n0: 192512 0x4076DF: main (Nwtun.c:55)
  n1: 73728 0x5EAFDD0: MPID_Type_dup (in /home/rlchen/mpich2/lib/libmpich.so.3.0)
   n1: 73728 0x5EFFF19: PMPI_Type_dup (in /home/rlchen/mpich2/lib/libmpich.so.3.0)
    n1: 73728 0xC8750A: PetscSFBasicPackTypeSetup (sfbasic.c:654)
     n1: 73728 0xC88D96: PetscSFBasicGetPack (sfbasic.c:778)
      n1: 73728 0xC8B6AF: PetscSFReduceBegin_Basic (sfbasic.c:973)
       n1: 73728 0x68814F: PetscSFReduceBegin (sf.c:993)
        n1: 73728 0x682011: PetscSFCreateInverseSF (sf.c:459)
         n1: 73728 0x53E180: PetscSFConvertPartition (vsectionis.c:1937)
          n1: 73728 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101)
           n1: 73728 0x4154E6: CreateMesh (SetupMeshes.c:34)
            n0: 73728 0x4076DF: main (Nwtun.c:55)
 n1: 180480 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202)
  n1: 180480 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
   n1: 180480 0x4152C9: CreateMesh (SetupMeshes.c:23)
    n0: 180480 0x4076DF: main (Nwtun.c:55)
 n0: 155179 in 150 places, all below massif's threshold (01.00%)
#-----------
snapshot=26
#-----------
time=11692400418
mem_heap_B=5309252
mem_heap_extra_B=42028
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=27
#-----------
time=11960769989
mem_heap_B=5309252
mem_heap_extra_B=42028
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=28
#-----------
time=12362682614
mem_heap_B=5309276
mem_heap_extra_B=42020
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=29
#-----------
time=12896123403
mem_heap_B=5309276
mem_heap_extra_B=42020
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=30
#-----------
time=13162816658
mem_heap_B=5309276
mem_heap_extra_B=42020
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=31
#-----------
time=13572082155
mem_heap_B=5309252
mem_heap_extra_B=42028
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=32
#-----------
time=13854485968
mem_heap_B=5310218
mem_heap_extra_B=42094
mem_stacks_B=0
heap_tree=detailed
n4: 5310218 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n21: 4708335 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 712000 0x8518E1: MatFDColoringSetUpBlocked_AIJ_Private (fdaij.c:121)
   n1: 712000 0x919037: MatFDColoringSetUp_MPIXAIJ (fdmpiaij.c:570)
    n1: 712000 0x74CD34: MatFDColoringSetUp (fdmatrix.c:250)
     n1: 712000 0x43E33A: SNESComputeJacobianDefaultColorNwtun (SetupJacobian.c:907)
      n1: 712000 0x434942: FormJacobian (SetupJacobian.c:56)
       n1: 712000 0xA8F850: SNESComputeJacobian (snes.c:2193)
        n1: 712000 0xAF7C15: SNESSolve_NEWTONLS (ls.c:230)
         n1: 712000 0xA9E3CC: SNESSolve (snes.c:3743)
          n1: 712000 0x410469: SolveTimeDependent (Nwtun.c:806)
           n0: 712000 0x40BC62: main (Nwtun.c:417)
  n0: 520815 in 389 places, all below massif's threshold (01.00%)
  n1: 408112 0xC9786A: VecCreate_Seq (bvec3.c:38)
   n2: 408112 0x6BF560: VecSetType (vecreg.c:53)
    n1: 405312 0xC99F47: VecCreate_Standard (pbvec.c:269)
     n1: 405312 0x6BF560: VecSetType (vecreg.c:53)
      n1: 405312 0xA0773A: DMCreateLocalVector_Section_Private (dmi.c:65)
       n1: 405312 0x976C9B: DMCreateLocalVector_Plex (plexcreate.c:931)
        n3: 405312 0x50D220: DMCreateLocalVector (dm.c:709)
         n3: 249600 0xA035E0: DMGetLocalVector (dmget.c:49)
          n1: 93600 0x425614: CaculateLocalFunction_LS (SetupFunctions.c:742)
           n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
            n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52)
             n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033)
              n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
               n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743)
                n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806)
                 n0: 93600 0x40BC62: main (Nwtun.c:417)
          n1: 93600 0x4257D8: CaculateLocalFunction_LS (SetupFunctions.c:748)
           n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
            n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52)
             n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033)
              n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
               n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743)
                n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806)
                 n0: 93600 0x40BC62: main (Nwtun.c:417)
          n0: 62400 in 2 places, all below massif's threshold (01.00%)
         n1: 130752 0x417EA6: ConstructGeometryFVM (SetupMeshes.c:306)
          n0: 130752 0x407898: main (Nwtun.c:60)
         n0: 24960 in 1 place, below massif's threshold (01.00%)
    n0: 2800 in 1 place, below massif's threshold (01.00%)
  n1: 362400 0x8512B1: MatFDColoringSetUpBlocked_AIJ_Private (fdaij.c:70)
   n1: 362400 0x919037: MatFDColoringSetUp_MPIXAIJ (fdmpiaij.c:570)
    n1: 362400 0x74CD34: MatFDColoringSetUp (fdmatrix.c:250)
     n1: 362400 0x43E33A: SNESComputeJacobianDefaultColorNwtun (SetupJacobian.c:907)
      n1: 362400 0x434942: FormJacobian (SetupJacobian.c:56)
       n1: 362400 0xA8F850: SNESComputeJacobian (snes.c:2193)
        n1: 362400 0xAF7C15: SNESSolve_NEWTONLS (ls.c:230)
         n1: 362400 0xA9E3CC: SNESSolve (snes.c:3743)
          n1: 362400 0x410469: SolveTimeDependent (Nwtun.c:806)
           n0: 362400 0x40BC62: main (Nwtun.c:417)
  n1: 356000 0x81EE1C: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719)
   n2: 356000 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664)
    n1: 340000 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255)
     n1: 340000 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959)
      n1: 340000 0x955310: MatXAIJSetPreallocation (gcreate.c:252)
       n1: 340000 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435)
        n1: 340000 0x459619: DMCreateMatrix_Plex (plex.c:615)
         n1: 340000 0x50F42F: DMCreateMatrix (dm.c:941)
          n0: 340000 0x40B906: main (Nwtun.c:396)
    n0: 16000 in 2 places, all below massif's threshold (01.00%)
  n1: 356000 0x81EDCA: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719)
   n2: 356000 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664)
    n1: 340000 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255)
     n1: 340000 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959)
      n1: 340000 0x955310: MatXAIJSetPreallocation (gcreate.c:252)
       n1: 340000 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435)
        n1: 340000 0x459619: DMCreateMatrix_Plex (plex.c:615)
         n1: 340000 0x50F42F: DMCreateMatrix (dm.c:941)
          n0: 340000 0x40B906: main (Nwtun.c:396)
    n0: 16000 in 2 places, all below massif's threshold (01.00%)
  n2: 233704 0xC99701: VecCreate_MPI_Private (pbvec.c:201)
   n1: 137064 0xC99C3D: VecCreate_MPI (pbvec.c:244)
    n1: 137064 0x6BF560: VecSetType (vecreg.c:53)
     n1: 137064 0xC99FA5: VecCreate_Standard (pbvec.c:271)
      n2: 137064 0x6BF560: VecSetType (vecreg.c:53)
       n1: 109360 0xA07131: DMCreateGlobalVector_Section_Private (dmi.c:34)
        n1: 109360 0x976932: DMCreateGlobalVector_Plex (plexcreate.c:917)
         n2: 109360 0x50CE38: DMCreateGlobalVector (dm.c:681)
          n2: 97280 0xA04005: DMGetGlobalVector (dmget.c:154)
           n1: 85200 0x426EC6: ConstructCellCentriodGradient (SetupFunctions.c:855)
            n1: 85200 0x42054B: FormTimeStepFunction (SetupFunctions.c:233)
             n1: 85200 0x41E3AB: FormFunction (SetupFunctions.c:52)
              n1: 85200 0xA8D3FC: SNESComputeFunction (snes.c:2033)
               n1: 85200 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
                n1: 85200 0xA9E3CC: SNESSolve (snes.c:3743)
                 n1: 85200 0x410469: SolveTimeDependent (Nwtun.c:806)
                  n0: 85200 0x40BC62: main (Nwtun.c:417)
           n0: 12080 in 1 place, below massif's threshold (01.00%)
          n0: 12080 in 1 place, below massif's threshold (01.00%)
       n0: 27704 in 4 places, all below massif's threshold (01.00%)
   n1: 96640 0xC98DCF: VecDuplicate_MPI (pbvec.c:68)
    n1: 96640 0x6B04CF: VecDuplicate (vector.c:399)
     n0: 96640 in 8 places, all below massif's threshold (01.00%)
  n1: 216960 0x521E45: DMCreateDefaultSF (dm.c:3075)
   n1: 216960 0x520D8A: DMGetDefaultSF (dm.c:2991)
    n2: 216960 0x5153DB: DMGlobalToLocalBegin (dm.c:1649)
     n1: 187200 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
      n1: 187200 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
       n1: 187200 0x41E3AB: FormFunction (SetupFunctions.c:52)
        n1: 187200 0xA8D3FC: SNESComputeFunction (snes.c:2033)
         n1: 187200 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
          n1: 187200 0xA9E3CC: SNESSolve (snes.c:3743)
           n1: 187200 0x410469: SolveTimeDependent (Nwtun.c:806)
            n0: 187200 0x40BC62: main (Nwtun.c:417)
     n0: 29760 in 1 place, below massif's threshold (01.00%)
  n1: 154640 0x52EBD7: PetscSectionSetChart (vsectionis.c:389)
   n0: 154640 in 19 places, all below massif's threshold (01.00%)
  n1: 154640 0x52EC32: PetscSectionSetChart (vsectionis.c:389)
   n0: 154640 in 19 places, all below massif's threshold (01.00%)
  n3: 153664 0x50FE4C: DMGetWorkArray (dm.c:1005)
   n2: 76832 0x45FF3D: DMPlexGetTransitiveClosure (plex.c:1287)
    n1: 76832 0x47B16A: DMPlexVecGetClosure (plex.c:4614)
     n0: 76832 in 2 places, all below massif's threshold (01.00%)
    n0: 0 in 2 places, all below massif's threshold (01.00%)
   n2: 76832 0x45FFD3: DMPlexGetTransitiveClosure (plex.c:1291)
    n1: 76832 0x47B16A: DMPlexVecGetClosure (plex.c:4614)
     n0: 76832 in 2 places, all below massif's threshold (01.00%)
    n0: 0 in 2 places, all below massif's threshold (01.00%)
   n0: 0 in 4 places, all below massif's threshold (01.00%)
  n2: 117480 0x6812ED: PetscSFSetGraph (sf.c:349)
   n1: 108480 0x522622: DMCreateDefaultSF (dm.c:3116)
    n1: 108480 0x520D8A: DMGetDefaultSF (dm.c:2991)
     n2: 108480 0x5153DB: DMGlobalToLocalBegin (dm.c:1649)
      n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
       n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
        n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52)
         n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033)
          n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
           n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743)
            n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806)
             n0: 93600 0x40BC62: main (Nwtun.c:417)
      n0: 14880 in 1 place, below massif's threshold (01.00%)
   n0: 9000 in 9 places, all below massif's threshold (01.00%)
  n2: 117480 0x681299: PetscSFSetGraph (sf.c:349)
   n1: 108480 0x522622: DMCreateDefaultSF (dm.c:3116)
    n1: 108480 0x520D8A: DMGetDefaultSF (dm.c:2991)
     n2: 108480 0x5153DB: DMGlobalToLocalBegin (dm.c:1649)
      n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
       n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
        n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52)
         n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033)
          n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
           n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743)
            n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806)
             n0: 93600 0x40BC62: main (Nwtun.c:417)
      n0: 14880 in 1 place, below massif's threshold (01.00%)
   n0: 9000 in 9 places, all below massif's threshold (01.00%)
  n2: 117480 0xC88EE6: PetscSFBasicGetPack (sfbasic.c:779)
   n1: 117480 0xC8AA7F: PetscSFBcastBegin_Basic (sfbasic.c:914)
    n2: 117480 0x687407: PetscSFBcastBegin (sf.c:926)
     n2: 108480 0x51554F: DMGlobalToLocalBegin (dm.c:1656)
      n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
       n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
        n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52)
         n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033)
          n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
           n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743)
            n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806)
             n0: 93600 0x40BC62: main (Nwtun.c:417)
      n0: 14880 in 1 place, below massif's threshold (01.00%)
     n0: 9000 in 8 places, all below massif's threshold (01.00%)
   n0: 0 in 1 place, below massif's threshold (01.00%)
  n2: 116688 0x659AFA: ISCreate (isreg.c:41)
   n2: 116688 0x64D670: ISCreateGeneral (general.c:632)
    n2: 116688 0x4E222E: DMLabelGetValueIS (plexlabel.c:518)
     n2: 116688 0x4E8069: DMPlexGetLabelIdIS (plexlabel.c:1013)
      n1: 116064 0x41D333: SetUpSymetricBoundaryMaching (SetupMeshes.c:1030)
       n0: 116064 0x40796B: main (Nwtun.c:65)
      n0: 624 in 2 places, all below massif's threshold (01.00%)
     n0: 0 in 2 places, all below massif's threshold (01.00%)
    n0: 0 in 6 places, all below massif's threshold (01.00%)
   n0: 0 in 1 place, below massif's threshold (01.00%)
  n2: 114480 0xC88E56: PetscSFBasicGetPack (sfbasic.c:779)
   n1: 114480 0xC8AA7F: PetscSFBcastBegin_Basic (sfbasic.c:914)
    n2: 114480 0x687407: PetscSFBcastBegin (sf.c:926)
     n2: 108800 0x51554F: DMGlobalToLocalBegin (dm.c:1656)
      n1: 93840 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
       n1: 93840 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
        n1: 93840 0x41E3AB: FormFunction (SetupFunctions.c:52)
         n1: 93840 0xA8D3FC: SNESComputeFunction (snes.c:2033)
          n1: 93840 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
           n1: 93840 0xA9E3CC: SNESSolve (snes.c:3743)
            n1: 93840 0x410469: SolveTimeDependent (Nwtun.c:806)
             n0: 93840 0x40BC62: main (Nwtun.c:417)
      n0: 14960 in 1 place, below massif's threshold (01.00%)
     n0: 5680 in 8 places, all below massif's threshold (01.00%)
   n0: 0 in 1 place, below massif's threshold (01.00%)
  n1: 114480 0xC861ED: PetscSFSetUp_Basic (sfbasic.c:557)
   n2: 114480 0x67F6D7: PetscSFSetUp (sf.c:192)
    n2: 114480 0x68739A: PetscSFBcastBegin (sf.c:925)
     n2: 108800 0x51554F: DMGlobalToLocalBegin (dm.c:1656)
      n1: 93840 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
       n1: 93840 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
        n1: 93840 0x41E3AB: FormFunction (SetupFunctions.c:52)
         n1: 93840 0xA8D3FC: SNESComputeFunction (snes.c:2033)
          n1: 93840 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
           n1: 93840 0xA9E3CC: SNESSolve (snes.c:3743)
            n1: 93840 0x410469: SolveTimeDependent (Nwtun.c:806)
             n0: 93840 0x40BC62: main (Nwtun.c:417)
      n0: 14960 in 1 place, below massif's threshold (01.00%)
     n0: 5680 in 7 places, all below massif's threshold (01.00%)
    n0: 0 in 2 places, all below massif's threshold (01.00%)
  n1: 108480 0x521DAD: DMCreateDefaultSF (dm.c:3074)
   n1: 108480 0x520D8A: DMGetDefaultSF (dm.c:2991)
    n2: 108480 0x5153DB: DMGlobalToLocalBegin (dm.c:1649)
     n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
      n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
       n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52)
        n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033)
         n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
          n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743)
           n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806)
            n0: 93600 0x40BC62: main (Nwtun.c:417)
     n0: 14880 in 1 place, below massif's threshold (01.00%)
  n1: 90944 0x462C3D: DMSetUp_Plex (plex.c:1587)
   n1: 90944 0x50B88B: DMSetUp (dm.c:560)
    n1: 90944 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534)
     n1: 90944 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
      n1: 90944 0x415800: CreateMesh (SetupMeshes.c:53)
       n0: 90944 0x4076DF: main (Nwtun.c:55)
  n1: 90944 0x462AB1: DMSetUp_Plex (plex.c:1583)
   n2: 90944 0x50B88B: DMSetUp (dm.c:560)
    n1: 90944 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534)
     n1: 90944 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
      n1: 90944 0x415800: CreateMesh (SetupMeshes.c:53)
       n0: 90944 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 3 places, all below massif's threshold (01.00%)
  n1: 90944 0x462A1D: DMSetUp_Plex (plex.c:1582)
   n2: 90944 0x50B88B: DMSetUp (dm.c:560)
    n1: 90944 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534)
     n1: 90944 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
      n1: 90944 0x415800: CreateMesh (SetupMeshes.c:53)
       n0: 90944 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 3 places, all below massif's threshold (01.00%)
 n2: 266240 0x5E920BC: MPIU_Handle_obj_alloc_unsafe (in /home/rlchen/mpich2/lib/libmpich.so.3.0)
  n1: 192512 0x5EA339B: MPID_Isend (in /home/rlchen/mpich2/lib/libmpich.so.3.0)
   n1: 192512 0x5E9CBA9: PMPI_Isend (in /home/rlchen/mpich2/lib/libmpich.so.3.0)
    n1: 192512 0x11532F6: PetscCommBuildTwoSided_Allreduce (mpits.c:174)
     n1: 192512 0x11538F7: PetscCommBuildTwoSided (mpits.c:237)
      n1: 192512 0xC8603E: PetscSFSetUp_Basic (sfbasic.c:552)
       n1: 192512 0x67F6D7: PetscSFSetUp (sf.c:192)
        n1: 192512 0x689A32: PetscSFFetchAndOpBegin (sf.c:1138)
         n1: 192512 0x53DD1C: PetscSFConvertPartition (vsectionis.c:1916)
          n1: 192512 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101)
           n1: 192512 0x4154E6: CreateMesh (SetupMeshes.c:34)
            n0: 192512 0x4076DF: main (Nwtun.c:55)
  n1: 73728 0x5EAFDD0: MPID_Type_dup (in /home/rlchen/mpich2/lib/libmpich.so.3.0)
   n1: 73728 0x5EFFF19: PMPI_Type_dup (in /home/rlchen/mpich2/lib/libmpich.so.3.0)
    n1: 73728 0xC8750A: PetscSFBasicPackTypeSetup (sfbasic.c:654)
     n1: 73728 0xC88D96: PetscSFBasicGetPack (sfbasic.c:778)
      n1: 73728 0xC8B6AF: PetscSFReduceBegin_Basic (sfbasic.c:973)
       n1: 73728 0x68814F: PetscSFReduceBegin (sf.c:993)
        n1: 73728 0x682011: PetscSFCreateInverseSF (sf.c:459)
         n1: 73728 0x53E180: PetscSFConvertPartition (vsectionis.c:1937)
          n1: 73728 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101)
           n1: 73728 0x4154E6: CreateMesh (SetupMeshes.c:34)
            n0: 73728 0x4076DF: main (Nwtun.c:55)
 n1: 180480 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202)
  n1: 180480 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
   n1: 180480 0x4152C9: CreateMesh (SetupMeshes.c:23)
    n0: 180480 0x4076DF: main (Nwtun.c:55)
 n0: 155163 in 150 places, all below massif's threshold (01.00%)
#-----------
snapshot=33
#-----------
time=14373198930
mem_heap_B=5309284
mem_heap_extra_B=42044
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=34
#-----------
time=14654959061
mem_heap_B=5309268
mem_heap_extra_B=42036
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=35
#-----------
time=15172359569
mem_heap_B=5309252
mem_heap_extra_B=42028
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=36
#-----------
time=15458754406
mem_heap_B=5309268
mem_heap_extra_B=42036
mem_stacks_B=0
heap_tree=detailed
n4: 5309268 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n21: 4707353 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 712000 0x8518E1: MatFDColoringSetUpBlocked_AIJ_Private (fdaij.c:121)
   n1: 712000 0x919037: MatFDColoringSetUp_MPIXAIJ (fdmpiaij.c:570)
    n1: 712000 0x74CD34: MatFDColoringSetUp (fdmatrix.c:250)
     n1: 712000 0x43E33A: SNESComputeJacobianDefaultColorNwtun (SetupJacobian.c:907)
      n1: 712000 0x434942: FormJacobian (SetupJacobian.c:56)
       n1: 712000 0xA8F850: SNESComputeJacobian (snes.c:2193)
        n1: 712000 0xAF7C15: SNESSolve_NEWTONLS (ls.c:230)
         n1: 712000 0xA9E3CC: SNESSolve (snes.c:3743)
          n1: 712000 0x410469: SolveTimeDependent (Nwtun.c:806)
           n0: 712000 0x40BC62: main (Nwtun.c:417)
  n0: 520457 in 389 places, all below massif's threshold (01.00%)
  n1: 408112 0xC9786A: VecCreate_Seq (bvec3.c:38)
   n2: 408112 0x6BF560: VecSetType (vecreg.c:53)
    n1: 405312 0xC99F47: VecCreate_Standard (pbvec.c:269)
     n1: 405312 0x6BF560: VecSetType (vecreg.c:53)
      n1: 405312 0xA0773A: DMCreateLocalVector_Section_Private (dmi.c:65)
       n1: 405312 0x976C9B: DMCreateLocalVector_Plex (plexcreate.c:931)
        n3: 405312 0x50D220: DMCreateLocalVector (dm.c:709)
         n3: 249600 0xA035E0: DMGetLocalVector (dmget.c:49)
          n1: 93600 0x425614: CaculateLocalFunction_LS (SetupFunctions.c:742)
           n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
            n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52)
             n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033)
              n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
               n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743)
                n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806)
                 n0: 93600 0x40BC62: main (Nwtun.c:417)
          n1: 93600 0x4257D8: CaculateLocalFunction_LS (SetupFunctions.c:748)
           n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
            n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52)
             n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033)
              n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
               n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743)
                n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806)
                 n0: 93600 0x40BC62: main (Nwtun.c:417)
          n0: 62400 in 2 places, all below massif's threshold (01.00%)
         n1: 130752 0x417EA6: ConstructGeometryFVM (SetupMeshes.c:306)
          n0: 130752 0x407898: main (Nwtun.c:60)
         n0: 24960 in 1 place, below massif's threshold (01.00%)
    n0: 2800 in 1 place, below massif's threshold (01.00%)
  n1: 362400 0x8512B1: MatFDColoringSetUpBlocked_AIJ_Private (fdaij.c:70)
   n1: 362400 0x919037: MatFDColoringSetUp_MPIXAIJ (fdmpiaij.c:570)
    n1: 362400 0x74CD34: MatFDColoringSetUp (fdmatrix.c:250)
     n1: 362400 0x43E33A: SNESComputeJacobianDefaultColorNwtun (SetupJacobian.c:907)
      n1: 362400 0x434942: FormJacobian (SetupJacobian.c:56)
       n1: 362400 0xA8F850: SNESComputeJacobian (snes.c:2193)
        n1: 362400 0xAF7C15: SNESSolve_NEWTONLS (ls.c:230)
         n1: 362400 0xA9E3CC: SNESSolve (snes.c:3743)
          n1: 362400 0x410469: SolveTimeDependent (Nwtun.c:806)
           n0: 362400 0x40BC62: main (Nwtun.c:417)
  n1: 356000 0x81EE1C: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719)
   n2: 356000 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664)
    n1: 340000 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255)
     n1: 340000 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959)
      n1: 340000 0x955310: MatXAIJSetPreallocation (gcreate.c:252)
       n1: 340000 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435)
        n1: 340000 0x459619: DMCreateMatrix_Plex (plex.c:615)
         n1: 340000 0x50F42F: DMCreateMatrix (dm.c:941)
          n0: 340000 0x40B906: main (Nwtun.c:396)
    n0: 16000 in 2 places, all below massif's threshold (01.00%)
  n1: 356000 0x81EDCA: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719)
   n2: 356000 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664)
    n1: 340000 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255)
     n1: 340000 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959)
      n1: 340000 0x955310: MatXAIJSetPreallocation (gcreate.c:252)
       n1: 340000 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435)
        n1: 340000 0x459619: DMCreateMatrix_Plex (plex.c:615)
         n1: 340000 0x50F42F: DMCreateMatrix (dm.c:941)
          n0: 340000 0x40B906: main (Nwtun.c:396)
    n0: 16000 in 2 places, all below massif's threshold (01.00%)
  n2: 233704 0xC99701: VecCreate_MPI_Private (pbvec.c:201)
   n1: 137064 0xC99C3D: VecCreate_MPI (pbvec.c:244)
    n1: 137064 0x6BF560: VecSetType (vecreg.c:53)
     n1: 137064 0xC99FA5: VecCreate_Standard (pbvec.c:271)
      n2: 137064 0x6BF560: VecSetType (vecreg.c:53)
       n1: 109360 0xA07131: DMCreateGlobalVector_Section_Private (dmi.c:34)
        n1: 109360 0x976932: DMCreateGlobalVector_Plex (plexcreate.c:917)
         n2: 109360 0x50CE38: DMCreateGlobalVector (dm.c:681)
          n2: 97280 0xA04005: DMGetGlobalVector (dmget.c:154)
           n1: 85200 0x426EC6: ConstructCellCentriodGradient (SetupFunctions.c:855)
            n1: 85200 0x42054B: FormTimeStepFunction (SetupFunctions.c:233)
             n1: 85200 0x41E3AB: FormFunction (SetupFunctions.c:52)
              n1: 85200 0xA8D3FC: SNESComputeFunction (snes.c:2033)
               n1: 85200 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
                n1: 85200 0xA9E3CC: SNESSolve (snes.c:3743)
                 n1: 85200 0x410469: SolveTimeDependent (Nwtun.c:806)
                  n0: 85200 0x40BC62: main (Nwtun.c:417)
           n0: 12080 in 1 place, below massif's threshold (01.00%)
          n0: 12080 in 1 place, below massif's threshold (01.00%)
       n0: 27704 in 4 places, all below massif's threshold (01.00%)
   n1: 96640 0xC98DCF: VecDuplicate_MPI (pbvec.c:68)
    n1: 96640 0x6B04CF: VecDuplicate (vector.c:399)
     n0: 96640 in 8 places, all below massif's threshold (01.00%)
  n1: 216960 0x521E45: DMCreateDefaultSF (dm.c:3075)
   n1: 216960 0x520D8A: DMGetDefaultSF (dm.c:2991)
    n2: 216960 0x5153DB: DMGlobalToLocalBegin (dm.c:1649)
     n1: 187200 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
      n1: 187200 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
       n1: 187200 0x41E3AB: FormFunction (SetupFunctions.c:52)
        n1: 187200 0xA8D3FC: SNESComputeFunction (snes.c:2033)
         n1: 187200 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
          n1: 187200 0xA9E3CC: SNESSolve (snes.c:3743)
           n1: 187200 0x410469: SolveTimeDependent (Nwtun.c:806)
            n0: 187200 0x40BC62: main (Nwtun.c:417)
     n0: 29760 in 1 place, below massif's threshold (01.00%)
  n1: 154640 0x52EBD7: PetscSectionSetChart (vsectionis.c:389)
   n0: 154640 in 19 places, all below massif's threshold (01.00%)
  n1: 154640 0x52EC32: PetscSectionSetChart (vsectionis.c:389)
   n0: 154640 in 19 places, all below massif's threshold (01.00%)
  n3: 153664 0x50FE4C: DMGetWorkArray (dm.c:1005)
   n2: 76832 0x45FF3D: DMPlexGetTransitiveClosure (plex.c:1287)
    n1: 76832 0x47B16A: DMPlexVecGetClosure (plex.c:4614)
     n0: 76832 in 2 places, all below massif's threshold (01.00%)
    n0: 0 in 2 places, all below massif's threshold (01.00%)
   n2: 76832 0x45FFD3: DMPlexGetTransitiveClosure (plex.c:1291)
    n1: 76832 0x47B16A: DMPlexVecGetClosure (plex.c:4614)
     n0: 76832 in 2 places, all below massif's threshold (01.00%)
    n0: 0 in 2 places, all below massif's threshold (01.00%)
   n0: 0 in 4 places, all below massif's threshold (01.00%)
  n2: 117480 0x6812ED: PetscSFSetGraph (sf.c:349)
   n1: 108480 0x522622: DMCreateDefaultSF (dm.c:3116)
    n1: 108480 0x520D8A: DMGetDefaultSF (dm.c:2991)
     n2: 108480 0x5153DB: DMGlobalToLocalBegin (dm.c:1649)
      n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
       n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
        n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52)
         n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033)
          n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
           n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743)
            n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806)
             n0: 93600 0x40BC62: main (Nwtun.c:417)
      n0: 14880 in 1 place, below massif's threshold (01.00%)
   n0: 9000 in 9 places, all below massif's threshold (01.00%)
  n2: 117480 0x681299: PetscSFSetGraph (sf.c:349)
   n1: 108480 0x522622: DMCreateDefaultSF (dm.c:3116)
    n1: 108480 0x520D8A: DMGetDefaultSF (dm.c:2991)
     n2: 108480 0x5153DB: DMGlobalToLocalBegin (dm.c:1649)
      n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
       n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
        n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52)
         n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033)
          n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
           n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743)
            n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806)
             n0: 93600 0x40BC62: main (Nwtun.c:417)
      n0: 14880 in 1 place, below massif's threshold (01.00%)
   n0: 9000 in 9 places, all below massif's threshold (01.00%)
  n2: 117480 0xC88EE6: PetscSFBasicGetPack (sfbasic.c:779)
   n1: 117480 0xC8AA7F: PetscSFBcastBegin_Basic (sfbasic.c:914)
    n2: 117480 0x687407: PetscSFBcastBegin (sf.c:926)
     n2: 108480 0x51554F: DMGlobalToLocalBegin (dm.c:1656)
      n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
       n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
        n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52)
         n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033)
          n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
           n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743)
            n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806)
             n0: 93600 0x40BC62: main (Nwtun.c:417)
      n0: 14880 in 1 place, below massif's threshold (01.00%)
     n0: 9000 in 8 places, all below massif's threshold (01.00%)
   n0: 0 in 1 place, below massif's threshold (01.00%)
  n2: 116064 0x659AFA: ISCreate (isreg.c:41)
   n2: 116064 0x64D670: ISCreateGeneral (general.c:632)
    n2: 116064 0x4E222E: DMLabelGetValueIS (plexlabel.c:518)
     n2: 116064 0x4E8069: DMPlexGetLabelIdIS (plexlabel.c:1013)
      n1: 116064 0x41D333: SetUpSymetricBoundaryMaching (SetupMeshes.c:1030)
       n0: 116064 0x40796B: main (Nwtun.c:65)
      n0: 0 in 2 places, all below massif's threshold (01.00%)
     n0: 0 in 2 places, all below massif's threshold (01.00%)
    n0: 0 in 6 places, all below massif's threshold (01.00%)
   n0: 0 in 1 place, below massif's threshold (01.00%)
  n2: 114480 0xC88E56: PetscSFBasicGetPack (sfbasic.c:779)
   n1: 114480 0xC8AA7F: PetscSFBcastBegin_Basic (sfbasic.c:914)
    n2: 114480 0x687407: PetscSFBcastBegin (sf.c:926)
     n2: 108800 0x51554F: DMGlobalToLocalBegin (dm.c:1656)
      n1: 93840 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
       n1: 93840 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
        n1: 93840 0x41E3AB: FormFunction (SetupFunctions.c:52)
         n1: 93840 0xA8D3FC: SNESComputeFunction (snes.c:2033)
          n1: 93840 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
           n1: 93840 0xA9E3CC: SNESSolve (snes.c:3743)
            n1: 93840 0x410469: SolveTimeDependent (Nwtun.c:806)
             n0: 93840 0x40BC62: main (Nwtun.c:417)
      n0: 14960 in 1 place, below massif's threshold (01.00%)
     n0: 5680 in 8 places, all below massif's threshold (01.00%)
   n0: 0 in 1 place, below massif's threshold (01.00%)
  n1: 114480 0xC861ED: PetscSFSetUp_Basic (sfbasic.c:557)
   n2: 114480 0x67F6D7: PetscSFSetUp (sf.c:192)
    n2: 114480 0x68739A: PetscSFBcastBegin (sf.c:925)
     n2: 108800 0x51554F: DMGlobalToLocalBegin (dm.c:1656)
      n1: 93840 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
       n1: 93840 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
        n1: 93840 0x41E3AB: FormFunction (SetupFunctions.c:52)
         n1: 93840 0xA8D3FC: SNESComputeFunction (snes.c:2033)
          n1: 93840 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
           n1: 93840 0xA9E3CC: SNESSolve (snes.c:3743)
            n1: 93840 0x410469: SolveTimeDependent (Nwtun.c:806)
             n0: 93840 0x40BC62: main (Nwtun.c:417)
      n0: 14960 in 1 place, below massif's threshold (01.00%)
     n0: 5680 in 7 places, all below massif's threshold (01.00%)
    n0: 0 in 2 places, all below massif's threshold (01.00%)
  n1: 108480 0x521DAD: DMCreateDefaultSF (dm.c:3074)
   n1: 108480 0x520D8A: DMGetDefaultSF (dm.c:2991)
    n2: 108480 0x5153DB: DMGlobalToLocalBegin (dm.c:1649)
     n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
      n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
       n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52)
        n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033)
         n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
          n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743)
           n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806)
            n0: 93600 0x40BC62: main (Nwtun.c:417)
     n0: 14880 in 1 place, below massif's threshold (01.00%)
  n1: 90944 0x462C3D: DMSetUp_Plex (plex.c:1587)
   n1: 90944 0x50B88B: DMSetUp (dm.c:560)
    n1: 90944 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534)
     n1: 90944 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
      n1: 90944 0x415800: CreateMesh (SetupMeshes.c:53)
       n0: 90944 0x4076DF: main (Nwtun.c:55)
  n1: 90944 0x462AB1: DMSetUp_Plex (plex.c:1583)
   n2: 90944 0x50B88B: DMSetUp (dm.c:560)
    n1: 90944 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534)
     n1: 90944 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
      n1: 90944 0x415800: CreateMesh (SetupMeshes.c:53)
       n0: 90944 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 3 places, all below massif's threshold (01.00%)
  n1: 90944 0x462A1D: DMSetUp_Plex (plex.c:1582)
   n2: 90944 0x50B88B: DMSetUp (dm.c:560)
    n1: 90944 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534)
     n1: 90944 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
      n1: 90944 0x415800: CreateMesh (SetupMeshes.c:53)
       n0: 90944 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 3 places, all below massif's threshold (01.00%)
 n2: 266240 0x5E920BC: MPIU_Handle_obj_alloc_unsafe (in /home/rlchen/mpich2/lib/libmpich.so.3.0)
  n1: 192512 0x5EA339B: MPID_Isend (in /home/rlchen/mpich2/lib/libmpich.so.3.0)
   n1: 192512 0x5E9CBA9: PMPI_Isend (in /home/rlchen/mpich2/lib/libmpich.so.3.0)
    n1: 192512 0x11532F6: PetscCommBuildTwoSided_Allreduce (mpits.c:174)
     n1: 192512 0x11538F7: PetscCommBuildTwoSided (mpits.c:237)
      n1: 192512 0xC8603E: PetscSFSetUp_Basic (sfbasic.c:552)
       n1: 192512 0x67F6D7: PetscSFSetUp (sf.c:192)
        n1: 192512 0x689A32: PetscSFFetchAndOpBegin (sf.c:1138)
         n1: 192512 0x53DD1C: PetscSFConvertPartition (vsectionis.c:1916)
          n1: 192512 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101)
           n1: 192512 0x4154E6: CreateMesh (SetupMeshes.c:34)
            n0: 192512 0x4076DF: main (Nwtun.c:55)
  n1: 73728 0x5EAFDD0: MPID_Type_dup (in /home/rlchen/mpich2/lib/libmpich.so.3.0)
   n1: 73728 0x5EFFF19: PMPI_Type_dup (in /home/rlchen/mpich2/lib/libmpich.so.3.0)
    n1: 73728 0xC8750A: PetscSFBasicPackTypeSetup (sfbasic.c:654)
     n1: 73728 0xC88D96: PetscSFBasicGetPack (sfbasic.c:778)
      n1: 73728 0xC8B6AF: PetscSFReduceBegin_Basic (sfbasic.c:973)
       n1: 73728 0x68814F: PetscSFReduceBegin (sf.c:993)
        n1: 73728 0x682011: PetscSFCreateInverseSF (sf.c:459)
         n1: 73728 0x53E180: PetscSFConvertPartition (vsectionis.c:1937)
          n1: 73728 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101)
           n1: 73728 0x4154E6: CreateMesh (SetupMeshes.c:34)
            n0: 73728 0x4076DF: main (Nwtun.c:55)
 n1: 180480 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202)
  n1: 180480 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
   n1: 180480 0x4152C9: CreateMesh (SetupMeshes.c:23)
    n0: 180480 0x4076DF: main (Nwtun.c:55)
 n0: 155195 in 150 places, all below massif's threshold (01.00%)
#-----------
snapshot=37
#-----------
time=15977774523
mem_heap_B=5309284
mem_heap_extra_B=42044
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=38
#-----------
time=16396490298
mem_heap_B=5309268
mem_heap_extra_B=42036
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=39
#-----------
time=16768519490
mem_heap_B=5309284
mem_heap_extra_B=42044
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=40
#-----------
time=17331041754
mem_heap_B=5309252
mem_heap_extra_B=42028
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=41
#-----------
time=17841489784
mem_heap_B=5309252
mem_heap_extra_B=42028
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=42
#-----------
time=18369461201
mem_heap_B=5309252
mem_heap_extra_B=42028
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=43
#-----------
time=18660261684
mem_heap_B=5310218
mem_heap_extra_B=42094
mem_stacks_B=0
heap_tree=detailed
n4: 5310218 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n21: 4708335 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 712000 0x8518E1: MatFDColoringSetUpBlocked_AIJ_Private (fdaij.c:121)
   n1: 712000 0x919037: MatFDColoringSetUp_MPIXAIJ (fdmpiaij.c:570)
    n1: 712000 0x74CD34: MatFDColoringSetUp (fdmatrix.c:250)
     n1: 712000 0x43E33A: SNESComputeJacobianDefaultColorNwtun (SetupJacobian.c:907)
      n1: 712000 0x434942: FormJacobian (SetupJacobian.c:56)
       n1: 712000 0xA8F850: SNESComputeJacobian (snes.c:2193)
        n1: 712000 0xAF7C15: SNESSolve_NEWTONLS (ls.c:230)
         n1: 712000 0xA9E3CC: SNESSolve (snes.c:3743)
          n1: 712000 0x410469: SolveTimeDependent (Nwtun.c:806)
           n0: 712000 0x40BC62: main (Nwtun.c:417)
  n0: 520815 in 389 places, all below massif's threshold (01.00%)
  n1: 408112 0xC9786A: VecCreate_Seq (bvec3.c:38)
   n2: 408112 0x6BF560: VecSetType (vecreg.c:53)
    n1: 405312 0xC99F47: VecCreate_Standard (pbvec.c:269)
     n1: 405312 0x6BF560: VecSetType (vecreg.c:53)
      n1: 405312 0xA0773A: DMCreateLocalVector_Section_Private (dmi.c:65)
       n1: 405312 0x976C9B: DMCreateLocalVector_Plex (plexcreate.c:931)
        n3: 405312 0x50D220: DMCreateLocalVector (dm.c:709)
         n3: 249600 0xA035E0: DMGetLocalVector (dmget.c:49)
          n1: 93600 0x425614: CaculateLocalFunction_LS (SetupFunctions.c:742)
           n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
            n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52)
             n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033)
              n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
               n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743)
                n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806)
                 n0: 93600 0x40BC62: main (Nwtun.c:417)
          n1: 93600 0x4257D8: CaculateLocalFunction_LS (SetupFunctions.c:748)
           n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
            n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52)
             n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033)
              n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
               n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743)
                n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806)
                 n0: 93600 0x40BC62: main (Nwtun.c:417)
          n0: 62400 in 2 places, all below massif's threshold (01.00%)
         n1: 130752 0x417EA6: ConstructGeometryFVM (SetupMeshes.c:306)
          n0: 130752 0x407898: main (Nwtun.c:60)
         n0: 24960 in 1 place, below massif's threshold (01.00%)
    n0: 2800 in 1 place, below massif's threshold (01.00%)
  n1: 362400 0x8512B1: MatFDColoringSetUpBlocked_AIJ_Private (fdaij.c:70)
   n1: 362400 0x919037: MatFDColoringSetUp_MPIXAIJ (fdmpiaij.c:570)
    n1: 362400 0x74CD34: MatFDColoringSetUp (fdmatrix.c:250)
     n1: 362400 0x43E33A: SNESComputeJacobianDefaultColorNwtun (SetupJacobian.c:907)
      n1: 362400 0x434942: FormJacobian (SetupJacobian.c:56)
       n1: 362400 0xA8F850: SNESComputeJacobian (snes.c:2193)
        n1: 362400 0xAF7C15: SNESSolve_NEWTONLS (ls.c:230)
         n1: 362400 0xA9E3CC: SNESSolve (snes.c:3743)
          n1: 362400 0x410469: SolveTimeDependent (Nwtun.c:806)
           n0: 362400 0x40BC62: main (Nwtun.c:417)
  n1: 356000 0x81EE1C: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719)
   n2: 356000 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664)
    n1: 340000 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255)
     n1: 340000 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959)
      n1: 340000 0x955310: MatXAIJSetPreallocation (gcreate.c:252)
       n1: 340000 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435)
        n1: 340000 0x459619: DMCreateMatrix_Plex (plex.c:615)
         n1: 340000 0x50F42F: DMCreateMatrix (dm.c:941)
          n0: 340000 0x40B906: main (Nwtun.c:396)
    n0: 16000 in 2 places, all below massif's threshold (01.00%)
  n1: 356000 0x81EDCA: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719)
   n2: 356000 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664)
    n1: 340000 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255)
     n1: 340000 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959)
      n1: 340000 0x955310: MatXAIJSetPreallocation (gcreate.c:252)
       n1: 340000 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435)
        n1: 340000 0x459619: DMCreateMatrix_Plex (plex.c:615)
         n1: 340000 0x50F42F: DMCreateMatrix (dm.c:941)
          n0: 340000 0x40B906: main (Nwtun.c:396)
    n0: 16000 in 2 places, all below massif's threshold (01.00%)
  n2: 233704 0xC99701: VecCreate_MPI_Private (pbvec.c:201)
   n1: 137064 0xC99C3D: VecCreate_MPI (pbvec.c:244)
    n1: 137064 0x6BF560: VecSetType (vecreg.c:53)
     n1: 137064 0xC99FA5: VecCreate_Standard (pbvec.c:271)
      n2: 137064 0x6BF560: VecSetType (vecreg.c:53)
       n1: 109360 0xA07131: DMCreateGlobalVector_Section_Private (dmi.c:34)
        n1: 109360 0x976932: DMCreateGlobalVector_Plex (plexcreate.c:917)
         n2: 109360 0x50CE38: DMCreateGlobalVector (dm.c:681)
          n2: 97280 0xA04005: DMGetGlobalVector (dmget.c:154)
           n1: 85200 0x426EC6: ConstructCellCentriodGradient (SetupFunctions.c:855)
            n1: 85200 0x42054B: FormTimeStepFunction (SetupFunctions.c:233)
             n1: 85200 0x41E3AB: FormFunction (SetupFunctions.c:52)
              n1: 85200 0xA8D3FC: SNESComputeFunction (snes.c:2033)
               n1: 85200 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
                n1: 85200 0xA9E3CC: SNESSolve (snes.c:3743)
                 n1: 85200 0x410469: SolveTimeDependent (Nwtun.c:806)
                  n0: 85200 0x40BC62: main (Nwtun.c:417)
           n0: 12080 in 1 place, below massif's threshold (01.00%)
          n0: 12080 in 1 place, below massif's threshold (01.00%)
       n0: 27704 in 4 places, all below massif's threshold (01.00%)
   n1: 96640 0xC98DCF: VecDuplicate_MPI (pbvec.c:68)
    n1: 96640 0x6B04CF: VecDuplicate (vector.c:399)
     n0: 96640 in 8 places, all below massif's threshold (01.00%)
  n1: 216960 0x521E45: DMCreateDefaultSF (dm.c:3075)
   n1: 216960 0x520D8A: DMGetDefaultSF (dm.c:2991)
    n2: 216960 0x5153DB: DMGlobalToLocalBegin (dm.c:1649)
     n1: 187200 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
      n1: 187200 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
       n1: 187200 0x41E3AB: FormFunction (SetupFunctions.c:52)
        n1: 187200 0xA8D3FC: SNESComputeFunction (snes.c:2033)
         n1: 187200 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
          n1: 187200 0xA9E3CC: SNESSolve (snes.c:3743)
           n1: 187200 0x410469: SolveTimeDependent (Nwtun.c:806)
            n0: 187200 0x40BC62: main (Nwtun.c:417)
     n0: 29760 in 1 place, below massif's threshold (01.00%)
  n1: 154640 0x52EBD7: PetscSectionSetChart (vsectionis.c:389)
   n0: 154640 in 19 places, all below massif's threshold (01.00%)
  n1: 154640 0x52EC32: PetscSectionSetChart (vsectionis.c:389)
   n0: 154640 in 19 places, all below massif's threshold (01.00%)
  n3: 153664 0x50FE4C: DMGetWorkArray (dm.c:1005)
   n2: 76832 0x45FF3D: DMPlexGetTransitiveClosure (plex.c:1287)
    n1: 76832 0x47B16A: DMPlexVecGetClosure (plex.c:4614)
     n0: 76832 in 2 places, all below massif's threshold (01.00%)
    n0: 0 in 2 places, all below massif's threshold (01.00%)
   n2: 76832 0x45FFD3: DMPlexGetTransitiveClosure (plex.c:1291)
    n1: 76832 0x47B16A: DMPlexVecGetClosure (plex.c:4614)
     n0: 76832 in 2 places, all below massif's threshold (01.00%)
    n0: 0 in 2 places, all below massif's threshold (01.00%)
   n0: 0 in 4 places, all below massif's threshold (01.00%)
  n2: 117480 0x6812ED: PetscSFSetGraph (sf.c:349)
   n1: 108480 0x522622: DMCreateDefaultSF (dm.c:3116)
    n1: 108480 0x520D8A: DMGetDefaultSF (dm.c:2991)
     n2: 108480 0x5153DB: DMGlobalToLocalBegin (dm.c:1649)
      n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
       n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
        n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52)
         n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033)
          n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
           n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743)
            n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806)
             n0: 93600 0x40BC62: main (Nwtun.c:417)
      n0: 14880 in 1 place, below massif's threshold (01.00%)
   n0: 9000 in 9 places, all below massif's threshold (01.00%)
  n2: 117480 0x681299: PetscSFSetGraph (sf.c:349)
   n1: 108480 0x522622: DMCreateDefaultSF (dm.c:3116)
    n1: 108480 0x520D8A: DMGetDefaultSF (dm.c:2991)
     n2: 108480 0x5153DB: DMGlobalToLocalBegin (dm.c:1649)
      n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
       n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
        n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52)
         n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033)
          n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
           n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743)
            n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806)
             n0: 93600 0x40BC62: main (Nwtun.c:417)
      n0: 14880 in 1 place, below massif's threshold (01.00%)
   n0: 9000 in 9 places, all below massif's threshold (01.00%)
  n2: 117480 0xC88EE6: PetscSFBasicGetPack (sfbasic.c:779)
   n1: 117480 0xC8AA7F: PetscSFBcastBegin_Basic (sfbasic.c:914)
    n2: 117480 0x687407: PetscSFBcastBegin (sf.c:926)
     n2: 108480 0x51554F: DMGlobalToLocalBegin (dm.c:1656)
      n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
       n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
        n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52)
         n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033)
          n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
           n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743)
            n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806)
             n0: 93600 0x40BC62: main (Nwtun.c:417)
      n0: 14880 in 1 place, below massif's threshold (01.00%)
     n0: 9000 in 8 places, all below massif's threshold (01.00%)
   n0: 0 in 1 place, below massif's threshold (01.00%)
  n2: 116688 0x659AFA: ISCreate (isreg.c:41)
   n2: 116688 0x64D670: ISCreateGeneral (general.c:632)
    n2: 116688 0x4E222E: DMLabelGetValueIS (plexlabel.c:518)
     n2: 116688 0x4E8069: DMPlexGetLabelIdIS (plexlabel.c:1013)
      n1: 116064 0x41D333: SetUpSymetricBoundaryMaching (SetupMeshes.c:1030)
       n0: 116064 0x40796B: main (Nwtun.c:65)
      n0: 624 in 2 places, all below massif's threshold (01.00%)
     n0: 0 in 2 places, all below massif's threshold (01.00%)
    n0: 0 in 6 places, all below massif's threshold (01.00%)
   n0: 0 in 1 place, below massif's threshold (01.00%)
  n2: 114480 0xC88E56: PetscSFBasicGetPack (sfbasic.c:779)
   n1: 114480 0xC8AA7F: PetscSFBcastBegin_Basic (sfbasic.c:914)
    n2: 114480 0x687407: PetscSFBcastBegin (sf.c:926)
     n2: 108800 0x51554F: DMGlobalToLocalBegin (dm.c:1656)
      n1: 93840 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
       n1: 93840 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
        n1: 93840 0x41E3AB: FormFunction (SetupFunctions.c:52)
         n1: 93840 0xA8D3FC: SNESComputeFunction (snes.c:2033)
          n1: 93840 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
           n1: 93840 0xA9E3CC: SNESSolve (snes.c:3743)
            n1: 93840 0x410469: SolveTimeDependent (Nwtun.c:806)
             n0: 93840 0x40BC62: main (Nwtun.c:417)
      n0: 14960 in 1 place, below massif's threshold (01.00%)
     n0: 5680 in 8 places, all below massif's threshold (01.00%)
   n0: 0 in 1 place, below massif's threshold (01.00%)
  n1: 114480 0xC861ED: PetscSFSetUp_Basic (sfbasic.c:557)
   n2: 114480 0x67F6D7: PetscSFSetUp (sf.c:192)
    n2: 114480 0x68739A: PetscSFBcastBegin (sf.c:925)
     n2: 108800 0x51554F: DMGlobalToLocalBegin (dm.c:1656)
      n1: 93840 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
       n1: 93840 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
        n1: 93840 0x41E3AB: FormFunction (SetupFunctions.c:52)
         n1: 93840 0xA8D3FC: SNESComputeFunction (snes.c:2033)
          n1: 93840 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
           n1: 93840 0xA9E3CC: SNESSolve (snes.c:3743)
            n1: 93840 0x410469: SolveTimeDependent (Nwtun.c:806)
             n0: 93840 0x40BC62: main (Nwtun.c:417)
      n0: 14960 in 1 place, below massif's threshold (01.00%)
     n0: 5680 in 7 places, all below massif's threshold (01.00%)
    n0: 0 in 2 places, all below massif's threshold (01.00%)
  n1: 108480 0x521DAD: DMCreateDefaultSF (dm.c:3074)
   n1: 108480 0x520D8A: DMGetDefaultSF (dm.c:2991)
    n2: 108480 0x5153DB: DMGlobalToLocalBegin (dm.c:1649)
     n1: 93600 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
      n1: 93600 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
       n1: 93600 0x41E3AB: FormFunction (SetupFunctions.c:52)
        n1: 93600 0xA8D3FC: SNESComputeFunction (snes.c:2033)
         n1: 93600 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
          n1: 93600 0xA9E3CC: SNESSolve (snes.c:3743)
           n1: 93600 0x410469: SolveTimeDependent (Nwtun.c:806)
            n0: 93600 0x40BC62: main (Nwtun.c:417)
     n0: 14880 in 1 place, below massif's threshold (01.00%)
  n1: 90944 0x462C3D: DMSetUp_Plex (plex.c:1587)
   n1: 90944 0x50B88B: DMSetUp (dm.c:560)
    n1: 90944 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534)
     n1: 90944 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
      n1: 90944 0x415800: CreateMesh (SetupMeshes.c:53)
       n0: 90944 0x4076DF: main (Nwtun.c:55)
  n1: 90944 0x462AB1: DMSetUp_Plex (plex.c:1583)
   n2: 90944 0x50B88B: DMSetUp (dm.c:560)
    n1: 90944 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534)
     n1: 90944 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
      n1: 90944 0x415800: CreateMesh (SetupMeshes.c:53)
       n0: 90944 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 3 places, all below massif's threshold (01.00%)
  n1: 90944 0x462A1D: DMSetUp_Plex (plex.c:1582)
   n2: 90944 0x50B88B: DMSetUp (dm.c:560)
    n1: 90944 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534)
     n1: 90944 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
      n1: 90944 0x415800: CreateMesh (SetupMeshes.c:53)
       n0: 90944 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 3 places, all below massif's threshold (01.00%)
 n2: 266240 0x5E920BC: MPIU_Handle_obj_alloc_unsafe (in /home/rlchen/mpich2/lib/libmpich.so.3.0)
  n1: 192512 0x5EA339B: MPID_Isend (in /home/rlchen/mpich2/lib/libmpich.so.3.0)
   n1: 192512 0x5E9CBA9: PMPI_Isend (in /home/rlchen/mpich2/lib/libmpich.so.3.0)
    n1: 192512 0x11532F6: PetscCommBuildTwoSided_Allreduce (mpits.c:174)
     n1: 192512 0x11538F7: PetscCommBuildTwoSided (mpits.c:237)
      n1: 192512 0xC8603E: PetscSFSetUp_Basic (sfbasic.c:552)
       n1: 192512 0x67F6D7: PetscSFSetUp (sf.c:192)
        n1: 192512 0x689A32: PetscSFFetchAndOpBegin (sf.c:1138)
         n1: 192512 0x53DD1C: PetscSFConvertPartition (vsectionis.c:1916)
          n1: 192512 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101)
           n1: 192512 0x4154E6: CreateMesh (SetupMeshes.c:34)
            n0: 192512 0x4076DF: main (Nwtun.c:55)
  n1: 73728 0x5EAFDD0: MPID_Type_dup (in /home/rlchen/mpich2/lib/libmpich.so.3.0)
   n1: 73728 0x5EFFF19: PMPI_Type_dup (in /home/rlchen/mpich2/lib/libmpich.so.3.0)
    n1: 73728 0xC8750A: PetscSFBasicPackTypeSetup (sfbasic.c:654)
     n1: 73728 0xC88D96: PetscSFBasicGetPack (sfbasic.c:778)
      n1: 73728 0xC8B6AF: PetscSFReduceBegin_Basic (sfbasic.c:973)
       n1: 73728 0x68814F: PetscSFReduceBegin (sf.c:993)
        n1: 73728 0x682011: PetscSFCreateInverseSF (sf.c:459)
         n1: 73728 0x53E180: PetscSFConvertPartition (vsectionis.c:1937)
          n1: 73728 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101)
           n1: 73728 0x4154E6: CreateMesh (SetupMeshes.c:34)
            n0: 73728 0x4076DF: main (Nwtun.c:55)
 n1: 180480 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202)
  n1: 180480 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
   n1: 180480 0x4152C9: CreateMesh (SetupMeshes.c:23)
    n0: 180480 0x4076DF: main (Nwtun.c:55)
 n0: 155163 in 150 places, all below massif's threshold (01.00%)
#-----------
snapshot=44
#-----------
time=19180269224
mem_heap_B=5309284
mem_heap_extra_B=42044
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=45
#-----------
time=19461214172
mem_heap_B=5310218
mem_heap_extra_B=42094
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=46
#-----------
time=19980095559
mem_heap_B=5309252
mem_heap_extra_B=42028
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=47
#-----------
time=20484932365
mem_heap_B=11658972
mem_heap_extra_B=46180
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=48
#-----------
time=20818609640
mem_heap_B=8938979
mem_heap_extra_B=46125
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=49
#-----------
time=21431132383
mem_heap_B=8970627
mem_heap_extra_B=46525
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=50
#-----------
time=21762203435
mem_heap_B=8970627
mem_heap_extra_B=46525
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=51
#-----------
time=22092956732
mem_heap_B=8970627
mem_heap_extra_B=46525
mem_stacks_B=0
heap_tree=empty