[petsc-users] SNESComputeJacobianDefaultColor use too much memory

Rongliang Chen rongliang.chan at gmail.com
Thu Feb 18 19:37:14 CST 2016


Hi Barry,

When I increase the size of the mesh, another memory issue comes out. 
The DMPlexInterpolateFaces_Internal takes too much memory. The massif 
output is attached. Any suggestions for this? Thanks.

Best,
Rongliang

On 02/18/2016 04:11 AM, Barry Smith wrote:
>    Hmm, Matt, this is nuts. This
>
>   if (size > 1) {
>      /* create a sequential iscoloring on all processors */
>      ierr = MatGetSeqNonzeroStructure(mat,&mat_seq);CHKERRQ(ierr);
>    }
>
>    It sequentializes the graph.
>
>     It looks like the only parallel coloring for Jacobians is MATCOLORINGGREEDY? Try that.
>
>     Maybe that should be the default?
>
>     Barry
>
>
>> On Feb 17, 2016, at 1:49 AM, Rongliang Chen <rongliang.chan at gmail.com> wrote:
>>
>> Dear Barry,
>>
>> The massif output for the large case is attached. It shows that the job was kill in the function  MatGetSubMatrix_MPIAIJ_All(). Any suggestions?
>>
>> --------------------
>> [8]PETSC ERROR: #1 MatGetSubMatrix_MPIAIJ_All() line 622 in /home/rlchen/soft/petsc-3.5.2/src/mat/impls/aij/mpi/mpiov.c
>> [8]PETSC ERROR: #2 MatGetSeqNonzeroStructure_MPIAIJ() line 3010 in /home/rlchen/soft/petsc-3.5.2/src/mat/impls/aij/mpi/mpiaij.c
>> [8]PETSC ERROR: #3 MatGetSeqNonzeroStructure() line 6487 in /home/rlchen/soft/petsc-3.5.2/src/mat/interface/matrix.c
>> [8]PETSC ERROR: #4 MatColoringApply_SL() line 78 in /home/rlchen/soft/petsc-3.5.2/src/mat/color/impls/minpack/color.c
>> [8]PETSC ERROR: #5 MatColoringApply() line 379 in /home/rlchen/soft/petsc-3.5.2/src/mat/color/interface/matcoloring.c
>> [8]PETSC ERROR: #6 SNESComputeJacobianDefaultColor() line 71 in /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snesj2.c
>> [8]PETSC ERROR: #7 FormJacobian() line 58 in /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/SetupJacobian.c
>> [8]PETSC ERROR: #8 SNESComputeJacobian() line 2193 in /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c
>> [8]PETSC ERROR: #9 SNESSolve_NEWTONLS() line 230 in /home/rlchen/soft/petsc-3.5.2/src/snes/impls/ls/ls.c
>> [8]PETSC ERROR: #10 SNESSolve() line 3743 in /home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c
>> [8]PETSC ERROR: #11 SolveTimeDependent() line 758 in /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/Nwtun.c
>> [8]PETSC ERROR: #12 main() line 417 in /home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/Nwtun.c
>> -------------------
>>
>> Best,
>> Rongliang
>>
>> On 02/17/2016 02:09 PM, Barry Smith wrote:
>>>    Yes, this is the type of output I was expecting. Now you need to produce it for a large case.
>>>
>>>    Barry
>>>
>>>> On Feb 16, 2016, at 11:49 PM, Rongliang Chen <rongliang.chan at gmail.com> wrote:
>>>>
>>>> Hi Barry,
>>>>
>>>> I run the code with valgrind on a workstation for a smaller case and it produces some ASCII information (see attached).  Is this helpful?
>>>>
>>>> Best,
>>>> Rongliang
>>>>
>>>> On 02/17/2016 01:30 PM, Barry Smith wrote:
>>>>>    Hmm, something didn't work right with massif. I should give a bunch of ASCII information about how much memory is used at different times in the code. You may need to play around with the massif options and google documentation on how to get it to provide useful information. Once it produces the useful information it will be very helpful.
>>>>>
>>>>> time=0
>>>>> mem_heap_B=0
>>>>> mem_heap_extra_B=0
>>>>> mem_stacks_B=0
>>>>> heap_tree=empty
>>>>>
>>>>>
>>>>>> On Feb 16, 2016, at 9:44 PM, Rongliang Chen <rongliang.chan at gmail.com> wrote:
>>>>>>
>>>>>> Dear Barry,
>>>>>>
>>>>>> Many thanks for your reply.
>>>>>>
>>>>>> I checked with the valgrind and did not obtain any outputs (massif.out.<pid>) because the job was killed before it reached the end.
>>>>>>
>>>>>> Then I switch to a smaller case, it works well and one of the output is attached (I did not find any useful information in it). The output with the option -mat_coloring_view is followed, which shows that the number of colors is 65. Any ideas for this?
>>>>>>
>>>>>> MatColoring Object: 480 MPI processes
>>>>>>    type: sl
>>>>>>    Weight type: RANDOM
>>>>>>    Distance 2, Max. Colors 65535
>>>>>>    Number of colors 65
>>>>>>    Number of total columns 1637350
>>>>>>
>>>>>> Best regards,
>>>>>> Rongliang
>>>>>>
>>>>>> On 02/17/2016 01:13 AM, Barry Smith wrote:
>>>>>>>    How many colors are needed?
>>>>>>>
>>>>>>>    You need to produce a breakdown of where all the memory is being used. For example valgrind with the
>>>>>>> http://valgrind.org/docs/manual/ms-manual.html
>>>>>>>
>>>>>>>
>>>>>>>    Barry
>>>>>>>
>>>>>>>
>>>>>>>> On Feb 16, 2016, at 6:45 AM, Rongliang Chen <rongliang.chan at gmail.com>
>>>>>>>>   wrote:
>>>>>>>>
>>>>>>>> Dear all,
>>>>>>>>
>>>>>>>> I am using the DMPlex to solve a PDE on a unstructured mesh and I use the SNESComputeJacobianDefaultColor to compute the Jacobian matrix.
>>>>>>>>
>>>>>>>> My code works well for small problems (such as problem with 3.3x10^5 cells using 120 cores) but when I increase the number of cells (2.6x10^6 cells using 1920 cores), I go out of memory in the function MatColoringApply. It shows that one of the cores uses over 11G memory. I think this is unreasonable. Do you have any suggestions for debugging this problem?
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> Rongliang
>>>>>>>>
>>>>>> <massif.out.12562>
>>>> <massif.out.26358>
>> <massif.out.26539>

-------------- next part --------------
desc: (none)
cmd: ./Nwtun -f ./../meshes/3DNACA0012AIRFOIL_Hex300x100x200x160.exo -overlap 1 -options_file options.txt -time_integral_method 1 -initial_time 0.0 -final_time 4.0 -dt 1.e-3 -timestep 2 -max_time_its 0 -includeenergy -reconstruct -PressureFlux -testcase 3DNACA -output_solution -solutionfile results/3DNACA0012AIRFOIL_Hex150x50x100x80_imp -steps_output 4000 -inflow_u 349.02 -viscosity 0.01 -k 0.026 -monitor_step_gap -R 287.02 -adiabatic 1.4 -Euler -Attack 10 -Mach 0.8 -fd_jacobian_color -limiter zero -RiemannSolver Rusanov -CFL 0.9 -reconstruct -second_order -parallel_csr -parallel_output
time_unit: i
#-----------
snapshot=0
#-----------
time=0
mem_heap_B=0
mem_heap_extra_B=0
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=1
#-----------
time=36881410931
mem_heap_B=7458834367
mem_heap_extra_B=33297
mem_stacks_B=0
heap_tree=detailed
n3: 7458834367 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n7: 7302506748 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132)
   n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
    n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23)
     n0: 2544974400 0x4076DF: main (Nwtun.c:55)
  n1: 1357319680 0x462A1D: DMSetUp_Plex (plex.c:1582)
   n1: 1357319680 0x50B88B: DMSetUp (dm.c:560)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 1357319680 0x462AB1: DMSetUp_Plex (plex.c:1583)
   n1: 1357319680 0x50B88B: DMSetUp (dm.c:560)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n2: 682078080 0x52EC32: PetscSectionSetChart (vsectionis.c:389)
   n1: 341039040 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
   n1: 341039040 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n2: 682078080 0x52EBD7: PetscSectionSetChart (vsectionis.c:389)
   n1: 341039040 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
   n1: 341039040 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n1: 678659840 0x44390A: DMPlexCreateExodusNwtun (plexexodusii.c:149)
   n1: 678659840 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
    n1: 678659840 0x4152C9: CreateMesh (SetupMeshes.c:23)
     n0: 678659840 0x4076DF: main (Nwtun.c:55)
  n0: 76988 in 59 places, all below massif's threshold (01.00%)
 n1: 100663336 0x4DBEB6: kh_resize_HASHI (hash.h:510)
  n1: 100663336 0x4DC2CD: kh_put_HASHI (hash.h:510)
   n1: 100663336 0x4E14DD: DMLabelSetValue (plexlabel.c:450)
    n1: 100663336 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909)
     n1: 100663336 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173)
      n1: 100663336 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 100663336 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 100663336 0x4076DF: main (Nwtun.c:55)
 n0: 55664283 in 105 places, all below massif's threshold (01.00%)
#-----------
snapshot=2
#-----------
time=53537892721
mem_heap_B=7614548075
mem_heap_extra_B=37141
mem_stacks_B=0
heap_tree=detailed
n4: 7614548075 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n7: 7302506748 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132)
   n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
    n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23)
     n0: 2544974400 0x4076DF: main (Nwtun.c:55)
  n1: 1357319680 0x462A1D: DMSetUp_Plex (plex.c:1582)
   n1: 1357319680 0x50B88B: DMSetUp (dm.c:560)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 1357319680 0x462AB1: DMSetUp_Plex (plex.c:1583)
   n1: 1357319680 0x50B88B: DMSetUp (dm.c:560)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n2: 682078080 0x52EC32: PetscSectionSetChart (vsectionis.c:389)
   n1: 341039040 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
   n1: 341039040 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n2: 682078080 0x52EBD7: PetscSectionSetChart (vsectionis.c:389)
   n1: 341039040 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
   n1: 341039040 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n1: 678659840 0x44390A: DMPlexCreateExodusNwtun (plexexodusii.c:149)
   n1: 678659840 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
    n1: 678659840 0x4152C9: CreateMesh (SetupMeshes.c:23)
     n0: 678659840 0x4076DF: main (Nwtun.c:55)
  n0: 76988 in 59 places, all below massif's threshold (01.00%)
 n1: 201326744 0x4DBEB6: kh_resize_HASHI (hash.h:510)
  n1: 201326744 0x4DC2CD: kh_put_HASHI (hash.h:510)
   n1: 201326744 0x4E14DD: DMLabelSetValue (plexlabel.c:450)
    n1: 201326744 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909)
     n1: 201326744 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173)
      n1: 201326744 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 201326744 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 201326744 0x4076DF: main (Nwtun.c:55)
 n1: 100663372 0x4DBE90: kh_resize_HASHI (hash.h:510)
  n1: 100663372 0x4DC2CD: kh_put_HASHI (hash.h:510)
   n1: 100663372 0x4E14DD: DMLabelSetValue (plexlabel.c:450)
    n1: 100663372 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909)
     n1: 100663372 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173)
      n1: 100663372 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 100663372 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 100663372 0x4076DF: main (Nwtun.c:55)
 n0: 10051211 in 104 places, all below massif's threshold (01.00%)
#-----------
snapshot=3
#-----------
time=83815558495
mem_heap_B=7623985259
mem_heap_extra_B=41157
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=4
#-----------
time=115170250037
mem_heap_B=11993400907
mem_heap_extra_B=57749
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=5
#-----------
time=204639329496
mem_heap_B=13007979483
mem_heap_extra_B=58317
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=6
#-----------
time=296347515250
mem_heap_B=13007979483
mem_heap_extra_B=58317
mem_stacks_B=0
heap_tree=detailed
n11: 13007979483 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n6: 6623846844 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132)
   n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
    n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23)
     n0: 2544974400 0x4076DF: main (Nwtun.c:55)
  n1: 1357319680 0x462A1D: DMSetUp_Plex (plex.c:1582)
   n1: 1357319680 0x50B88B: DMSetUp (dm.c:560)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 1357319680 0x462AB1: DMSetUp_Plex (plex.c:1583)
   n1: 1357319680 0x50B88B: DMSetUp (dm.c:560)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n2: 682078080 0x52EBD7: PetscSectionSetChart (vsectionis.c:389)
   n1: 341039040 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
   n1: 341039040 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n2: 682078080 0x52EC32: PetscSectionSetChart (vsectionis.c:389)
   n1: 341039040 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
   n1: 341039040 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n0: 76924 in 60 places, all below massif's threshold (01.00%)
 n2: 1868023688 0x13D6C59: gk_malloc (memory.c:147)
  n2: 1526984640 0x13DA642: libmetis__imalloc (gklib.c:24)
   n1: 1357319680 0x13F1019: libmetis__CreateGraphDual (mesh.c:177)
    n1: 1357319680 0x13F0CF7: METIS_MeshToDual (mesh.c:67)
     n1: 1357319680 0x4440CA: DMPlexCreateExodusNwtun (plexexodusii.c:218)
      n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 1357319680 0x4076DF: main (Nwtun.c:55)
   n1: 169664960 0x13F12B6: libmetis__CreateGraphDual (mesh.c:202)
    n1: 169664960 0x13F0CF7: METIS_MeshToDual (mesh.c:67)
     n1: 169664960 0x4440CA: DMPlexCreateExodusNwtun (plexexodusii.c:218)
      n1: 169664960 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 169664960 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 169664960 0x4076DF: main (Nwtun.c:55)
  n2: 341039048 0x13DA6A8: libmetis__ismalloc (gklib.c:24)
   n1: 171374088 0x13F0FF0: libmetis__CreateGraphDual (mesh.c:176)
    n1: 171374088 0x13F0CF7: METIS_MeshToDual (mesh.c:67)
     n1: 171374088 0x4440CA: DMPlexCreateExodusNwtun (plexexodusii.c:218)
      n1: 171374088 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 171374088 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 171374088 0x4076DF: main (Nwtun.c:55)
   n1: 169664960 0x13F129F: libmetis__CreateGraphDual (mesh.c:201)
    n1: 169664960 0x13F0CF7: METIS_MeshToDual (mesh.c:67)
     n1: 169664960 0x4440CA: DMPlexCreateExodusNwtun (plexexodusii.c:218)
      n1: 169664960 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 169664960 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 169664960 0x4076DF: main (Nwtun.c:55)
 n1: 1357319680 0x443F5C: DMPlexCreateExodusNwtun (plexexodusii.c:205)
  n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
   n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
    n0: 1357319680 0x4076DF: main (Nwtun.c:55)
 n1: 1017989760 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202)
  n1: 1017989760 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
   n1: 1017989760 0x4152C9: CreateMesh (SetupMeshes.c:23)
    n0: 1017989760 0x4076DF: main (Nwtun.c:55)
 n1: 1014578576 0x13F145D: libmetis__CreateGraphDual (mesh.c:213)
  n1: 1014578576 0x13F0CF7: METIS_MeshToDual (mesh.c:67)
   n1: 1014578576 0x4440CA: DMPlexCreateExodusNwtun (plexexodusii.c:218)
    n1: 1014578576 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
     n1: 1014578576 0x4152C9: CreateMesh (SetupMeshes.c:23)
      n0: 1014578576 0x4076DF: main (Nwtun.c:55)
 n1: 402653224 0x4DBEB6: kh_resize_HASHI (hash.h:510)
  n1: 402653224 0x4DC2CD: kh_put_HASHI (hash.h:510)
   n1: 402653224 0x4E14DD: DMLabelSetValue (plexlabel.c:450)
    n1: 402653224 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909)
     n1: 402653224 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173)
      n1: 402653224 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 402653224 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 402653224 0x4076DF: main (Nwtun.c:55)
 n1: 201326612 0x4DBE90: kh_resize_HASHI (hash.h:510)
  n1: 201326612 0x4DC2CD: kh_put_HASHI (hash.h:510)
   n1: 201326612 0x4E14DD: DMLabelSetValue (plexlabel.c:450)
    n1: 201326612 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909)
     n1: 201326612 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173)
      n1: 201326612 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 201326612 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 201326612 0x4076DF: main (Nwtun.c:55)
 n1: 169664968 0x443EEF: DMPlexCreateExodusNwtun (plexexodusii.c:201)
  n1: 169664968 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
   n1: 169664968 0x4152C9: CreateMesh (SetupMeshes.c:23)
    n0: 169664968 0x4076DF: main (Nwtun.c:55)
 n1: 169664968 0x443F34: DMPlexCreateExodusNwtun (plexexodusii.c:204)
  n1: 169664968 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
   n1: 169664968 0x4152C9: CreateMesh (SetupMeshes.c:23)
    n0: 169664968 0x4076DF: main (Nwtun.c:55)
 n1: 169664968 0x13F123F: libmetis__CreateGraphDual (mesh.c:195)
  n1: 169664968 0x13F0CF7: METIS_MeshToDual (mesh.c:67)
   n1: 169664968 0x4440CA: DMPlexCreateExodusNwtun (plexexodusii.c:218)
    n1: 169664968 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
     n1: 169664968 0x4152C9: CreateMesh (SetupMeshes.c:23)
      n0: 169664968 0x4076DF: main (Nwtun.c:55)
 n0: 13246195 in 106 places, all below massif's threshold (01.00%)
#-----------
snapshot=7
#-----------
time=302711546355
mem_heap_B=10970241563
mem_heap_extra_B=44581
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=8
#-----------
time=417185571136
mem_heap_B=9785998015
mem_heap_extra_B=40409
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=9
#-----------
time=495477070490
mem_heap_B=9785998015
mem_heap_extra_B=40409
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=10
#-----------
time=521966018039
mem_heap_B=9785998133
mem_heap_extra_B=40459
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=11
#-----------
time=556872466967
mem_heap_B=10106862677
mem_heap_extra_B=56411
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=12
#-----------
time=582193895580
mem_heap_B=10710842569
mem_heap_extra_B=64431
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=13
#-----------
time=601361206419
mem_heap_B=11019123741
mem_heap_extra_B=64603
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=14
#-----------
time=630993261443
mem_heap_B=11019123757
mem_heap_extra_B=64611
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=15
#-----------
time=667400928559
mem_heap_B=10834836067
mem_heap_extra_B=50781
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=16
#-----------
time=688635956717
mem_heap_B=11542625655
mem_heap_extra_B=54009
mem_stacks_B=0
heap_tree=detailed
n8: 11542625655 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n8: 8322215448 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132)
   n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
    n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23)
     n0: 2544974400 0x4076DF: main (Nwtun.c:55)
  n1: 1357319680 0x462AB1: DMSetUp_Plex (plex.c:1583)
   n1: 1357319680 0x50B88B: DMSetUp (dm.c:560)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 1357319680 0x46371F: DMPlexSymmetrize (plex.c:1655)
   n1: 1357319680 0x444320: DMPlexCreateExodusNwtun (plexexodusii.c:247)
    n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
     n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
      n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 1357319680 0x462A1D: DMSetUp_Plex (plex.c:1582)
   n1: 1357319680 0x50B88B: DMSetUp (dm.c:560)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n2: 682078080 0x52EBD7: PetscSectionSetChart (vsectionis.c:389)
   n1: 341039040 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
   n1: 341039040 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n2: 682078080 0x52EC32: PetscSectionSetChart (vsectionis.c:389)
   n1: 341039040 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
   n1: 341039040 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n1: 341039040 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46)
   n1: 341039040 0x4E1E75: DMLabelGetNumValues (plexlabel.c:504)
    n1: 341039040 0x473CE3: DMPlexGetDepth (plex.c:3971)
     n1: 341039040 0x9E510C: DMPlexInterpolate (plexinterpolate.c:335)
      n1: 341039040 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n0: 86808 in 68 places, all below massif's threshold (01.00%)
 n1: 1017989760 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202)
  n1: 1017989760 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
   n1: 1017989760 0x4152C9: CreateMesh (SetupMeshes.c:23)
    n0: 1017989760 0x4076DF: main (Nwtun.c:55)
 n1: 805306976 0x9E01D5: kh_resize_HASHIJKL (hash.h:1199)
  n1: 805306976 0x9E07CD: kh_put_HASHIJKL (hash.h:1199)
   n1: 805306976 0x9E1116: PetscHashIJKLPut (hash.h:1262)
    n1: 805306976 0x9E3866: DMPlexInterpolateFaces_Internal (plexinterpolate.c:195)
     n1: 805306976 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 805306976 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 805306976 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 805306976 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 805306976 0x4076DF: main (Nwtun.c:55)
 n1: 603980232 0x9E020C: kh_resize_HASHIJKL (hash.h:1199)
  n1: 603980232 0x9E07CD: kh_put_HASHIJKL (hash.h:1199)
   n1: 603980232 0x9E1116: PetscHashIJKLPut (hash.h:1262)
    n1: 603980232 0x9E3866: DMPlexInterpolateFaces_Internal (plexinterpolate.c:195)
     n1: 603980232 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 603980232 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 603980232 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 603980232 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 603980232 0x4076DF: main (Nwtun.c:55)
 n1: 402653224 0x4DBEB6: kh_resize_HASHI (hash.h:510)
  n1: 402653224 0x4DC2CD: kh_put_HASHI (hash.h:510)
   n2: 402653224 0x4E14DD: DMLabelSetValue (plexlabel.c:450)
    n1: 402653224 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909)
     n1: 402653224 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173)
      n1: 402653224 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 402653224 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 402653224 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 2 places, all below massif's threshold (01.00%)
 n1: 201326612 0x4DBE90: kh_resize_HASHI (hash.h:510)
  n1: 201326612 0x4DC2CD: kh_put_HASHI (hash.h:510)
   n2: 201326612 0x4E14DD: DMLabelSetValue (plexlabel.c:450)
    n1: 201326612 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909)
     n1: 201326612 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173)
      n1: 201326612 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 201326612 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 201326612 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 2 places, all below massif's threshold (01.00%)
 n1: 169664968 0x443EEF: DMPlexCreateExodusNwtun (plexexodusii.c:201)
  n1: 169664968 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
   n1: 169664968 0x4152C9: CreateMesh (SetupMeshes.c:23)
    n0: 169664968 0x4076DF: main (Nwtun.c:55)
 n0: 19488435 in 113 places, all below massif's threshold (01.00%)
#-----------
snapshot=17
#-----------
time=727470680494
mem_heap_B=11555208571
mem_heap_extra_B=58045
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=18
#-----------
time=731287750178
mem_heap_B=12958202467
mem_heap_extra_B=54797
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=19
#-----------
time=808892033505
mem_heap_B=12983368299
mem_heap_extra_B=58829
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=20
#-----------
time=816421281943
mem_heap_B=15801941595
mem_heap_extra_B=57821
mem_stacks_B=0
heap_tree=detailed
n8: 15801941595 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n8: 8322215448 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132)
   n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
    n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23)
     n0: 2544974400 0x4076DF: main (Nwtun.c:55)
  n1: 1357319680 0x462AB1: DMSetUp_Plex (plex.c:1583)
   n1: 1357319680 0x50B88B: DMSetUp (dm.c:560)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 1357319680 0x46371F: DMPlexSymmetrize (plex.c:1655)
   n1: 1357319680 0x444320: DMPlexCreateExodusNwtun (plexexodusii.c:247)
    n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
     n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
      n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 1357319680 0x462A1D: DMSetUp_Plex (plex.c:1582)
   n1: 1357319680 0x50B88B: DMSetUp (dm.c:560)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n2: 682078080 0x52EBD7: PetscSectionSetChart (vsectionis.c:389)
   n1: 341039040 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
   n1: 341039040 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n2: 682078080 0x52EC32: PetscSectionSetChart (vsectionis.c:389)
   n1: 341039040 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
   n1: 341039040 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n1: 341039040 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46)
   n1: 341039040 0x4E1E75: DMLabelGetNumValues (plexlabel.c:504)
    n1: 341039040 0x473CE3: DMPlexGetDepth (plex.c:3971)
     n1: 341039040 0x9E510C: DMPlexInterpolate (plexinterpolate.c:335)
      n1: 341039040 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n0: 86808 in 68 places, all below massif's threshold (01.00%)
 n1: 3221226208 0x9E01D5: kh_resize_HASHIJKL (hash.h:1199)
  n1: 3221226208 0x9E07CD: kh_put_HASHIJKL (hash.h:1199)
   n1: 3221226208 0x9E1116: PetscHashIJKLPut (hash.h:1262)
    n1: 3221226208 0x9E3866: DMPlexInterpolateFaces_Internal (plexinterpolate.c:195)
     n1: 3221226208 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 3221226208 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 3221226208 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 3221226208 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 3221226208 0x4076DF: main (Nwtun.c:55)
 n1: 2415919656 0x9E020C: kh_resize_HASHIJKL (hash.h:1199)
  n1: 2415919656 0x9E07CD: kh_put_HASHIJKL (hash.h:1199)
   n1: 2415919656 0x9E1116: PetscHashIJKLPut (hash.h:1262)
    n1: 2415919656 0x9E3866: DMPlexInterpolateFaces_Internal (plexinterpolate.c:195)
     n1: 2415919656 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 2415919656 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 2415919656 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 2415919656 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 2415919656 0x4076DF: main (Nwtun.c:55)
 n1: 1017989760 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202)
  n1: 1017989760 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
   n1: 1017989760 0x4152C9: CreateMesh (SetupMeshes.c:23)
    n0: 1017989760 0x4076DF: main (Nwtun.c:55)
 n1: 402653224 0x4DBEB6: kh_resize_HASHI (hash.h:510)
  n1: 402653224 0x4DC2CD: kh_put_HASHI (hash.h:510)
   n2: 402653224 0x4E14DD: DMLabelSetValue (plexlabel.c:450)
    n1: 402653224 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909)
     n1: 402653224 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173)
      n1: 402653224 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 402653224 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 402653224 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 2 places, all below massif's threshold (01.00%)
 n1: 201326612 0x4DBE90: kh_resize_HASHI (hash.h:510)
  n1: 201326612 0x4DC2CD: kh_put_HASHI (hash.h:510)
   n2: 201326612 0x4E14DD: DMLabelSetValue (plexlabel.c:450)
    n1: 201326612 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909)
     n1: 201326612 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173)
      n1: 201326612 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 201326612 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 201326612 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 2 places, all below massif's threshold (01.00%)
 n1: 169664968 0x443EEF: DMPlexCreateExodusNwtun (plexexodusii.c:201)
  n1: 169664968 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
   n1: 169664968 0x4152C9: CreateMesh (SetupMeshes.c:23)
    n0: 169664968 0x4076DF: main (Nwtun.c:55)
 n0: 50945719 in 113 places, all below massif's threshold (01.00%)
#-----------
snapshot=21
#-----------
time=915052183074
mem_heap_B=12568132471
mem_heap_extra_B=50481
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=22
#-----------
time=1053918439822
mem_heap_B=16594796631
mem_heap_extra_B=57177
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=23
#-----------
time=1079607070946
mem_heap_B=20010269895
mem_heap_extra_B=66241
mem_stacks_B=0
heap_tree=detailed
n6: 20010269895 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n8: 17850756792 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 4418111328 0x462A1D: DMSetUp_Plex (plex.c:1582)
   n2: 4418111328 0x50B88B: DMSetUp (dm.c:560)
    n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229)
     n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 3060791648 0x4076DF: main (Nwtun.c:55)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 4418111328 0x462AB1: DMSetUp_Plex (plex.c:1583)
   n2: 4418111328 0x50B88B: DMSetUp (dm.c:560)
    n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229)
     n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 3060791648 0x4076DF: main (Nwtun.c:55)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132)
   n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
    n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23)
     n0: 2544974400 0x4076DF: main (Nwtun.c:55)
  n2: 2385557104 0x52EBD7: PetscSectionSetChart (vsectionis.c:389)
   n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
   n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n2: 2385557104 0x52EC32: PetscSectionSetChart (vsectionis.c:389)
   n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
   n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n1: 1357319680 0x46371F: DMPlexSymmetrize (plex.c:1655)
   n1: 1357319680 0x444320: DMPlexCreateExodusNwtun (plexexodusii.c:247)
    n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
     n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
      n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 341039040 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46)
   n1: 341039040 0x4E1E75: DMLabelGetNumValues (plexlabel.c:504)
    n1: 341039040 0x473CE3: DMPlexGetDepth (plex.c:3971)
     n1: 341039040 0x9E510C: DMPlexInterpolate (plexinterpolate.c:335)
      n1: 341039040 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n0: 86808 in 68 places, all below massif's threshold (01.00%)
 n1: 1017989760 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202)
  n1: 1017989760 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
   n1: 1017989760 0x4152C9: CreateMesh (SetupMeshes.c:23)
    n0: 1017989760 0x4076DF: main (Nwtun.c:55)
 n1: 402653224 0x4DBEB6: kh_resize_HASHI (hash.h:510)
  n1: 402653224 0x4DC2CD: kh_put_HASHI (hash.h:510)
   n2: 402653224 0x4E14DD: DMLabelSetValue (plexlabel.c:450)
    n1: 402653224 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909)
     n1: 402653224 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173)
      n1: 402653224 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 402653224 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 402653224 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 2 places, all below massif's threshold (01.00%)
 n0: 336216499 in 115 places, all below massif's threshold (01.00%)
 n1: 201327008 0x9E01D5: kh_resize_HASHIJKL (hash.h:1199)
  n1: 201327008 0x9E07CD: kh_put_HASHIJKL (hash.h:1199)
   n2: 201327008 0x9E1116: PetscHashIJKLPut (hash.h:1262)
    n1: 201327008 0x9E43F2: DMPlexInterpolateFaces_Internal (plexinterpolate.c:264)
     n1: 201327008 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 201327008 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 201327008 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 201327008 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 201327008 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
 n1: 201326612 0x4DBE90: kh_resize_HASHI (hash.h:510)
  n1: 201326612 0x4DC2CD: kh_put_HASHI (hash.h:510)
   n2: 201326612 0x4E14DD: DMLabelSetValue (plexlabel.c:450)
    n1: 201326612 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909)
     n1: 201326612 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173)
      n1: 201326612 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 201326612 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 201326612 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 2 places, all below massif's threshold (01.00%)
#-----------
snapshot=24
#-----------
time=1106254629312
mem_heap_B=20364950279
mem_heap_extra_B=66689
mem_stacks_B=0
heap_tree=detailed
n6: 20364950279 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n8: 17850756792 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 4418111328 0x462A1D: DMSetUp_Plex (plex.c:1582)
   n2: 4418111328 0x50B88B: DMSetUp (dm.c:560)
    n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229)
     n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 3060791648 0x4076DF: main (Nwtun.c:55)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 4418111328 0x462AB1: DMSetUp_Plex (plex.c:1583)
   n2: 4418111328 0x50B88B: DMSetUp (dm.c:560)
    n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229)
     n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 3060791648 0x4076DF: main (Nwtun.c:55)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132)
   n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
    n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23)
     n0: 2544974400 0x4076DF: main (Nwtun.c:55)
  n2: 2385557104 0x52EBD7: PetscSectionSetChart (vsectionis.c:389)
   n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
   n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n2: 2385557104 0x52EC32: PetscSectionSetChart (vsectionis.c:389)
   n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
   n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n1: 1357319680 0x46371F: DMPlexSymmetrize (plex.c:1655)
   n1: 1357319680 0x444320: DMPlexCreateExodusNwtun (plexexodusii.c:247)
    n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
     n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
      n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 341039040 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46)
   n1: 341039040 0x4E1E75: DMLabelGetNumValues (plexlabel.c:504)
    n1: 341039040 0x473CE3: DMPlexGetDepth (plex.c:3971)
     n1: 341039040 0x9E510C: DMPlexInterpolate (plexinterpolate.c:335)
      n1: 341039040 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n0: 86808 in 68 places, all below massif's threshold (01.00%)
 n1: 1017989760 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202)
  n1: 1017989760 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
   n1: 1017989760 0x4152C9: CreateMesh (SetupMeshes.c:23)
    n0: 1017989760 0x4076DF: main (Nwtun.c:55)
 n1: 402653344 0x9E01D5: kh_resize_HASHIJKL (hash.h:1199)
  n1: 402653344 0x9E07CD: kh_put_HASHIJKL (hash.h:1199)
   n2: 402653344 0x9E1116: PetscHashIJKLPut (hash.h:1262)
    n1: 402653344 0x9E43F2: DMPlexInterpolateFaces_Internal (plexinterpolate.c:264)
     n1: 402653344 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 402653344 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 402653344 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 402653344 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 402653344 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
 n1: 402653224 0x4DBEB6: kh_resize_HASHI (hash.h:510)
  n1: 402653224 0x4DC2CD: kh_put_HASHI (hash.h:510)
   n2: 402653224 0x4E14DD: DMLabelSetValue (plexlabel.c:450)
    n1: 402653224 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909)
     n1: 402653224 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173)
      n1: 402653224 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 402653224 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 402653224 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 2 places, all below massif's threshold (01.00%)
 n0: 388907151 in 115 places, all below massif's threshold (01.00%)
 n1: 301990008 0x9E020C: kh_resize_HASHIJKL (hash.h:1199)
  n1: 301990008 0x9E07CD: kh_put_HASHIJKL (hash.h:1199)
   n2: 301990008 0x9E1116: PetscHashIJKLPut (hash.h:1262)
    n1: 301990008 0x9E43F2: DMPlexInterpolateFaces_Internal (plexinterpolate.c:264)
     n1: 301990008 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 301990008 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 301990008 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 301990008 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 301990008 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
#-----------
snapshot=25
#-----------
time=1157665534790
mem_heap_B=20369668875
mem_heap_extra_B=70701
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=26
#-----------
time=1159549737997
mem_heap_B=21074312731
mem_heap_extra_B=69917
mem_stacks_B=0
heap_tree=detailed
n6: 21074312731 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n8: 17850756792 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 4418111328 0x462A1D: DMSetUp_Plex (plex.c:1582)
   n2: 4418111328 0x50B88B: DMSetUp (dm.c:560)
    n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229)
     n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 3060791648 0x4076DF: main (Nwtun.c:55)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 4418111328 0x462AB1: DMSetUp_Plex (plex.c:1583)
   n2: 4418111328 0x50B88B: DMSetUp (dm.c:560)
    n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229)
     n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 3060791648 0x4076DF: main (Nwtun.c:55)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132)
   n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
    n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23)
     n0: 2544974400 0x4076DF: main (Nwtun.c:55)
  n2: 2385557104 0x52EBD7: PetscSectionSetChart (vsectionis.c:389)
   n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
   n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n2: 2385557104 0x52EC32: PetscSectionSetChart (vsectionis.c:389)
   n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
   n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n1: 1357319680 0x46371F: DMPlexSymmetrize (plex.c:1655)
   n1: 1357319680 0x444320: DMPlexCreateExodusNwtun (plexexodusii.c:247)
    n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
     n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
      n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 341039040 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46)
   n1: 341039040 0x4E1E75: DMLabelGetNumValues (plexlabel.c:504)
    n1: 341039040 0x473CE3: DMPlexGetDepth (plex.c:3971)
     n1: 341039040 0x9E510C: DMPlexInterpolate (plexinterpolate.c:335)
      n1: 341039040 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n0: 86808 in 68 places, all below massif's threshold (01.00%)
 n1: 1017989760 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202)
  n1: 1017989760 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
   n1: 1017989760 0x4152C9: CreateMesh (SetupMeshes.c:23)
    n0: 1017989760 0x4076DF: main (Nwtun.c:55)
 n1: 805306976 0x9E01D5: kh_resize_HASHIJKL (hash.h:1199)
  n1: 805306976 0x9E07CD: kh_put_HASHIJKL (hash.h:1199)
   n2: 805306976 0x9E1116: PetscHashIJKLPut (hash.h:1262)
    n1: 805306976 0x9E43F2: DMPlexInterpolateFaces_Internal (plexinterpolate.c:264)
     n1: 805306976 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 805306976 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 805306976 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 805306976 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 805306976 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
 n1: 603980232 0x9E020C: kh_resize_HASHIJKL (hash.h:1199)
  n1: 603980232 0x9E07CD: kh_put_HASHIJKL (hash.h:1199)
   n2: 603980232 0x9E1116: PetscHashIJKLPut (hash.h:1262)
    n1: 603980232 0x9E43F2: DMPlexInterpolateFaces_Internal (plexinterpolate.c:264)
     n1: 603980232 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 603980232 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 603980232 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 603980232 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 603980232 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
 n1: 402653224 0x4DBEB6: kh_resize_HASHI (hash.h:510)
  n1: 402653224 0x4DC2CD: kh_put_HASHI (hash.h:510)
   n2: 402653224 0x4E14DD: DMLabelSetValue (plexlabel.c:450)
    n1: 402653224 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909)
     n1: 402653224 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173)
      n1: 402653224 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 402653224 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 402653224 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 2 places, all below massif's threshold (01.00%)
 n0: 393625747 in 115 places, all below massif's threshold (01.00%)
#-----------
snapshot=27
#-----------
time=1262479157617
mem_heap_B=21083749915
mem_heap_extra_B=73933
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=28
#-----------
time=1266296227301
mem_heap_B=22493035275
mem_heap_extra_B=74717
mem_stacks_B=0
heap_tree=detailed
n6: 22493035275 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n8: 17850756792 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 4418111328 0x462A1D: DMSetUp_Plex (plex.c:1582)
   n2: 4418111328 0x50B88B: DMSetUp (dm.c:560)
    n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229)
     n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 3060791648 0x4076DF: main (Nwtun.c:55)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 4418111328 0x462AB1: DMSetUp_Plex (plex.c:1583)
   n2: 4418111328 0x50B88B: DMSetUp (dm.c:560)
    n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229)
     n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 3060791648 0x4076DF: main (Nwtun.c:55)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132)
   n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
    n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23)
     n0: 2544974400 0x4076DF: main (Nwtun.c:55)
  n2: 2385557104 0x52EBD7: PetscSectionSetChart (vsectionis.c:389)
   n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
   n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n2: 2385557104 0x52EC32: PetscSectionSetChart (vsectionis.c:389)
   n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
   n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n1: 1357319680 0x46371F: DMPlexSymmetrize (plex.c:1655)
   n1: 1357319680 0x444320: DMPlexCreateExodusNwtun (plexexodusii.c:247)
    n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
     n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
      n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 341039040 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46)
   n1: 341039040 0x4E1E75: DMLabelGetNumValues (plexlabel.c:504)
    n1: 341039040 0x473CE3: DMPlexGetDepth (plex.c:3971)
     n1: 341039040 0x9E510C: DMPlexInterpolate (plexinterpolate.c:335)
      n1: 341039040 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n0: 86808 in 68 places, all below massif's threshold (01.00%)
 n1: 1610612896 0x9E01D5: kh_resize_HASHIJKL (hash.h:1199)
  n1: 1610612896 0x9E07CD: kh_put_HASHIJKL (hash.h:1199)
   n2: 1610612896 0x9E1116: PetscHashIJKLPut (hash.h:1262)
    n1: 1610612896 0x9E43F2: DMPlexInterpolateFaces_Internal (plexinterpolate.c:264)
     n1: 1610612896 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 1610612896 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 1610612896 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 1610612896 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 1610612896 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
 n1: 1207959672 0x9E020C: kh_resize_HASHIJKL (hash.h:1199)
  n1: 1207959672 0x9E07CD: kh_put_HASHIJKL (hash.h:1199)
   n2: 1207959672 0x9E1116: PetscHashIJKLPut (hash.h:1262)
    n1: 1207959672 0x9E43F2: DMPlexInterpolateFaces_Internal (plexinterpolate.c:264)
     n1: 1207959672 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 1207959672 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 1207959672 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 1207959672 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 1207959672 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
 n1: 1017989760 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202)
  n1: 1017989760 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
   n1: 1017989760 0x4152C9: CreateMesh (SetupMeshes.c:23)
    n0: 1017989760 0x4076DF: main (Nwtun.c:55)
 n0: 403062931 in 115 places, all below massif's threshold (01.00%)
 n1: 402653224 0x4DBEB6: kh_resize_HASHI (hash.h:510)
  n1: 402653224 0x4DC2CD: kh_put_HASHI (hash.h:510)
   n2: 402653224 0x4E14DD: DMLabelSetValue (plexlabel.c:450)
    n1: 402653224 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909)
     n1: 402653224 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173)
      n1: 402653224 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 402653224 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 402653224 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 2 places, all below massif's threshold (01.00%)
#-----------
snapshot=29
#-----------
time=1472102781759
mem_heap_B=22511909643
mem_heap_extra_B=74717
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=30
#-----------
time=1479632030197
mem_heap_B=25330482939
mem_heap_extra_B=73709
mem_stacks_B=0
heap_tree=detailed
n6: 25330482939 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n8: 17850756792 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 4418111328 0x462A1D: DMSetUp_Plex (plex.c:1582)
   n2: 4418111328 0x50B88B: DMSetUp (dm.c:560)
    n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229)
     n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 3060791648 0x4076DF: main (Nwtun.c:55)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 4418111328 0x462AB1: DMSetUp_Plex (plex.c:1583)
   n2: 4418111328 0x50B88B: DMSetUp (dm.c:560)
    n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229)
     n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 3060791648 0x4076DF: main (Nwtun.c:55)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132)
   n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
    n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23)
     n0: 2544974400 0x4076DF: main (Nwtun.c:55)
  n2: 2385557104 0x52EBD7: PetscSectionSetChart (vsectionis.c:389)
   n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
   n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n2: 2385557104 0x52EC32: PetscSectionSetChart (vsectionis.c:389)
   n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
   n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n1: 1357319680 0x46371F: DMPlexSymmetrize (plex.c:1655)
   n1: 1357319680 0x444320: DMPlexCreateExodusNwtun (plexexodusii.c:247)
    n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
     n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
      n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 341039040 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46)
   n1: 341039040 0x4E1E75: DMLabelGetNumValues (plexlabel.c:504)
    n1: 341039040 0x473CE3: DMPlexGetDepth (plex.c:3971)
     n1: 341039040 0x9E510C: DMPlexInterpolate (plexinterpolate.c:335)
      n1: 341039040 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n0: 86808 in 68 places, all below massif's threshold (01.00%)
 n1: 3221226208 0x9E01D5: kh_resize_HASHIJKL (hash.h:1199)
  n1: 3221226208 0x9E07CD: kh_put_HASHIJKL (hash.h:1199)
   n2: 3221226208 0x9E1116: PetscHashIJKLPut (hash.h:1262)
    n1: 3221226208 0x9E43F2: DMPlexInterpolateFaces_Internal (plexinterpolate.c:264)
     n1: 3221226208 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 3221226208 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 3221226208 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 3221226208 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 3221226208 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
 n1: 2415919656 0x9E020C: kh_resize_HASHIJKL (hash.h:1199)
  n1: 2415919656 0x9E07CD: kh_put_HASHIJKL (hash.h:1199)
   n2: 2415919656 0x9E1116: PetscHashIJKLPut (hash.h:1262)
    n1: 2415919656 0x9E43F2: DMPlexInterpolateFaces_Internal (plexinterpolate.c:264)
     n1: 2415919656 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 2415919656 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 2415919656 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 2415919656 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 2415919656 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
 n1: 1017989760 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202)
  n1: 1017989760 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
   n1: 1017989760 0x4152C9: CreateMesh (SetupMeshes.c:23)
    n0: 1017989760 0x4076DF: main (Nwtun.c:55)
 n0: 421937299 in 115 places, all below massif's threshold (01.00%)
 n1: 402653224 0x4DBEB6: kh_resize_HASHI (hash.h:510)
  n1: 402653224 0x4DC2CD: kh_put_HASHI (hash.h:510)
   n2: 402653224 0x4E14DD: DMLabelSetValue (plexlabel.c:450)
    n1: 402653224 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909)
     n1: 402653224 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173)
      n1: 402653224 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 402653224 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 402653224 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 2 places, all below massif's threshold (01.00%)
#-----------
snapshot=31
#-----------
time=1744264900288
mem_heap_B=25317899999
mem_heap_extra_B=69657
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=32
#-----------
time=2012308656155
mem_heap_B=22716379879
mem_heap_extra_B=60425
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=33
#-----------
time=2194841265212
mem_heap_B=22716379879
mem_heap_extra_B=60425
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=34
#-----------
time=2221330212467
mem_heap_B=22716379997
mem_heap_extra_B=60491
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=35
#-----------
time=2259009892248
mem_heap_B=23332942797
mem_heap_extra_B=72579
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=36
#-----------
time=2341996204267
mem_heap_B=23332942805
mem_heap_extra_B=72595
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=37
#-----------
time=2359064457778
mem_heap_B=23675692019
mem_heap_extra_B=77781
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=38
#-----------
time=2384204482868
mem_heap_B=23812599551
mem_heap_extra_B=87121
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=39
#-----------
time=2409252239559
mem_heap_B=24120880723
mem_heap_extra_B=87293
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=40
#-----------
time=2453442168129
mem_heap_B=24146046555
mem_heap_extra_B=91325
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=41
#-----------
time=2458984993722
mem_heap_B=24737443631
mem_heap_extra_B=87073
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=42
#-----------
time=2516474050926
mem_heap_B=24566069551
mem_heap_extra_B=84569
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=43
#-----------
time=2548399887187
mem_heap_B=23398454597
mem_heap_extra_B=64107
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=44
#-----------
time=2576502409399
mem_heap_B=25771427531
mem_heap_extra_B=93285
mem_stacks_B=0
heap_tree=detailed
n5: 25771427531 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n9: 22104324620 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 4418111328 0x462AB1: DMSetUp_Plex (plex.c:1583)
   n2: 4418111328 0x50B88B: DMSetUp (dm.c:560)
    n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229)
     n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 3060791648 0x4076DF: main (Nwtun.c:55)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n2: 4418111328 0x46371F: DMPlexSymmetrize (plex.c:1655)
   n1: 3060791648 0x9E4D06: DMPlexInterpolateFaces_Internal (plexinterpolate.c:303)
    n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
     n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
      n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 3060791648 0x4076DF: main (Nwtun.c:55)
   n1: 1357319680 0x444320: DMPlexCreateExodusNwtun (plexexodusii.c:247)
    n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
     n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
      n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 4418111328 0x462A1D: DMSetUp_Plex (plex.c:1582)
   n2: 4418111328 0x50B88B: DMSetUp (dm.c:560)
    n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229)
     n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 3060791648 0x4076DF: main (Nwtun.c:55)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132)
   n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
    n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23)
     n0: 2544974400 0x4076DF: main (Nwtun.c:55)
  n2: 2385557104 0x52EBD7: PetscSectionSetChart (vsectionis.c:389)
   n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
   n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n2: 2385557104 0x52EC32: PetscSectionSetChart (vsectionis.c:389)
   n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
   n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n2: 1023113592 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46)
   n2: 682074552 0x4E2DE5: DMLabelGetStratumIS (plexlabel.c:571)
    n1: 682074552 0x46481D: DMPlexStratify (plex.c:1768)
     n1: 682074552 0x9E4D5E: DMPlexInterpolateFaces_Internal (plexinterpolate.c:304)
      n1: 682074552 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
       n1: 682074552 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
        n1: 682074552 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
         n1: 682074552 0x4152C9: CreateMesh (SetupMeshes.c:23)
          n0: 682074552 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
   n1: 341039040 0x4E1E75: DMLabelGetNumValues (plexlabel.c:504)
    n1: 341039040 0x473CE3: DMPlexGetDepth (plex.c:3971)
     n1: 341039040 0x9E510C: DMPlexInterpolate (plexinterpolate.c:335)
      n1: 341039040 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n1: 510700472 0x64DFA9: ISGeneralSetIndices_General (general.c:681)
   n1: 510700472 0x64DA50: ISGeneralSetIndices (general.c:663)
    n1: 510700472 0x64D730: ISCreateGeneral (general.c:634)
     n2: 510700472 0x4E2EC9: DMLabelGetStratumIS (plexlabel.c:576)
      n1: 510700472 0x46481D: DMPlexStratify (plex.c:1768)
       n1: 510700472 0x9E4D5E: DMPlexInterpolateFaces_Internal (plexinterpolate.c:304)
        n1: 510700472 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
         n1: 510700472 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
          n1: 510700472 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
           n1: 510700472 0x4152C9: CreateMesh (SetupMeshes.c:23)
            n0: 510700472 0x4076DF: main (Nwtun.c:55)
      n0: 0 in 1 place, below massif's threshold (01.00%)
  n0: 87964 in 75 places, all below massif's threshold (01.00%)
 n1: 1610613000 0x4DBEB6: kh_resize_HASHI (hash.h:510)
  n2: 1610613000 0x4DC2CD: kh_put_HASHI (hash.h:510)
   n1: 1207959776 0x4DD75D: DMLabelMakeInvalid_Private (plexlabel.c:85)
    n1: 1207959776 0x4E1115: DMLabelSetValue (plexlabel.c:425)
     n1: 1207959776 0x4646BF: DMPlexStratify (plex.c:1762)
      n1: 1207959776 0x9E4D5E: DMPlexInterpolateFaces_Internal (plexinterpolate.c:304)
       n1: 1207959776 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
        n1: 1207959776 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
         n1: 1207959776 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
          n1: 1207959776 0x4152C9: CreateMesh (SetupMeshes.c:23)
           n0: 1207959776 0x4076DF: main (Nwtun.c:55)
   n2: 402653224 0x4E14DD: DMLabelSetValue (plexlabel.c:450)
    n1: 402653224 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909)
     n1: 402653224 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173)
      n1: 402653224 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 402653224 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 402653224 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 3 places, all below massif's threshold (01.00%)
 n1: 1017989760 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202)
  n1: 1017989760 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
   n1: 1017989760 0x4152C9: CreateMesh (SetupMeshes.c:23)
    n0: 1017989760 0x4076DF: main (Nwtun.c:55)
 n1: 805306500 0x4DBE90: kh_resize_HASHI (hash.h:510)
  n2: 805306500 0x4DC2CD: kh_put_HASHI (hash.h:510)
   n1: 603979888 0x4DD75D: DMLabelMakeInvalid_Private (plexlabel.c:85)
    n1: 603979888 0x4E1115: DMLabelSetValue (plexlabel.c:425)
     n1: 603979888 0x4646BF: DMPlexStratify (plex.c:1762)
      n1: 603979888 0x9E4D5E: DMPlexInterpolateFaces_Internal (plexinterpolate.c:304)
       n1: 603979888 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
        n1: 603979888 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
         n1: 603979888 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
          n1: 603979888 0x4152C9: CreateMesh (SetupMeshes.c:23)
           n0: 603979888 0x4076DF: main (Nwtun.c:55)
   n0: 201326612 in 1 place, below massif's threshold (01.00%)
 n0: 233193651 in 116 places, all below massif's threshold (01.00%)
#-----------
snapshot=45
#-----------
time=2612741728105
mem_heap_B=25156986307
mem_heap_extra_B=93605
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=46
#-----------
time=2647328222574
mem_heap_B=25385051659
mem_heap_extra_B=97525
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=47
#-----------
time=2711933920814
mem_heap_B=25397634575
mem_heap_extra_B=101561
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=48
#-----------
time=2729611625174
mem_heap_B=26034370889
mem_heap_extra_B=99663
mem_stacks_B=0
heap_tree=detailed
n5: 26034370889 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n8: 21763288110 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 4418111328 0x462AB1: DMSetUp_Plex (plex.c:1583)
   n2: 4418111328 0x50B88B: DMSetUp (dm.c:560)
    n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229)
     n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 3060791648 0x4076DF: main (Nwtun.c:55)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n2: 4418111328 0x46371F: DMPlexSymmetrize (plex.c:1655)
   n1: 3060791648 0x9E4D06: DMPlexInterpolateFaces_Internal (plexinterpolate.c:303)
    n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
     n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
      n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 3060791648 0x4076DF: main (Nwtun.c:55)
   n1: 1357319680 0x444320: DMPlexCreateExodusNwtun (plexexodusii.c:247)
    n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
     n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
      n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 4418111328 0x462A1D: DMSetUp_Plex (plex.c:1582)
   n2: 4418111328 0x50B88B: DMSetUp (dm.c:560)
    n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229)
     n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 3060791648 0x4076DF: main (Nwtun.c:55)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132)
   n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
    n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23)
     n0: 2544974400 0x4076DF: main (Nwtun.c:55)
  n2: 2385557104 0x52EBD7: PetscSectionSetChart (vsectionis.c:389)
   n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
   n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n2: 2385557104 0x52EC32: PetscSectionSetChart (vsectionis.c:389)
   n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
   n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n2: 1192778552 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46)
   n2: 851739512 0x4E2DE5: DMLabelGetStratumIS (plexlabel.c:571)
    n1: 851739512 0x46481D: DMPlexStratify (plex.c:1768)
     n1: 851739512 0x9E4D5E: DMPlexInterpolateFaces_Internal (plexinterpolate.c:304)
      n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
       n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
        n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
         n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
          n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
   n1: 341039040 0x4E1E75: DMLabelGetNumValues (plexlabel.c:504)
    n1: 341039040 0x473CE3: DMPlexGetDepth (plex.c:3971)
     n1: 341039040 0x9E510C: DMPlexInterpolate (plexinterpolate.c:335)
      n1: 341039040 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n0: 86966 in 76 places, all below massif's threshold (01.00%)
 n1: 2013266224 0x4DBEB6: kh_resize_HASHI (hash.h:510)
  n2: 2013266224 0x4DC2CD: kh_put_HASHI (hash.h:510)
   n1: 1207959776 0x4DD75D: DMLabelMakeInvalid_Private (plexlabel.c:85)
    n1: 1207959776 0x4E1115: DMLabelSetValue (plexlabel.c:425)
     n1: 1207959776 0x4646BF: DMPlexStratify (plex.c:1762)
      n1: 1207959776 0x9E4D5E: DMPlexInterpolateFaces_Internal (plexinterpolate.c:304)
       n1: 1207959776 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
        n1: 1207959776 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
         n1: 1207959776 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
          n1: 1207959776 0x4152C9: CreateMesh (SetupMeshes.c:23)
           n0: 1207959776 0x4076DF: main (Nwtun.c:55)
   n3: 805306448 0x4E14DD: DMLabelSetValue (plexlabel.c:450)
    n1: 402653224 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909)
     n1: 402653224 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173)
      n1: 402653224 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 402653224 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 402653224 0x4076DF: main (Nwtun.c:55)
    n1: 402653224 0x4646BF: DMPlexStratify (plex.c:1762)
     n1: 402653224 0x9E4D5E: DMPlexInterpolateFaces_Internal (plexinterpolate.c:304)
      n1: 402653224 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
       n1: 402653224 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
        n1: 402653224 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
         n1: 402653224 0x4152C9: CreateMesh (SetupMeshes.c:23)
          n0: 402653224 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 2 places, all below massif's threshold (01.00%)
 n1: 1017989760 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202)
  n1: 1017989760 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
   n1: 1017989760 0x4152C9: CreateMesh (SetupMeshes.c:23)
    n0: 1017989760 0x4076DF: main (Nwtun.c:55)
 n1: 1006633112 0x4DBE90: kh_resize_HASHI (hash.h:510)
  n2: 1006633112 0x4DC2CD: kh_put_HASHI (hash.h:510)
   n1: 603979888 0x4DD75D: DMLabelMakeInvalid_Private (plexlabel.c:85)
    n1: 603979888 0x4E1115: DMLabelSetValue (plexlabel.c:425)
     n1: 603979888 0x4646BF: DMPlexStratify (plex.c:1762)
      n1: 603979888 0x9E4D5E: DMPlexInterpolateFaces_Internal (plexinterpolate.c:304)
       n1: 603979888 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
        n1: 603979888 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
         n1: 603979888 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
          n1: 603979888 0x4152C9: CreateMesh (SetupMeshes.c:23)
           n0: 603979888 0x4076DF: main (Nwtun.c:55)
   n1: 402653224 0x4E14DD: DMLabelSetValue (plexlabel.c:450)
    n0: 402653224 in 4 places, all below massif's threshold (01.00%)
 n0: 233193683 in 116 places, all below massif's threshold (01.00%)
#-----------
snapshot=49
#-----------
time=2761530816318
mem_heap_B=23983355785
mem_heap_extra_B=71719
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=50
#-----------
time=2767533197595
mem_heap_B=23568119581
mem_heap_extra_B=63651
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=51
#-----------
time=2794984499889
mem_heap_B=23568120619
mem_heap_extra_B=63821
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=52
#-----------
time=2797183222241
mem_heap_B=23568126541
mem_heap_extra_B=63795
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=53
#-----------
time=2799999920116
mem_heap_B=23590446779
mem_heap_extra_B=70773
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=54
#-----------
time=2802219005929
mem_heap_B=23612760439
mem_heap_extra_B=72121
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=55
#-----------
time=2806720304926
mem_heap_B=23657390679
mem_heap_extra_B=71897
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=56
#-----------
time=2832414634760
mem_heap_B=23746651047
mem_heap_extra_B=71561
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=57
#-----------
time=2850303228602
mem_heap_B=23925171223
mem_heap_extra_B=71449
mem_stacks_B=0
heap_tree=detailed
n4: 23925171223 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n8: 21763298824 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 4418111328 0x462AB1: DMSetUp_Plex (plex.c:1583)
   n2: 4418111328 0x50B88B: DMSetUp (dm.c:560)
    n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229)
     n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 3060791648 0x4076DF: main (Nwtun.c:55)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n2: 4418111328 0x46371F: DMPlexSymmetrize (plex.c:1655)
   n1: 3060791648 0x9E4D06: DMPlexInterpolateFaces_Internal (plexinterpolate.c:303)
    n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
     n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
      n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 3060791648 0x4076DF: main (Nwtun.c:55)
   n1: 1357319680 0x444320: DMPlexCreateExodusNwtun (plexexodusii.c:247)
    n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
     n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
      n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 4418111328 0x462A1D: DMSetUp_Plex (plex.c:1582)
   n2: 4418111328 0x50B88B: DMSetUp (dm.c:560)
    n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229)
     n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 3060791648 0x4076DF: main (Nwtun.c:55)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132)
   n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
    n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23)
     n0: 2544974400 0x4076DF: main (Nwtun.c:55)
  n2: 2385557104 0x52EBD7: PetscSectionSetChart (vsectionis.c:389)
   n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
   n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n2: 2385557104 0x52EC32: PetscSectionSetChart (vsectionis.c:389)
   n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
   n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n2: 1192778552 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46)
   n2: 851739512 0x4E2DE5: DMLabelGetStratumIS (plexlabel.c:571)
    n1: 851739512 0x46481D: DMPlexStratify (plex.c:1768)
     n1: 851739512 0x9E4D5E: DMPlexInterpolateFaces_Internal (plexinterpolate.c:304)
      n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
       n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
        n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
         n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
          n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
   n1: 341039040 0x4E1E75: DMLabelGetNumValues (plexlabel.c:504)
    n1: 341039040 0x473CE3: DMPlexGetDepth (plex.c:3971)
     n1: 341039040 0x9E510C: DMPlexInterpolate (plexinterpolate.c:335)
      n1: 341039040 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n0: 97680 in 76 places, all below massif's threshold (01.00%)
 n1: 1017989760 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202)
  n1: 1017989760 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
   n1: 1017989760 0x4152C9: CreateMesh (SetupMeshes.c:23)
    n0: 1017989760 0x4076DF: main (Nwtun.c:55)
 n0: 741229415 in 117 places, all below massif's threshold (01.00%)
 n1: 402653224 0x4DBEB6: kh_resize_HASHI (hash.h:510)
  n2: 402653224 0x4DC2CD: kh_put_HASHI (hash.h:510)
   n2: 402653224 0x4E14DD: DMLabelSetValue (plexlabel.c:450)
    n1: 402653224 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909)
     n1: 402653224 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173)
      n1: 402653224 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 402653224 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 402653224 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 3 places, all below massif's threshold (01.00%)
   n0: 0 in 1 place, below massif's threshold (01.00%)
#-----------
snapshot=58
#-----------
time=2891840800954
mem_heap_B=24282210907
mem_heap_extra_B=75909
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=59
#-----------
time=2894223977411
mem_heap_B=24983709031
mem_heap_extra_B=75105
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=60
#-----------
time=3056581214185
mem_heap_B=24996291947
mem_heap_extra_B=79141
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=61
#-----------
time=3060855258657
mem_heap_B=26405577307
mem_heap_extra_B=79925
mem_stacks_B=0
heap_tree=detailed
n6: 26405577307 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n8: 21763298824 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 4418111328 0x462AB1: DMSetUp_Plex (plex.c:1583)
   n2: 4418111328 0x50B88B: DMSetUp (dm.c:560)
    n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229)
     n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 3060791648 0x4076DF: main (Nwtun.c:55)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n2: 4418111328 0x46371F: DMPlexSymmetrize (plex.c:1655)
   n1: 3060791648 0x9E4D06: DMPlexInterpolateFaces_Internal (plexinterpolate.c:303)
    n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
     n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
      n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 3060791648 0x4076DF: main (Nwtun.c:55)
   n1: 1357319680 0x444320: DMPlexCreateExodusNwtun (plexexodusii.c:247)
    n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
     n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
      n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 4418111328 0x462A1D: DMSetUp_Plex (plex.c:1582)
   n2: 4418111328 0x50B88B: DMSetUp (dm.c:560)
    n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229)
     n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 3060791648 0x4076DF: main (Nwtun.c:55)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132)
   n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
    n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23)
     n0: 2544974400 0x4076DF: main (Nwtun.c:55)
  n2: 2385557104 0x52EBD7: PetscSectionSetChart (vsectionis.c:389)
   n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
   n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n2: 2385557104 0x52EC32: PetscSectionSetChart (vsectionis.c:389)
   n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
   n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n2: 1192778552 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46)
   n2: 851739512 0x4E2DE5: DMLabelGetStratumIS (plexlabel.c:571)
    n1: 851739512 0x46481D: DMPlexStratify (plex.c:1768)
     n1: 851739512 0x9E4D5E: DMPlexInterpolateFaces_Internal (plexinterpolate.c:304)
      n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
       n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
        n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
         n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
          n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
   n1: 341039040 0x4E1E75: DMLabelGetNumValues (plexlabel.c:504)
    n1: 341039040 0x473CE3: DMPlexGetDepth (plex.c:3971)
     n1: 341039040 0x9E510C: DMPlexInterpolate (plexinterpolate.c:335)
      n1: 341039040 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n0: 97680 in 76 places, all below massif's threshold (01.00%)
 n1: 1610612896 0x9E01D5: kh_resize_HASHIJKL (hash.h:1199)
  n1: 1610612896 0x9E07CD: kh_put_HASHIJKL (hash.h:1199)
   n2: 1610612896 0x9E1116: PetscHashIJKLPut (hash.h:1262)
    n1: 1610612896 0x9E3866: DMPlexInterpolateFaces_Internal (plexinterpolate.c:195)
     n1: 1610612896 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 1610612896 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 1610612896 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 1610612896 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 1610612896 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
 n1: 1207959672 0x9E020C: kh_resize_HASHIJKL (hash.h:1199)
  n1: 1207959672 0x9E07CD: kh_put_HASHIJKL (hash.h:1199)
   n2: 1207959672 0x9E1116: PetscHashIJKLPut (hash.h:1262)
    n1: 1207959672 0x9E3866: DMPlexInterpolateFaces_Internal (plexinterpolate.c:195)
     n1: 1207959672 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 1207959672 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 1207959672 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 1207959672 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 1207959672 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
 n1: 1017989760 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202)
  n1: 1017989760 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
   n1: 1017989760 0x4152C9: CreateMesh (SetupMeshes.c:23)
    n0: 1017989760 0x4076DF: main (Nwtun.c:55)
 n0: 403062931 in 115 places, all below massif's threshold (01.00%)
 n1: 402653224 0x4DBEB6: kh_resize_HASHI (hash.h:510)
  n2: 402653224 0x4DC2CD: kh_put_HASHI (hash.h:510)
   n2: 402653224 0x4E14DD: DMLabelSetValue (plexlabel.c:450)
    n1: 402653224 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909)
     n1: 402653224 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173)
      n1: 402653224 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 402653224 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 402653224 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 3 places, all below massif's threshold (01.00%)
   n0: 0 in 1 place, below massif's threshold (01.00%)
#-----------
snapshot=62
#-----------
time=3200681585287
mem_heap_B=26424451675
mem_heap_extra_B=79925
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=63
#-----------
time=3209016113153
mem_heap_B=29243024971
mem_heap_extra_B=78917
mem_stacks_B=0
heap_tree=detailed
n6: 29243024971 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n8: 21763298824 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 4418111328 0x462AB1: DMSetUp_Plex (plex.c:1583)
   n2: 4418111328 0x50B88B: DMSetUp (dm.c:560)
    n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229)
     n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 3060791648 0x4076DF: main (Nwtun.c:55)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n2: 4418111328 0x46371F: DMPlexSymmetrize (plex.c:1655)
   n1: 3060791648 0x9E4D06: DMPlexInterpolateFaces_Internal (plexinterpolate.c:303)
    n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
     n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
      n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 3060791648 0x4076DF: main (Nwtun.c:55)
   n1: 1357319680 0x444320: DMPlexCreateExodusNwtun (plexexodusii.c:247)
    n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
     n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
      n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 4418111328 0x462A1D: DMSetUp_Plex (plex.c:1582)
   n2: 4418111328 0x50B88B: DMSetUp (dm.c:560)
    n1: 3060791648 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229)
     n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 3060791648 0x4076DF: main (Nwtun.c:55)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132)
   n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
    n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23)
     n0: 2544974400 0x4076DF: main (Nwtun.c:55)
  n2: 2385557104 0x52EBD7: PetscSectionSetChart (vsectionis.c:389)
   n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
   n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n2: 2385557104 0x52EC32: PetscSectionSetChart (vsectionis.c:389)
   n2: 1192778552 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
   n2: 1192778552 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 851739512 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n1: 341039040 0x443417: DMPlexCreateExodusNwtun (plexexodusii.c:115)
     n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n2: 1192778552 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46)
   n2: 851739512 0x4E2DE5: DMLabelGetStratumIS (plexlabel.c:571)
    n1: 851739512 0x46481D: DMPlexStratify (plex.c:1768)
     n1: 851739512 0x9E4D5E: DMPlexInterpolateFaces_Internal (plexinterpolate.c:304)
      n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
       n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
        n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
         n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
          n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
   n1: 341039040 0x4E1E75: DMLabelGetNumValues (plexlabel.c:504)
    n1: 341039040 0x473CE3: DMPlexGetDepth (plex.c:3971)
     n1: 341039040 0x9E510C: DMPlexInterpolate (plexinterpolate.c:335)
      n1: 341039040 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 341039040 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 341039040 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 341039040 0x4076DF: main (Nwtun.c:55)
  n0: 97680 in 76 places, all below massif's threshold (01.00%)
 n1: 3221226208 0x9E01D5: kh_resize_HASHIJKL (hash.h:1199)
  n1: 3221226208 0x9E07CD: kh_put_HASHIJKL (hash.h:1199)
   n2: 3221226208 0x9E1116: PetscHashIJKLPut (hash.h:1262)
    n1: 3221226208 0x9E3866: DMPlexInterpolateFaces_Internal (plexinterpolate.c:195)
     n1: 3221226208 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 3221226208 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 3221226208 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 3221226208 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 3221226208 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
 n1: 2415919656 0x9E020C: kh_resize_HASHIJKL (hash.h:1199)
  n1: 2415919656 0x9E07CD: kh_put_HASHIJKL (hash.h:1199)
   n2: 2415919656 0x9E1116: PetscHashIJKLPut (hash.h:1262)
    n1: 2415919656 0x9E3866: DMPlexInterpolateFaces_Internal (plexinterpolate.c:195)
     n1: 2415919656 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 2415919656 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 2415919656 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 2415919656 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 2415919656 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
 n1: 1017989760 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202)
  n1: 1017989760 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
   n1: 1017989760 0x4152C9: CreateMesh (SetupMeshes.c:23)
    n0: 1017989760 0x4076DF: main (Nwtun.c:55)
 n0: 421937299 in 115 places, all below massif's threshold (01.00%)
 n1: 402653224 0x4DBEB6: kh_resize_HASHI (hash.h:510)
  n2: 402653224 0x4DC2CD: kh_put_HASHI (hash.h:510)
   n2: 402653224 0x4E14DD: DMLabelSetValue (plexlabel.c:450)
    n1: 402653224 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909)
     n1: 402653224 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173)
      n1: 402653224 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 402653224 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 402653224 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 3 places, all below massif's threshold (01.00%)
   n0: 0 in 1 place, below massif's threshold (01.00%)
#-----------
snapshot=64
#-----------
time=3385529782206
mem_heap_B=26009215847
mem_heap_extra_B=71577
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=65
#-----------
time=3686934054551
mem_heap_B=33110337511
mem_heap_extra_B=78369
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=66
#-----------
time=3815497683488
mem_heap_B=37195948379
mem_heap_extra_B=80093
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=67
#-----------
time=3815497691074
mem_heap_B=37195948999
mem_heap_extra_B=80137
mem_stacks_B=0
heap_tree=peak
n4: 37195948999 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n8: 35391116840 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 8503722144 0x462AB1: DMSetUp_Plex (plex.c:1583)
   n2: 8503722144 0x50B88B: DMSetUp (dm.c:560)
    n1: 7146402464 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229)
     n1: 7146402464 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 7146402464 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 7146402464 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 7146402464 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 7146402464 0x4076DF: main (Nwtun.c:55)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 8503722144 0x462A1D: DMSetUp_Plex (plex.c:1582)
   n2: 8503722144 0x50B88B: DMSetUp (dm.c:560)
    n1: 7146402464 0x9E3E70: DMPlexInterpolateFaces_Internal (plexinterpolate.c:229)
     n1: 7146402464 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 7146402464 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 7146402464 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 7146402464 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 7146402464 0x4076DF: main (Nwtun.c:55)
    n1: 1357319680 0x4437B2: DMPlexCreateExodusNwtun (plexexodusii.c:146)
     n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n2: 5113855296 0x52EBD7: PetscSectionSetChart (vsectionis.c:389)
   n2: 2556927648 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 2215888608 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 2215888608 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 2215888608 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 2215888608 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 2215888608 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 2215888608 0x4076DF: main (Nwtun.c:55)
    n0: 341039040 in 1 place, below massif's threshold (01.00%)
   n2: 2556927648 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 2215888608 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 2215888608 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 2215888608 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 2215888608 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 2215888608 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 2215888608 0x4076DF: main (Nwtun.c:55)
    n0: 341039040 in 1 place, below massif's threshold (01.00%)
  n2: 5113855296 0x52EC32: PetscSectionSetChart (vsectionis.c:389)
   n2: 2556927648 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 2215888608 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 2215888608 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 2215888608 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 2215888608 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 2215888608 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 2215888608 0x4076DF: main (Nwtun.c:55)
    n0: 341039040 in 1 place, below massif's threshold (01.00%)
   n2: 2556927648 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 2215888608 0x9E3ADA: DMPlexInterpolateFaces_Internal (plexinterpolate.c:206)
     n1: 2215888608 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
      n1: 2215888608 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
       n1: 2215888608 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 2215888608 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 2215888608 0x4076DF: main (Nwtun.c:55)
    n0: 341039040 in 1 place, below massif's threshold (01.00%)
  n2: 4418111328 0x46371F: DMPlexSymmetrize (plex.c:1655)
   n1: 3060791648 0x9E4D06: DMPlexInterpolateFaces_Internal (plexinterpolate.c:303)
    n1: 3060791648 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
     n1: 3060791648 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
      n1: 3060791648 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 3060791648 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 3060791648 0x4076DF: main (Nwtun.c:55)
   n1: 1357319680 0x444320: DMPlexCreateExodusNwtun (plexexodusii.c:247)
    n1: 1357319680 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
     n1: 1357319680 0x4152C9: CreateMesh (SetupMeshes.c:23)
      n0: 1357319680 0x4076DF: main (Nwtun.c:55)
  n1: 2544974400 0x4434CB: DMPlexCreateExodusNwtun (plexexodusii.c:132)
   n1: 2544974400 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
    n1: 2544974400 0x4152C9: CreateMesh (SetupMeshes.c:23)
     n0: 2544974400 0x4076DF: main (Nwtun.c:55)
  n2: 1192778552 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46)
   n2: 851739512 0x4E2DE5: DMLabelGetStratumIS (plexlabel.c:571)
    n1: 851739512 0x46481D: DMPlexStratify (plex.c:1768)
     n1: 851739512 0x9E4D5E: DMPlexInterpolateFaces_Internal (plexinterpolate.c:304)
      n1: 851739512 0x9E5383: DMPlexInterpolate (plexinterpolate.c:347)
       n1: 851739512 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
        n1: 851739512 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
         n1: 851739512 0x4152C9: CreateMesh (SetupMeshes.c:23)
          n0: 851739512 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
   n0: 341039040 in 1 place, below massif's threshold (01.00%)
  n0: 97680 in 76 places, all below massif's threshold (01.00%)
 n1: 1017989760 0x443F16: DMPlexCreateExodusNwtun (plexexodusii.c:202)
  n1: 1017989760 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
   n1: 1017989760 0x4152C9: CreateMesh (SetupMeshes.c:23)
    n0: 1017989760 0x4076DF: main (Nwtun.c:55)
 n1: 402653224 0x4DBEB6: kh_resize_HASHI (hash.h:510)
  n2: 402653224 0x4DC2CD: kh_put_HASHI (hash.h:510)
   n2: 402653224 0x4E14DD: DMLabelSetValue (plexlabel.c:450)
    n1: 402653224 0x4E6D9B: DMPlexSetLabelValue (plexlabel.c:909)
     n1: 402653224 0x443C39: DMPlexCreateExodusNwtun (plexexodusii.c:173)
      n1: 402653224 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
       n1: 402653224 0x4152C9: CreateMesh (SetupMeshes.c:23)
        n0: 402653224 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 3 places, all below massif's threshold (01.00%)
   n0: 0 in 1 place, below massif's threshold (01.00%)
 n0: 384189175 in 117 places, all below massif's threshold (01.00%)
#-----------
snapshot=68
#-----------
time=3818343537621
mem_heap_B=37207106171
mem_heap_extra_B=87909
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=69
#-----------
time=3821057220870
mem_heap_B=37218264795
mem_heap_extra_B=86789
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=70
#-----------
time=3826940637881
mem_heap_B=37240578455
mem_heap_extra_B=88137
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=71
#-----------
time=3840182989443
mem_heap_B=37285208695
mem_heap_extra_B=87913
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=72
#-----------
time=3883346460203
mem_heap_B=37374469063
mem_heap_extra_B=87577
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=73
#-----------
time=3936192268561
mem_heap_B=37552989239
mem_heap_extra_B=87465
mem_stacks_B=0
heap_tree=empty


More information about the petsc-users mailing list