[petsc-users] SNESComputeJacobianDefaultColor use too much memory

Rongliang Chen rongliang.chan at gmail.com
Wed Feb 17 01:49:12 CST 2016


Dear Barry,

The massif output for the large case is attached. It shows that the job 
was kill in the function  MatGetSubMatrix_MPIAIJ_All(). Any suggestions?

--------------------
[8]PETSC ERROR: #1 MatGetSubMatrix_MPIAIJ_All() line 622 in 
/home/rlchen/soft/petsc-3.5.2/src/mat/impls/aij/mpi/mpiov.c
[8]PETSC ERROR: #2 MatGetSeqNonzeroStructure_MPIAIJ() line 3010 in 
/home/rlchen/soft/petsc-3.5.2/src/mat/impls/aij/mpi/mpiaij.c
[8]PETSC ERROR: #3 MatGetSeqNonzeroStructure() line 6487 in 
/home/rlchen/soft/petsc-3.5.2/src/mat/interface/matrix.c
[8]PETSC ERROR: #4 MatColoringApply_SL() line 78 in 
/home/rlchen/soft/petsc-3.5.2/src/mat/color/impls/minpack/color.c
[8]PETSC ERROR: #5 MatColoringApply() line 379 in 
/home/rlchen/soft/petsc-3.5.2/src/mat/color/interface/matcoloring.c
[8]PETSC ERROR: #6 SNESComputeJacobianDefaultColor() line 71 in 
/home/rlchen/soft/petsc-3.5.2/src/snes/interface/snesj2.c
[8]PETSC ERROR: #7 FormJacobian() line 58 in 
/home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/SetupJacobian.c
[8]PETSC ERROR: #8 SNESComputeJacobian() line 2193 in 
/home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c
[8]PETSC ERROR: #9 SNESSolve_NEWTONLS() line 230 in 
/home/rlchen/soft/petsc-3.5.2/src/snes/impls/ls/ls.c
[8]PETSC ERROR: #10 SNESSolve() line 3743 in 
/home/rlchen/soft/petsc-3.5.2/src/snes/interface/snes.c
[8]PETSC ERROR: #11 SolveTimeDependent() line 758 in 
/home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/Nwtun.c
[8]PETSC ERROR: #12 main() line 417 in 
/home/rlchen/soft/3D_fluid/FiniteVolumeMethod/PETScCodes/codefor3.5/Nwtun.c
-------------------

Best,
Rongliang

On 02/17/2016 02:09 PM, Barry Smith wrote:
>    Yes, this is the type of output I was expecting. Now you need to produce it for a large case.
>
>    Barry
>
>> On Feb 16, 2016, at 11:49 PM, Rongliang Chen <rongliang.chan at gmail.com> wrote:
>>
>> Hi Barry,
>>
>> I run the code with valgrind on a workstation for a smaller case and it produces some ASCII information (see attached).  Is this helpful?
>>
>> Best,
>> Rongliang
>>
>> On 02/17/2016 01:30 PM, Barry Smith wrote:
>>>    Hmm, something didn't work right with massif. I should give a bunch of ASCII information about how much memory is used at different times in the code. You may need to play around with the massif options and google documentation on how to get it to provide useful information. Once it produces the useful information it will be very helpful.
>>>
>>> time=0
>>> mem_heap_B=0
>>> mem_heap_extra_B=0
>>> mem_stacks_B=0
>>> heap_tree=empty
>>>
>>>
>>>> On Feb 16, 2016, at 9:44 PM, Rongliang Chen <rongliang.chan at gmail.com> wrote:
>>>>
>>>> Dear Barry,
>>>>
>>>> Many thanks for your reply.
>>>>
>>>> I checked with the valgrind and did not obtain any outputs (massif.out.<pid>) because the job was killed before it reached the end.
>>>>
>>>> Then I switch to a smaller case, it works well and one of the output is attached (I did not find any useful information in it). The output with the option -mat_coloring_view is followed, which shows that the number of colors is 65. Any ideas for this?
>>>>
>>>> MatColoring Object: 480 MPI processes
>>>>    type: sl
>>>>    Weight type: RANDOM
>>>>    Distance 2, Max. Colors 65535
>>>>    Number of colors 65
>>>>    Number of total columns 1637350
>>>>
>>>> Best regards,
>>>> Rongliang
>>>>
>>>> On 02/17/2016 01:13 AM, Barry Smith wrote:
>>>>>    How many colors are needed?
>>>>>
>>>>>    You need to produce a breakdown of where all the memory is being used. For example valgrind with the
>>>>> http://valgrind.org/docs/manual/ms-manual.html
>>>>>
>>>>>
>>>>>    Barry
>>>>>
>>>>>
>>>>>> On Feb 16, 2016, at 6:45 AM, Rongliang Chen <rongliang.chan at gmail.com>
>>>>>>   wrote:
>>>>>>
>>>>>> Dear all,
>>>>>>
>>>>>> I am using the DMPlex to solve a PDE on a unstructured mesh and I use the SNESComputeJacobianDefaultColor to compute the Jacobian matrix.
>>>>>>
>>>>>> My code works well for small problems (such as problem with 3.3x10^5 cells using 120 cores) but when I increase the number of cells (2.6x10^6 cells using 1920 cores), I go out of memory in the function MatColoringApply. It shows that one of the cores uses over 11G memory. I think this is unreasonable. Do you have any suggestions for debugging this problem?
>>>>>>
>>>>>> Best regards,
>>>>>> Rongliang
>>>>>>
>>>> <massif.out.12562>
>> <massif.out.26358>

-------------- next part --------------
desc: (none)
cmd: ./Nwtun -f ./../meshes/3DNACA0012AIRFOIL_Hex150x50x100x80.exo -overlap 1 -options_file options.txt -time_integral_method 1 -initial_time 0.0 -final_time 4.0 -dt 1.e-3 -timestep 2 -max_time_its 0 -includeenergy -reconstruct -PressureFlux -testcase 3DNACA -output_solution -solutionfile results/3DNACA0012AIRFOIL_Hex150x50x100x80_imp -steps_output 4000 -inflow_u 349.02 -viscosity 0.01 -k 0.026 -monitor_step_gap -R 287.02 -adiabatic 1.4 -Euler -Attack 10 -Mach 0.8 -fd_jacobian_color -limiter zero -RiemannSolver Rusanov -CFL 0.9 -reconstruct -second_order -parallel_csr -parallel_output
time_unit: i
#-----------
snapshot=0
#-----------
time=0
mem_heap_B=0
mem_heap_extra_B=0
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=1
#-----------
time=224006276
mem_heap_B=134033
mem_heap_extra_B=8831
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=2
#-----------
time=85544444037
mem_heap_B=139207
mem_heap_extra_B=8913
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=3
#-----------
time=85546748915
mem_heap_B=147017
mem_heap_extra_B=9191
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=4
#-----------
time=288378803176
mem_heap_B=148049
mem_heap_extra_B=9239
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=5
#-----------
time=288380578373
mem_heap_B=137997
mem_heap_extra_B=8811
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=6
#-----------
time=667538944418
mem_heap_B=139029
mem_heap_extra_B=8859
mem_stacks_B=0
heap_tree=detailed
n9: 139029 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n12: 81895 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 19200 0x5DF2BC: EventPerfLogEnsureSize (eventlog.c:211)
   n1: 19200 0x5C78F5: PetscLogEventRegister (plog.c:755)
    n1: 19200 0x7BA2C9: MatInitializePackage (dlregismat.c:174)
     n1: 19200 0x505D39: DMCreate (dm.c:40)
      n1: 19200 0x443017: DMPlexCreateExodusNwtun (plexexodusii.c:102)
       n1: 19200 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 19200 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 19200 0x4076DF: main (Nwtun.c:55)
  n0: 16880 0x4072F7: main (Nwtun.c:43)
  n1: 10032 0x56BC70: PetscSegBufferCreate (segbuffer.c:68)
   n1: 10032 0x5B076C: PetscCitationsInitialize (pinit.c:450)
    n1: 10032 0x5B228E: PetscInitialize (pinit.c:883)
     n0: 10032 0x406F74: main (Nwtun.c:24)
  n0: 9784 in 44 places, all below massif's threshold (01.00%)
  n3: 8272 0x505DF4: DMCreate (dm.c:43)
   n1: 4136 0x9E5266: DMPlexInterpolate (plexinterpolate.c:344)
    n1: 4136 0x44440A: DMPlexCreateExodusNwtun (plexexodusii.c:252)
     n1: 4136 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
      n1: 4136 0x4152C9: CreateMesh (SetupMeshes.c:23)
       n0: 4136 0x4076DF: main (Nwtun.c:55)
   n1: 4136 0x5066C4: DMClone (dm.c:99)
    n1: 4136 0x47819A: DMCreateCoordinateDM_Plex (plex.c:4355)
     n1: 4136 0x52788B: DMGetCoordinateDM (dm.c:3557)
      n1: 4136 0x528290: DMGetCoordinateSection (dm.c:3618)
       n1: 4136 0x444E4E: DMPlexCreateExodusNwtun (plexexodusii.c:301)
        n1: 4136 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
         n1: 4136 0x4152C9: CreateMesh (SetupMeshes.c:23)
          n0: 4136 0x4076DF: main (Nwtun.c:55)
   n0: 0 in 1 place, below massif's threshold (01.00%)
  n2: 3271 0x561F27: PetscStrallocpy (str.c:188)
   n1: 2318 0x5DFA3D: EventRegLogRegister (eventlog.c:317)
    n1: 2318 0x5C7873: PetscLogEventRegister (plog.c:753)
     n0: 2318 in 166 places, all below massif's threshold (01.00%)
   n0: 953 in 15 places, all below massif's threshold (01.00%)
  n1: 3200 0x5DF8BC: EventRegLogRegister (eventlog.c:310)
   n1: 3200 0x5C7873: PetscLogEventRegister (plog.c:753)
    n1: 3200 0x7BA2C9: MatInitializePackage (dlregismat.c:174)
     n1: 3200 0x505D39: DMCreate (dm.c:40)
      n1: 3200 0x443017: DMPlexCreateExodusNwtun (plexexodusii.c:102)
       n1: 3200 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
        n1: 3200 0x4152C9: CreateMesh (SetupMeshes.c:23)
         n0: 3200 0x4076DF: main (Nwtun.c:55)
  n1: 3200 0x5D5CD5: ClassPerfLogCreate (classlog.c:121)
   n1: 3200 0x5DA811: PetscStageLogRegister (stagelog.c:230)
    n1: 3200 0x5C4C70: PetscLogBegin_Private (plog.c:214)
     n1: 3200 0x5B22E5: PetscInitialize (pinit.c:891)
      n0: 3200 0x406F74: main (Nwtun.c:24)
  n1: 2560 0xC169B0: PetscThreadCommWorldInitialize (threadcomm.c:1240)
   n1: 2560 0xC0DA8A: PetscGetThreadCommWorld (threadcomm.c:82)
    n1: 2560 0xC0DD9D: PetscCommGetThreadComm (threadcomm.c:117)
     n1: 2560 0x58238E: PetscCommDuplicate (tagm.c:195)
      n1: 2560 0x586EE2: PetscHeaderCreate_Private (inherit.c:59)
       n1: 2560 0x505EEE: DMCreate (dm.c:43)
        n1: 2560 0x443017: DMPlexCreateExodusNwtun (plexexodusii.c:102)
         n1: 2560 0x442B9B: DMPlexCreateExodusFromFileNwtun (plexexodusii.c:53)
          n1: 2560 0x4152C9: CreateMesh (SetupMeshes.c:23)
           n0: 2560 0x4076DF: main (Nwtun.c:55)
  n1: 2136 0x67D9C3: PetscSFCreate (sf.c:44)
   n0: 2136 in 2 places, all below massif's threshold (01.00%)
  n1: 1760 0x56EE38: PetscFunctionListAdd_Private (reg.c:210)
   n0: 1760 in 10 places, all below massif's threshold (01.00%)
  n1: 1600 0x5D52AA: PetscClassRegLogCreate (classlog.c:36)
   n1: 1600 0x5DD4FB: PetscStageLogCreate (stagelog.c:638)
    n1: 1600 0x5C4C0F: PetscLogBegin_Private (plog.c:213)
     n1: 1600 0x5B22E5: PetscInitialize (pinit.c:891)
      n0: 1600 0x406F74: main (Nwtun.c:24)
 n1: 17152 0x5A3ED5: PetscOptionsCreate (options.c:2322)
  n1: 17152 0x5B1876: PetscInitialize (pinit.c:763)
   n0: 17152 0x406F74: main (Nwtun.c:24)
 n0: 14910 in 68 places, all below massif's threshold (01.00%)
 n1: 13152 0x5EB3FDA: MPIDI_PG_Create (in /home/rlchen/mpich2/lib/libmpich.so.3.0)
  n1: 13152 0x5EA24B9: MPID_Init (in /home/rlchen/mpich2/lib/libmpich.so.3.0)
   n1: 13152 0x5E98088: MPIR_Init_thread (in /home/rlchen/mpich2/lib/libmpich.so.3.0)
    n1: 13152 0x5E981CF: PMPI_Init_thread (in /home/rlchen/mpich2/lib/libmpich.so.3.0)
     n1: 13152 0x5B1A7E: PetscInitialize (pinit.c:781)
      n0: 13152 0x406F74: main (Nwtun.c:24)
 n1: 4096 0x660F8EA: register_printf_specifier (reg-printf.c:58)
  n1: 4096 0x6BA4770: ??? (in /usr/lib/x86_64-linux-gnu/libquadmath.so.0.0.0)
   n1: 4096 0x4010138: call_init.part.0 (dl-init.c:78)
    n1: 4096 0x4010221: _dl_init (dl-init.c:36)
     n1: 4096 0x4001308: ??? (in /lib/x86_64-linux-gnu/ld-2.19.so)
      n1: 4096 0x36: ???
       n1: 4096 0xFFF0002F5: ???
        n1: 4096 0xFFF0002FD: ???
         n1: 4096 0xFFF000300: ???
          n1: 4096 0xFFF000333: ???
           n1: 4096 0xFFF00033C: ???
            n1: 4096 0xFFF00033E: ???
             n1: 4096 0xFFF00034C: ???
              n1: 4096 0xFFF000358: ???
               n1: 4096 0xFFF00036E: ???
                n1: 4096 0xFFF000370: ???
                 n1: 4096 0xFFF00037E: ???
                  n1: 4096 0xFFF000382: ???
                   n1: 4096 0xFFF00038E: ???
                    n1: 4096 0xFFF000392: ???
                     n1: 4096 0xFFF000396: ???
                      n1: 4096 0xFFF00039C: ???
                       n1: 4096 0xFFF0003A6: ???
                        n1: 4096 0xFFF0003A8: ???
                         n1: 4096 0xFFF0003B6: ???
                          n1: 4096 0xFFF0003B8: ???
                           n1: 4096 0xFFF0003C7: ???
                            n1: 4096 0xFFF0003D4: ???
                             n1: 4096 0xFFF0003E2: ???
                              n0: 4096 0xFFF0003EC: ???
 n2: 2256 0x58BBC21: ??? (in /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0)
  n1: 2016 0x597F755: ??? (in /usr/lib/x86_64-linux-gnu/libgfortran.so.3.0.0)
   n0: 2016 in 3 places, all below massif's threshold (01.00%)
  n0: 240 in 1 place, below massif's threshold (01.00%)
 n1: 2040 0x6611753: register_printf_modifier (reg-modifier.c:69)
  n1: 2040 0x6BA4749: ??? (in /usr/lib/x86_64-linux-gnu/libquadmath.so.0.0.0)
   n1: 2040 0x4010138: call_init.part.0 (dl-init.c:78)
    n1: 2040 0x4010221: _dl_init (dl-init.c:36)
     n1: 2040 0x4001308: ??? (in /lib/x86_64-linux-gnu/ld-2.19.so)
      n1: 2040 0x36: ???
       n1: 2040 0xFFF0002F5: ???
        n1: 2040 0xFFF0002FD: ???
         n1: 2040 0xFFF000300: ???
          n1: 2040 0xFFF000333: ???
           n1: 2040 0xFFF00033C: ???
            n1: 2040 0xFFF00033E: ???
             n1: 2040 0xFFF00034C: ???
              n1: 2040 0xFFF000358: ???
               n1: 2040 0xFFF00036E: ???
                n1: 2040 0xFFF000370: ???
                 n1: 2040 0xFFF00037E: ???
                  n1: 2040 0xFFF000382: ???
                   n1: 2040 0xFFF00038E: ???
                    n1: 2040 0xFFF000392: ???
                     n1: 2040 0xFFF000396: ???
                      n1: 2040 0xFFF00039C: ???
                       n1: 2040 0xFFF0003A6: ???
                        n1: 2040 0xFFF0003A8: ???
                         n1: 2040 0xFFF0003B6: ???
                          n1: 2040 0xFFF0003B8: ???
                           n1: 2040 0xFFF0003C7: ???
                            n1: 2040 0xFFF0003D4: ???
                             n1: 2040 0xFFF0003E2: ???
                              n0: 2040 0xFFF0003EC: ???
 n1: 1984 0x6611A0D: register_printf_type (reg-type.c:43)
  n1: 1984 0x6BA472E: ??? (in /usr/lib/x86_64-linux-gnu/libquadmath.so.0.0.0)
   n1: 1984 0x4010138: call_init.part.0 (dl-init.c:78)
    n1: 1984 0x4010221: _dl_init (dl-init.c:36)
     n1: 1984 0x4001308: ??? (in /lib/x86_64-linux-gnu/ld-2.19.so)
      n1: 1984 0x36: ???
       n1: 1984 0xFFF0002F5: ???
        n1: 1984 0xFFF0002FD: ???
         n1: 1984 0xFFF000300: ???
          n1: 1984 0xFFF000333: ???
           n1: 1984 0xFFF00033C: ???
            n1: 1984 0xFFF00033E: ???
             n1: 1984 0xFFF00034C: ???
              n1: 1984 0xFFF000358: ???
               n1: 1984 0xFFF00036E: ???
                n1: 1984 0xFFF000370: ???
                 n1: 1984 0xFFF00037E: ???
                  n1: 1984 0xFFF000382: ???
                   n1: 1984 0xFFF00038E: ???
                    n1: 1984 0xFFF000392: ???
                     n1: 1984 0xFFF000396: ???
                      n1: 1984 0xFFF00039C: ???
                       n1: 1984 0xFFF0003A6: ???
                        n1: 1984 0xFFF0003A8: ???
                         n1: 1984 0xFFF0003B6: ???
                          n1: 1984 0xFFF0003B8: ???
                           n1: 1984 0xFFF0003C7: ???
                            n1: 1984 0xFFF0003D4: ???
                             n1: 1984 0xFFF0003E2: ???
                              n0: 1984 0xFFF0003EC: ???
 n1: 1544 0x63F2B3: PetscStackCreate (pstack.c:105)
  n1: 1544 0x5B2739: PetscInitialize (pinit.c:946)
   n0: 1544 0x406F74: main (Nwtun.c:24)
#-----------
snapshot=7
#-----------
time=669178970344
mem_heap_B=143791
mem_heap_extra_B=9305
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=8
#-----------
time=686722556504
mem_heap_B=143807
mem_heap_extra_B=9305
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=9
#-----------
time=694044915791
mem_heap_B=146405
mem_heap_extra_B=9475
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=10
#-----------
time=733393978137
mem_heap_B=147309
mem_heap_extra_B=9515
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=11
#-----------
time=733467252555
mem_heap_B=150111
mem_heap_extra_B=9729
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=12
#-----------
time=1139843034827
mem_heap_B=150191
mem_heap_extra_B=9737
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=13
#-----------
time=1141977712727
mem_heap_B=30255663
mem_heap_extra_B=10697
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=14
#-----------
time=1180154243665
mem_heap_B=30255711
mem_heap_extra_B=10705
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=15
#-----------
time=1188784700125
mem_heap_B=105582168
mem_heap_extra_B=11936
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=16
#-----------
time=1200655909356
mem_heap_B=359461085
mem_heap_extra_B=14739
mem_stacks_B=0
heap_tree=detailed
n2: 359461085 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n13: 359321645 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 89512864 0x54037E: PetscSFCreateSectionSF (vsectionis.c:2098)
   n1: 89512864 0x557BF3: DMPlexDistributeNwtun (plexdistribute.c:128)
    n1: 89512864 0x4154E6: CreateMesh (SetupMeshes.c:34)
     n0: 89512864 0x4076DF: main (Nwtun.c:55)
  n1: 44756432 0x462AB1: DMSetUp_Plex (plex.c:1583)
   n1: 44756432 0x50B88B: DMSetUp (dm.c:560)
    n1: 44756432 0x557A48: DMPlexDistributeNwtun (plexdistribute.c:116)
     n1: 44756432 0x4154E6: CreateMesh (SetupMeshes.c:34)
      n0: 44756432 0x4076DF: main (Nwtun.c:55)
  n1: 44756432 0x5402EC: PetscSFCreateSectionSF (vsectionis.c:2097)
   n1: 44756432 0x557BF3: DMPlexDistributeNwtun (plexdistribute.c:128)
    n1: 44756432 0x4154E6: CreateMesh (SetupMeshes.c:34)
     n0: 44756432 0x4076DF: main (Nwtun.c:55)
  n1: 44756432 0x462A1D: DMSetUp_Plex (plex.c:1582)
   n1: 44756432 0x50B88B: DMSetUp (dm.c:560)
    n1: 44756432 0x557A48: DMPlexDistributeNwtun (plexdistribute.c:116)
     n1: 44756432 0x4154E6: CreateMesh (SetupMeshes.c:34)
      n0: 44756432 0x4076DF: main (Nwtun.c:55)
  n1: 30096944 0x680E6B: PetscSFSetGraph (sf.c:328)
   n1: 30096944 0x6822D2: PetscSFCreateInverseSF (sf.c:478)
    n1: 30096944 0x53E180: PetscSFConvertPartition (vsectionis.c:1937)
     n1: 30096944 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101)
      n1: 30096944 0x4154E6: CreateMesh (SetupMeshes.c:34)
       n0: 30096944 0x4076DF: main (Nwtun.c:55)
  n2: 15048664 0x52EC32: PetscSectionSetChart (vsectionis.c:389)
   n1: 15048472 0x53EA67: PetscSFDistributeSection (vsectionis.c:1998)
    n1: 15048472 0x5579E4: DMPlexDistributeNwtun (plexdistribute.c:115)
     n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34)
      n0: 15048472 0x4076DF: main (Nwtun.c:55)
   n0: 192 in 3 places, all below massif's threshold (01.00%)
  n2: 15048664 0x52EBD7: PetscSectionSetChart (vsectionis.c:389)
   n1: 15048472 0x53EA67: PetscSFDistributeSection (vsectionis.c:1998)
    n1: 15048472 0x5579E4: DMPlexDistributeNwtun (plexdistribute.c:115)
     n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34)
      n0: 15048472 0x4076DF: main (Nwtun.c:55)
   n0: 192 in 3 places, all below massif's threshold (01.00%)
  n2: 15048472 0x681299: PetscSFSetGraph (sf.c:349)
   n1: 15048472 0x6822D2: PetscSFCreateInverseSF (sf.c:478)
    n1: 15048472 0x53E180: PetscSFConvertPartition (vsectionis.c:1937)
     n1: 15048472 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101)
      n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34)
       n0: 15048472 0x4076DF: main (Nwtun.c:55)
   n0: 0 in 1 place, below massif's threshold (01.00%)
  n2: 15048472 0x6812ED: PetscSFSetGraph (sf.c:349)
   n1: 15048472 0x6822D2: PetscSFCreateInverseSF (sf.c:478)
    n1: 15048472 0x53E180: PetscSFConvertPartition (vsectionis.c:1937)
     n1: 15048472 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101)
      n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34)
       n0: 15048472 0x4076DF: main (Nwtun.c:55)
   n0: 0 in 1 place, below massif's threshold (01.00%)
  n1: 15048472 0x66B28C: ISLocalToGlobalMappingCreateSF (isltog.c:193)
   n1: 15048472 0x53E29B: PetscSFConvertPartition (vsectionis.c:1942)
    n1: 15048472 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101)
     n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34)
      n0: 15048472 0x4076DF: main (Nwtun.c:55)
  n1: 15048472 0xC88EE6: PetscSFBasicGetPack (sfbasic.c:779)
   n1: 15048472 0xC8AA7F: PetscSFBcastBegin_Basic (sfbasic.c:914)
    n2: 15048472 0x687407: PetscSFBcastBegin (sf.c:926)
     n1: 15048472 0x66B362: ISLocalToGlobalMappingCreateSF (isltog.c:196)
      n1: 15048472 0x53E29B: PetscSFConvertPartition (vsectionis.c:1942)
       n1: 15048472 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101)
        n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34)
         n0: 15048472 0x4076DF: main (Nwtun.c:55)
     n0: 0 in 1 place, below massif's threshold (01.00%)
  n1: 15048472 0x53EDF4: PetscSFDistributeSection (vsectionis.c:2007)
   n1: 15048472 0x5579E4: DMPlexDistributeNwtun (plexdistribute.c:115)
    n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34)
     n0: 15048472 0x4076DF: main (Nwtun.c:55)
  n0: 102853 in 108 places, all below massif's threshold (01.00%)
 n0: 139440 in 88 places, all below massif's threshold (01.00%)
#-----------
snapshot=17
#-----------
time=1218361277415
mem_heap_B=642593563
mem_heap_extra_B=19765
mem_stacks_B=0
heap_tree=detailed
n2: 642593563 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n13: 642453035 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 163910184 0x66DAC9: ISGlobalToLocalMappingSetUp_Private (isltog.c:489)
   n1: 163910184 0x66E7B8: ISGlobalToLocalMappingApplyBlock (isltog.c:639)
    n1: 163910184 0x557EA3: DMPlexDistributeNwtun (plexdistribute.c:134)
     n1: 163910184 0x4154E6: CreateMesh (SetupMeshes.c:34)
      n0: 163910184 0x4076DF: main (Nwtun.c:55)
  n1: 89512864 0x54037E: PetscSFCreateSectionSF (vsectionis.c:2098)
   n1: 89512864 0x557BF3: DMPlexDistributeNwtun (plexdistribute.c:128)
    n1: 89512864 0x4154E6: CreateMesh (SetupMeshes.c:34)
     n0: 89512864 0x4076DF: main (Nwtun.c:55)
  n1: 59804904 0xC88EE6: PetscSFBasicGetPack (sfbasic.c:779)
   n1: 59804904 0xC8AA7F: PetscSFBcastBegin_Basic (sfbasic.c:914)
    n3: 59804904 0x687407: PetscSFBcastBegin (sf.c:926)
     n1: 44756432 0x557D40: DMPlexDistributeNwtun (plexdistribute.c:131)
      n1: 44756432 0x4154E6: CreateMesh (SetupMeshes.c:34)
       n0: 44756432 0x4076DF: main (Nwtun.c:55)
     n1: 15048472 0x66B362: ISLocalToGlobalMappingCreateSF (isltog.c:196)
      n1: 15048472 0x53E29B: PetscSFConvertPartition (vsectionis.c:1942)
       n1: 15048472 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101)
        n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34)
         n0: 15048472 0x4076DF: main (Nwtun.c:55)
     n0: 0 in 1 place, below massif's threshold (01.00%)
  n3: 59804904 0x681299: PetscSFSetGraph (sf.c:349)
   n1: 44756432 0x5406F5: PetscSFCreateSectionSF (vsectionis.c:2119)
    n1: 44756432 0x557BF3: DMPlexDistributeNwtun (plexdistribute.c:128)
     n1: 44756432 0x4154E6: CreateMesh (SetupMeshes.c:34)
      n0: 44756432 0x4076DF: main (Nwtun.c:55)
   n1: 15048472 0x6822D2: PetscSFCreateInverseSF (sf.c:478)
    n1: 15048472 0x53E180: PetscSFConvertPartition (vsectionis.c:1937)
     n1: 15048472 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101)
      n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34)
       n0: 15048472 0x4076DF: main (Nwtun.c:55)
   n0: 0 in 1 place, below massif's threshold (01.00%)
  n3: 59804904 0x6812ED: PetscSFSetGraph (sf.c:349)
   n1: 44756432 0x5406F5: PetscSFCreateSectionSF (vsectionis.c:2119)
    n1: 44756432 0x557BF3: DMPlexDistributeNwtun (plexdistribute.c:128)
     n1: 44756432 0x4154E6: CreateMesh (SetupMeshes.c:34)
      n0: 44756432 0x4076DF: main (Nwtun.c:55)
   n1: 15048472 0x6822D2: PetscSFCreateInverseSF (sf.c:478)
    n1: 15048472 0x53E180: PetscSFConvertPartition (vsectionis.c:1937)
     n1: 15048472 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101)
      n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34)
       n0: 15048472 0x4076DF: main (Nwtun.c:55)
   n0: 0 in 1 place, below massif's threshold (01.00%)
  n1: 44756432 0x5402EC: PetscSFCreateSectionSF (vsectionis.c:2097)
   n1: 44756432 0x557BF3: DMPlexDistributeNwtun (plexdistribute.c:128)
    n1: 44756432 0x4154E6: CreateMesh (SetupMeshes.c:34)
     n0: 44756432 0x4076DF: main (Nwtun.c:55)
  n1: 44756432 0x462AB1: DMSetUp_Plex (plex.c:1583)
   n1: 44756432 0x50B88B: DMSetUp (dm.c:560)
    n1: 44756432 0x557A48: DMPlexDistributeNwtun (plexdistribute.c:116)
     n1: 44756432 0x4154E6: CreateMesh (SetupMeshes.c:34)
      n0: 44756432 0x4076DF: main (Nwtun.c:55)
  n1: 44756432 0x462A1D: DMSetUp_Plex (plex.c:1582)
   n1: 44756432 0x50B88B: DMSetUp (dm.c:560)
    n1: 44756432 0x557A48: DMPlexDistributeNwtun (plexdistribute.c:116)
     n1: 44756432 0x4154E6: CreateMesh (SetupMeshes.c:34)
      n0: 44756432 0x4076DF: main (Nwtun.c:55)
  n1: 30096944 0x680E6B: PetscSFSetGraph (sf.c:328)
   n1: 30096944 0x6822D2: PetscSFCreateInverseSF (sf.c:478)
    n1: 30096944 0x53E180: PetscSFConvertPartition (vsectionis.c:1937)
     n1: 30096944 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101)
      n1: 30096944 0x4154E6: CreateMesh (SetupMeshes.c:34)
       n0: 30096944 0x4076DF: main (Nwtun.c:55)
  n2: 15048664 0x52EBD7: PetscSectionSetChart (vsectionis.c:389)
   n1: 15048472 0x53EA67: PetscSFDistributeSection (vsectionis.c:1998)
    n1: 15048472 0x5579E4: DMPlexDistributeNwtun (plexdistribute.c:115)
     n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34)
      n0: 15048472 0x4076DF: main (Nwtun.c:55)
   n0: 192 in 3 places, all below massif's threshold (01.00%)
  n2: 15048664 0x52EC32: PetscSectionSetChart (vsectionis.c:389)
   n1: 15048472 0x53EA67: PetscSFDistributeSection (vsectionis.c:1998)
    n1: 15048472 0x5579E4: DMPlexDistributeNwtun (plexdistribute.c:115)
     n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34)
      n0: 15048472 0x4076DF: main (Nwtun.c:55)
   n0: 192 in 3 places, all below massif's threshold (01.00%)
  n1: 15048472 0x66B28C: ISLocalToGlobalMappingCreateSF (isltog.c:193)
   n1: 15048472 0x53E29B: PetscSFConvertPartition (vsectionis.c:1942)
    n1: 15048472 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101)
     n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34)
      n0: 15048472 0x4076DF: main (Nwtun.c:55)
  n0: 103235 in 109 places, all below massif's threshold (01.00%)
 n0: 140528 in 88 places, all below massif's threshold (01.00%)
#-----------
snapshot=18
#-----------
time=1236225419622
mem_heap_B=463955737
mem_heap_extra_B=17695
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=19
#-----------
time=1252147611144
mem_heap_B=463959867
mem_heap_extra_B=17589
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=20
#-----------
time=1263783858129
mem_heap_B=330597269
mem_heap_extra_B=16011
mem_stacks_B=0
heap_tree=detailed
n2: 330597269 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n13: 330457997 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 44756432 0x462AB1: DMSetUp_Plex (plex.c:1583)
   n1: 44756432 0x50B88B: DMSetUp (dm.c:560)
    n1: 44756432 0x557A48: DMPlexDistributeNwtun (plexdistribute.c:116)
     n1: 44756432 0x4154E6: CreateMesh (SetupMeshes.c:34)
      n0: 44756432 0x4076DF: main (Nwtun.c:55)
  n1: 44756432 0x462A1D: DMSetUp_Plex (plex.c:1582)
   n1: 44756432 0x50B88B: DMSetUp (dm.c:560)
    n1: 44756432 0x557A48: DMPlexDistributeNwtun (plexdistribute.c:116)
     n1: 44756432 0x4154E6: CreateMesh (SetupMeshes.c:34)
      n0: 44756432 0x4076DF: main (Nwtun.c:55)
  n1: 44756432 0x46371F: DMPlexSymmetrize (plex.c:1655)
   n1: 44756432 0x5585AB: DMPlexDistributeNwtun (plexdistribute.c:156)
    n1: 44756432 0x4154E6: CreateMesh (SetupMeshes.c:34)
     n0: 44756432 0x4076DF: main (Nwtun.c:55)
  n3: 33957328 0x52EBD7: PetscSectionSetChart (vsectionis.c:389)
   n2: 16978568 0x53EA67: PetscSFDistributeSection (vsectionis.c:1998)
    n1: 15048472 0x5579E4: DMPlexDistributeNwtun (plexdistribute.c:115)
     n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34)
      n0: 15048472 0x4076DF: main (Nwtun.c:55)
    n0: 1930096 in 1 place, below massif's threshold (01.00%)
   n1: 15048472 0x558547: DMPlexDistributeNwtun (plexdistribute.c:154)
    n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34)
     n0: 15048472 0x4076DF: main (Nwtun.c:55)
   n0: 1930288 in 4 places, all below massif's threshold (01.00%)
  n3: 33957328 0x52EC32: PetscSectionSetChart (vsectionis.c:389)
   n2: 16978568 0x53EA67: PetscSFDistributeSection (vsectionis.c:1998)
    n1: 15048472 0x5579E4: DMPlexDistributeNwtun (plexdistribute.c:115)
     n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34)
      n0: 15048472 0x4076DF: main (Nwtun.c:55)
    n0: 1930096 in 1 place, below massif's threshold (01.00%)
   n1: 15048472 0x558547: DMPlexDistributeNwtun (plexdistribute.c:154)
    n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34)
     n0: 15048472 0x4076DF: main (Nwtun.c:55)
   n0: 1930288 in 4 places, all below massif's threshold (01.00%)
  n1: 30096944 0x680E6B: PetscSFSetGraph (sf.c:328)
   n1: 30096944 0x6822D2: PetscSFCreateInverseSF (sf.c:478)
    n1: 30096944 0x53E180: PetscSFConvertPartition (vsectionis.c:1937)
     n1: 30096944 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101)
      n1: 30096944 0x4154E6: CreateMesh (SetupMeshes.c:34)
       n0: 30096944 0x4076DF: main (Nwtun.c:55)
  n1: 30096944 0x559905: DMPlexDistributeNwtun (plexdistribute.c:247)
   n1: 30096944 0x4154E6: CreateMesh (SetupMeshes.c:34)
    n0: 30096944 0x4076DF: main (Nwtun.c:55)
  n2: 15048472 0x681299: PetscSFSetGraph (sf.c:349)
   n1: 15048472 0x6822D2: PetscSFCreateInverseSF (sf.c:478)
    n1: 15048472 0x53E180: PetscSFConvertPartition (vsectionis.c:1937)
     n1: 15048472 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101)
      n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34)
       n0: 15048472 0x4076DF: main (Nwtun.c:55)
   n0: 0 in 2 places, all below massif's threshold (01.00%)
  n1: 15048472 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46)
   n2: 15048472 0x4E2DE5: DMLabelGetStratumIS (plexlabel.c:571)
    n1: 15048472 0x46481D: DMPlexStratify (plex.c:1768)
     n1: 15048472 0x55860F: DMPlexDistributeNwtun (plexdistribute.c:157)
      n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34)
       n0: 15048472 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
  n1: 15048472 0xC88EE6: PetscSFBasicGetPack (sfbasic.c:779)
   n1: 15048472 0xC8AA7F: PetscSFBcastBegin_Basic (sfbasic.c:914)
    n2: 15048472 0x687407: PetscSFBcastBegin (sf.c:926)
     n1: 15048472 0x66B362: ISLocalToGlobalMappingCreateSF (isltog.c:196)
      n1: 15048472 0x53E29B: PetscSFConvertPartition (vsectionis.c:1942)
       n1: 15048472 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101)
        n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34)
         n0: 15048472 0x4076DF: main (Nwtun.c:55)
     n0: 0 in 3 places, all below massif's threshold (01.00%)
  n2: 15048472 0x6812ED: PetscSFSetGraph (sf.c:349)
   n1: 15048472 0x6822D2: PetscSFCreateInverseSF (sf.c:478)
    n1: 15048472 0x53E180: PetscSFConvertPartition (vsectionis.c:1937)
     n1: 15048472 0x55740F: DMPlexDistributeNwtun (plexdistribute.c:101)
      n1: 15048472 0x4154E6: CreateMesh (SetupMeshes.c:34)
       n0: 15048472 0x4076DF: main (Nwtun.c:55)
   n0: 0 in 2 places, all below massif's threshold (01.00%)
  n1: 5790288 0xC99701: VecCreate_MPI_Private (pbvec.c:201)
   n1: 5790288 0xC99C3D: VecCreate_MPI (pbvec.c:244)
    n1: 5790288 0x6BF560: VecSetType (vecreg.c:53)
     n1: 5790288 0xC99FA5: VecCreate_Standard (pbvec.c:271)
      n1: 5790288 0x6BF560: VecSetType (vecreg.c:53)
       n1: 5790288 0x983E7C: DMPlexDistributeField (plexdistribute.c:301)
        n1: 5790288 0x558916: DMPlexDistributeNwtun (plexdistribute.c:172)
         n1: 5790288 0x4154E6: CreateMesh (SetupMeshes.c:34)
          n0: 5790288 0x4076DF: main (Nwtun.c:55)
  n0: 2095981 in 125 places, all below massif's threshold (01.00%)
 n0: 139272 in 92 places, all below massif's threshold (01.00%)
#-----------
snapshot=21
#-----------
time=1688188453621
mem_heap_B=330598157
mem_heap_extra_B=16059
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=22
#-----------
time=1696654881238
mem_heap_B=423787503
mem_heap_extra_B=28321
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=23
#-----------
time=1714030650389
mem_heap_B=423787567
mem_heap_extra_B=28337
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=24
#-----------
time=1728929437932
mem_heap_B=269391227
mem_heap_extra_B=32885
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=25
#-----------
time=1788668420970
mem_heap_B=269472603
mem_heap_extra_B=32973
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=26
#-----------
time=1791515716070
mem_heap_B=347795229
mem_heap_extra_B=38819
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=27
#-----------
time=1832411906731
mem_heap_B=347872613
mem_heap_extra_B=38899
mem_stacks_B=0
heap_tree=detailed
n4: 347872613 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n9: 336692827 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 74765536 0xC9786A: VecCreate_Seq (bvec3.c:38)
   n1: 74765536 0x6BF560: VecSetType (vecreg.c:53)
    n1: 74765536 0xC99F47: VecCreate_Standard (pbvec.c:269)
     n1: 74765536 0x6BF560: VecSetType (vecreg.c:53)
      n1: 74765536 0xA0773A: DMCreateLocalVector_Section_Private (dmi.c:65)
       n1: 74765536 0x976C9B: DMCreateLocalVector_Plex (plexcreate.c:931)
        n2: 74765536 0x50D220: DMCreateLocalVector (dm.c:709)
         n1: 67129728 0x417EA6: ConstructGeometryFVM (SetupMeshes.c:306)
          n0: 67129728 0x407898: main (Nwtun.c:60)
         n1: 7635808 0x4178BF: ConstructGeometryFVM (SetupMeshes.c:273)
          n0: 7635808 0x407898: main (Nwtun.c:60)
  n5: 51157200 0x52EC32: PetscSectionSetChart (vsectionis.c:389)
   n1: 15124576 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 15124576 0x4B39F0: DMPlexShiftSizes_Internal (plexsubmesh.c:189)
     n1: 15124576 0x4B8A41: DMPlexConstructGhostCells_Internal (plexsubmesh.c:507)
      n1: 15124576 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
       n1: 15124576 0x415800: CreateMesh (SetupMeshes.c:53)
        n0: 15124576 0x4076DF: main (Nwtun.c:55)
   n1: 15124576 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 15124576 0x4B39F0: DMPlexShiftSizes_Internal (plexsubmesh.c:189)
     n1: 15124576 0x4B8A41: DMPlexConstructGhostCells_Internal (plexsubmesh.c:507)
      n1: 15124576 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
       n1: 15124576 0x415800: CreateMesh (SetupMeshes.c:53)
        n0: 15124576 0x4076DF: main (Nwtun.c:55)
   n2: 7656952 0x52ED49: PetscSectionSetChart (vsectionis.c:392)
    n1: 5726856 0x4160EF: SetUpLocalSpace (SetupMeshes.c:112)
     n0: 5726856 0x40781E: main (Nwtun.c:59)
    n0: 1930096 in 2 places, all below massif's threshold (01.00%)
   n0: 7656952 in 9 places, all below massif's threshold (01.00%)
   n1: 5594144 0x417C82: ConstructGeometryFVM (SetupMeshes.c:292)
    n0: 5594144 0x407898: main (Nwtun.c:60)
  n5: 51157200 0x52EBD7: PetscSectionSetChart (vsectionis.c:389)
   n1: 15124576 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 15124576 0x4B39F0: DMPlexShiftSizes_Internal (plexsubmesh.c:189)
     n1: 15124576 0x4B8A41: DMPlexConstructGhostCells_Internal (plexsubmesh.c:507)
      n1: 15124576 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
       n1: 15124576 0x415800: CreateMesh (SetupMeshes.c:53)
        n0: 15124576 0x4076DF: main (Nwtun.c:55)
   n1: 15124576 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 15124576 0x4B39F0: DMPlexShiftSizes_Internal (plexsubmesh.c:189)
     n1: 15124576 0x4B8A41: DMPlexConstructGhostCells_Internal (plexsubmesh.c:507)
      n1: 15124576 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
       n1: 15124576 0x415800: CreateMesh (SetupMeshes.c:53)
        n0: 15124576 0x4076DF: main (Nwtun.c:55)
   n2: 7656952 0x52ED49: PetscSectionSetChart (vsectionis.c:392)
    n1: 5726856 0x4160EF: SetUpLocalSpace (SetupMeshes.c:112)
     n0: 5726856 0x40781E: main (Nwtun.c:59)
    n0: 1930096 in 2 places, all below massif's threshold (01.00%)
   n0: 7656952 in 9 places, all below massif's threshold (01.00%)
   n1: 5594144 0x417C82: ConstructGeometryFVM (SetupMeshes.c:292)
    n0: 5594144 0x407898: main (Nwtun.c:60)
  n1: 44832536 0x462C3D: DMSetUp_Plex (plex.c:1587)
   n1: 44832536 0x50B88B: DMSetUp (dm.c:560)
    n1: 44832536 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534)
     n1: 44832536 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
      n1: 44832536 0x415800: CreateMesh (SetupMeshes.c:53)
       n0: 44832536 0x4076DF: main (Nwtun.c:55)
  n1: 44832536 0x462A1D: DMSetUp_Plex (plex.c:1582)
   n2: 44832536 0x50B88B: DMSetUp (dm.c:560)
    n1: 44832536 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534)
     n1: 44832536 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
      n1: 44832536 0x415800: CreateMesh (SetupMeshes.c:53)
       n0: 44832536 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
  n1: 44832536 0x462AB1: DMSetUp_Plex (plex.c:1583)
   n2: 44832536 0x50B88B: DMSetUp (dm.c:560)
    n1: 44832536 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534)
     n1: 44832536 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
      n1: 44832536 0x415800: CreateMesh (SetupMeshes.c:53)
       n0: 44832536 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
  n1: 15124576 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46)
   n2: 15124576 0x4E2DE5: DMLabelGetStratumIS (plexlabel.c:571)
    n2: 15124576 0x46481D: DMPlexStratify (plex.c:1768)
     n1: 15124576 0x4B955D: DMPlexConstructGhostCells_Internal (plexsubmesh.c:560)
      n1: 15124576 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
       n1: 15124576 0x415800: CreateMesh (SetupMeshes.c:53)
        n0: 15124576 0x4076DF: main (Nwtun.c:55)
     n0: 0 in 1 place, below massif's threshold (01.00%)
    n0: 0 in 1 place, below massif's threshold (01.00%)
  n1: 5790288 0xC99701: VecCreate_MPI_Private (pbvec.c:201)
   n1: 5790288 0xC99C3D: VecCreate_MPI (pbvec.c:244)
    n1: 5790288 0x6BF560: VecSetType (vecreg.c:53)
     n1: 5790288 0xC99FA5: VecCreate_Standard (pbvec.c:271)
      n2: 5790288 0x6BF560: VecSetType (vecreg.c:53)
       n1: 5790288 0x4B5708: DMPlexShiftCoordinates_Internal (plexsubmesh.c:289)
        n1: 5790288 0x4B95CF: DMPlexConstructGhostCells_Internal (plexsubmesh.c:561)
         n1: 5790288 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
          n1: 5790288 0x415800: CreateMesh (SetupMeshes.c:53)
           n0: 5790288 0x4076DF: main (Nwtun.c:55)
       n0: 0 in 1 place, below massif's threshold (01.00%)
  n0: 4200419 in 167 places, all below massif's threshold (01.00%)
 n1: 7078832 0x4DBEB6: kh_resize_HASHI (hash.h:510)
  n2: 7078832 0x4DC2CD: kh_put_HASHI (hash.h:510)
   n2: 7078832 0x4E14DD: DMLabelSetValue (plexlabel.c:450)
    n1: 3539416 0x4B7611: DMPlexShiftLabels_Internal (plexsubmesh.c:410)
     n1: 3539416 0x4B96B3: DMPlexConstructGhostCells_Internal (plexsubmesh.c:563)
      n1: 3539416 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
       n1: 3539416 0x415800: CreateMesh (SetupMeshes.c:53)
        n0: 3539416 0x4076DF: main (Nwtun.c:55)
    n0: 3539416 in 5 places, all below massif's threshold (01.00%)
   n0: 0 in 1 place, below massif's threshold (01.00%)
 n1: 3539416 0x4DBE90: kh_resize_HASHI (hash.h:510)
  n2: 3539416 0x4DC2CD: kh_put_HASHI (hash.h:510)
   n1: 3539416 0x4E14DD: DMLabelSetValue (plexlabel.c:450)
    n0: 3539416 in 6 places, all below massif's threshold (01.00%)
   n0: 0 in 1 place, below massif's threshold (01.00%)
 n0: 561538 in 90 places, all below massif's threshold (01.00%)
#-----------
snapshot=28
#-----------
time=1842874956987
mem_heap_B=347873909
mem_heap_extra_B=38947
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=29
#-----------
time=1854113575440
mem_heap_B=347873909
mem_heap_extra_B=38947
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=30
#-----------
time=1863868911696
mem_heap_B=347873909
mem_heap_extra_B=38947
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=31
#-----------
time=1937202400181
mem_heap_B=347877637
mem_heap_extra_B=38955
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=32
#-----------
time=1946196214265
mem_heap_B=480322021
mem_heap_extra_B=81331
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=33
#-----------
time=1957929625920
mem_heap_B=868356835
mem_heap_extra_B=151861
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=34
#-----------
time=1973771437326
mem_heap_B=803234327
mem_heap_extra_B=118041
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=35
#-----------
time=1984387969511
mem_heap_B=1305045055
mem_heap_extra_B=124545
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=36
#-----------
time=2002831468916
mem_heap_B=1305045055
mem_heap_extra_B=124545
mem_stacks_B=0
heap_tree=detailed
n2: 1305045055 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n13: 1294166241 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 303810000 0x9EC1BF: DMPlexPreallocateOperator (plexpreallocate.c:337)
   n1: 303810000 0x459619: DMCreateMatrix_Plex (plex.c:615)
    n1: 303810000 0x50F42F: DMCreateMatrix (dm.c:941)
     n0: 303810000 0x40B906: main (Nwtun.c:396)
  n4: 273974304 0x52EBD7: PetscSectionSetChart (vsectionis.c:389)
   n1: 198000728 0x9EB86B: DMPlexPreallocateOperator (plexpreallocate.c:291)
    n1: 198000728 0x459619: DMCreateMatrix_Plex (plex.c:615)
     n1: 198000728 0x50F42F: DMCreateMatrix (dm.c:941)
      n0: 198000728 0x40B906: main (Nwtun.c:396)
   n0: 45724424 in 15 places, all below massif's threshold (01.00%)
   n1: 15124576 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 15124576 0x4B39F0: DMPlexShiftSizes_Internal (plexsubmesh.c:189)
     n1: 15124576 0x4B8A41: DMPlexConstructGhostCells_Internal (plexsubmesh.c:507)
      n1: 15124576 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
       n1: 15124576 0x415800: CreateMesh (SetupMeshes.c:53)
        n0: 15124576 0x4076DF: main (Nwtun.c:55)
   n1: 15124576 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 15124576 0x4B39F0: DMPlexShiftSizes_Internal (plexsubmesh.c:189)
     n1: 15124576 0x4B8A41: DMPlexConstructGhostCells_Internal (plexsubmesh.c:507)
      n1: 15124576 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
       n1: 15124576 0x415800: CreateMesh (SetupMeshes.c:53)
        n0: 15124576 0x4076DF: main (Nwtun.c:55)
  n4: 273974304 0x52EC32: PetscSectionSetChart (vsectionis.c:389)
   n1: 198000728 0x9EB86B: DMPlexPreallocateOperator (plexpreallocate.c:291)
    n1: 198000728 0x459619: DMCreateMatrix_Plex (plex.c:615)
     n1: 198000728 0x50F42F: DMCreateMatrix (dm.c:941)
      n0: 198000728 0x40B906: main (Nwtun.c:396)
   n0: 45724424 in 15 places, all below massif's threshold (01.00%)
   n1: 15124576 0x45A810: DMPlexSetChart (plex.c:729)
    n1: 15124576 0x4B39F0: DMPlexShiftSizes_Internal (plexsubmesh.c:189)
     n1: 15124576 0x4B8A41: DMPlexConstructGhostCells_Internal (plexsubmesh.c:507)
      n1: 15124576 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
       n1: 15124576 0x415800: CreateMesh (SetupMeshes.c:53)
        n0: 15124576 0x4076DF: main (Nwtun.c:55)
   n1: 15124576 0x45A874: DMPlexSetChart (plex.c:730)
    n1: 15124576 0x4B39F0: DMPlexShiftSizes_Internal (plexsubmesh.c:189)
     n1: 15124576 0x4B8A41: DMPlexConstructGhostCells_Internal (plexsubmesh.c:507)
      n1: 15124576 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
       n1: 15124576 0x415800: CreateMesh (SetupMeshes.c:53)
        n0: 15124576 0x4076DF: main (Nwtun.c:55)
  n0: 97551865 in 216 places, all below massif's threshold (01.00%)
  n1: 84310296 0xC9786A: VecCreate_Seq (bvec3.c:38)
   n1: 84310296 0x6BF560: VecSetType (vecreg.c:53)
    n1: 84310296 0xC99F47: VecCreate_Standard (pbvec.c:269)
     n1: 84310296 0x6BF560: VecSetType (vecreg.c:53)
      n1: 84310296 0xA0773A: DMCreateLocalVector_Section_Private (dmi.c:65)
       n1: 84310296 0x976C9B: DMCreateLocalVector_Plex (plexcreate.c:931)
        n2: 84310296 0x50D220: DMCreateLocalVector (dm.c:709)
         n1: 67129728 0x417EA6: ConstructGeometryFVM (SetupMeshes.c:306)
          n0: 67129728 0x407898: main (Nwtun.c:60)
         n0: 17180568 in 2 places, all below massif's threshold (01.00%)
  n1: 44832536 0x462C3D: DMSetUp_Plex (plex.c:1587)
   n1: 44832536 0x50B88B: DMSetUp (dm.c:560)
    n1: 44832536 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534)
     n1: 44832536 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
      n1: 44832536 0x415800: CreateMesh (SetupMeshes.c:53)
       n0: 44832536 0x4076DF: main (Nwtun.c:55)
  n1: 44832536 0x462AB1: DMSetUp_Plex (plex.c:1583)
   n2: 44832536 0x50B88B: DMSetUp (dm.c:560)
    n1: 44832536 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534)
     n1: 44832536 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
      n1: 44832536 0x415800: CreateMesh (SetupMeshes.c:53)
       n0: 44832536 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
  n1: 44832536 0x462A1D: DMSetUp_Plex (plex.c:1582)
   n2: 44832536 0x50B88B: DMSetUp (dm.c:560)
    n1: 44832536 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534)
     n1: 44832536 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
      n1: 44832536 0x415800: CreateMesh (SetupMeshes.c:53)
       n0: 44832536 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
  n2: 40721328 0xC99701: VecCreate_MPI_Private (pbvec.c:201)
   n1: 26198280 0xC98DCF: VecDuplicate_MPI (pbvec.c:68)
    n1: 26198280 0x6B04CF: VecDuplicate (vector.c:399)
     n0: 26198280 in 3 places, all below massif's threshold (01.00%)
   n1: 14523048 0xC99C3D: VecCreate_MPI (pbvec.c:244)
    n1: 14523048 0x6BF560: VecSetType (vecreg.c:53)
     n1: 14523048 0xC99FA5: VecCreate_Standard (pbvec.c:271)
      n1: 14523048 0x6BF560: VecSetType (vecreg.c:53)
       n0: 14523048 in 3 places, all below massif's threshold (01.00%)
  n1: 26903800 0x9EA61B: DMPlexPreallocateOperator (plexpreallocate.c:204)
   n1: 26903800 0x459619: DMCreateMatrix_Plex (plex.c:615)
    n1: 26903800 0x50F42F: DMCreateMatrix (dm.c:941)
     n0: 26903800 0x40B906: main (Nwtun.c:396)
  n1: 24893576 0x9832FA: DMPlexGetAdjacency_Internal (plexdistribute.c:220)
   n1: 24893576 0x9839A6: DMPlexGetAdjacency (plexdistribute.c:263)
    n1: 24893576 0x9E917B: DMPlexPreallocateOperator (plexpreallocate.c:101)
     n1: 24893576 0x459619: DMCreateMatrix_Plex (plex.c:615)
      n1: 24893576 0x50F42F: DMCreateMatrix (dm.c:941)
       n0: 24893576 0x40B906: main (Nwtun.c:396)
  n1: 18328480 0x521E45: DMCreateDefaultSF (dm.c:3075)
   n1: 18328480 0x520D8A: DMGetDefaultSF (dm.c:2991)
    n1: 18328480 0x5153DB: DMGlobalToLocalBegin (dm.c:1649)
     n1: 18328480 0x411EF2: SetInitialCondition (Nwtun.c:938)
      n0: 18328480 0x40B823: main (Nwtun.c:385)
  n2: 15200680 0x4DCF22: DMLabelMakeValid_Private (plexlabel.c:46)
   n2: 15124576 0x4E2DE5: DMLabelGetStratumIS (plexlabel.c:571)
    n2: 15124576 0x46481D: DMPlexStratify (plex.c:1768)
     n1: 15124576 0x4B955D: DMPlexConstructGhostCells_Internal (plexsubmesh.c:560)
      n1: 15124576 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
       n1: 15124576 0x415800: CreateMesh (SetupMeshes.c:53)
        n0: 15124576 0x4076DF: main (Nwtun.c:55)
     n0: 0 in 1 place, below massif's threshold (01.00%)
    n0: 0 in 1 place, below massif's threshold (01.00%)
   n0: 76104 in 1 place, below massif's threshold (01.00%)
 n0: 10878814 in 95 places, all below massif's threshold (01.00%)
#-----------
snapshot=37
#-----------
time=2031323634051
mem_heap_B=1215067359
mem_heap_extra_B=108017
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=38
#-----------
time=2035978518347
mem_heap_B=1875093065
mem_heap_extra_B=139567
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=39
#-----------
time=2053498338602
mem_heap_B=1875093065
mem_heap_extra_B=139567
mem_stacks_B=0
heap_tree=detailed
n2: 1875093065 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n11: 1864214251 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 303810000 0x81EE1C: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719)
   n2: 303810000 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664)
    n1: 301101800 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255)
     n1: 301101800 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959)
      n1: 301101800 0x955310: MatXAIJSetPreallocation (gcreate.c:252)
       n1: 301101800 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435)
        n1: 301101800 0x459619: DMCreateMatrix_Plex (plex.c:615)
         n1: 301101800 0x50F42F: DMCreateMatrix (dm.c:941)
          n0: 301101800 0x40B906: main (Nwtun.c:396)
    n0: 2708200 in 1 place, below massif's threshold (01.00%)
  n1: 303810000 0x81EDCA: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719)
   n2: 303810000 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664)
    n1: 301101800 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255)
     n1: 301101800 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959)
      n1: 301101800 0x955310: MatXAIJSetPreallocation (gcreate.c:252)
       n1: 301101800 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435)
        n1: 301101800 0x459619: DMCreateMatrix_Plex (plex.c:615)
         n1: 301101800 0x50F42F: DMCreateMatrix (dm.c:941)
          n0: 301101800 0x40B906: main (Nwtun.c:396)
    n0: 2708200 in 1 place, below massif's threshold (01.00%)
  n1: 303810000 0x9EC1BF: DMPlexPreallocateOperator (plexpreallocate.c:337)
   n1: 303810000 0x459619: DMCreateMatrix_Plex (plex.c:615)
    n1: 303810000 0x50F42F: DMCreateMatrix (dm.c:941)
     n0: 303810000 0x40B906: main (Nwtun.c:396)
  n2: 254884784 0x52EBD7: PetscSectionSetChart (vsectionis.c:389)
   n1: 198000728 0x9EB86B: DMPlexPreallocateOperator (plexpreallocate.c:291)
    n1: 198000728 0x459619: DMCreateMatrix_Plex (plex.c:615)
     n1: 198000728 0x50F42F: DMCreateMatrix (dm.c:941)
      n0: 198000728 0x40B906: main (Nwtun.c:396)
   n0: 56884056 in 17 places, all below massif's threshold (01.00%)
  n2: 254884784 0x52EC32: PetscSectionSetChart (vsectionis.c:389)
   n1: 198000728 0x9EB86B: DMPlexPreallocateOperator (plexpreallocate.c:291)
    n1: 198000728 0x459619: DMCreateMatrix_Plex (plex.c:615)
     n1: 198000728 0x50F42F: DMCreateMatrix (dm.c:941)
      n0: 198000728 0x40B906: main (Nwtun.c:396)
   n0: 56884056 in 17 places, all below massif's threshold (01.00%)
  n0: 183485451 in 225 places, all below massif's threshold (01.00%)
  n1: 84310296 0xC9786A: VecCreate_Seq (bvec3.c:38)
   n1: 84310296 0x6BF560: VecSetType (vecreg.c:53)
    n1: 84310296 0xC99F47: VecCreate_Standard (pbvec.c:269)
     n1: 84310296 0x6BF560: VecSetType (vecreg.c:53)
      n1: 84310296 0xA0773A: DMCreateLocalVector_Section_Private (dmi.c:65)
       n1: 84310296 0x976C9B: DMCreateLocalVector_Plex (plexcreate.c:931)
        n2: 84310296 0x50D220: DMCreateLocalVector (dm.c:709)
         n1: 67129728 0x417EA6: ConstructGeometryFVM (SetupMeshes.c:306)
          n0: 67129728 0x407898: main (Nwtun.c:60)
         n0: 17180568 in 2 places, all below massif's threshold (01.00%)
  n1: 44832536 0x462AB1: DMSetUp_Plex (plex.c:1583)
   n2: 44832536 0x50B88B: DMSetUp (dm.c:560)
    n1: 44832536 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534)
     n1: 44832536 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
      n1: 44832536 0x415800: CreateMesh (SetupMeshes.c:53)
       n0: 44832536 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
  n1: 44832536 0x462A1D: DMSetUp_Plex (plex.c:1582)
   n2: 44832536 0x50B88B: DMSetUp (dm.c:560)
    n1: 44832536 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534)
     n1: 44832536 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
      n1: 44832536 0x415800: CreateMesh (SetupMeshes.c:53)
       n0: 44832536 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
  n1: 44832536 0x462C3D: DMSetUp_Plex (plex.c:1587)
   n1: 44832536 0x50B88B: DMSetUp (dm.c:560)
    n1: 44832536 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534)
     n1: 44832536 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
      n1: 44832536 0x415800: CreateMesh (SetupMeshes.c:53)
       n0: 44832536 0x4076DF: main (Nwtun.c:55)
  n2: 40721328 0xC99701: VecCreate_MPI_Private (pbvec.c:201)
   n1: 26198280 0xC98DCF: VecDuplicate_MPI (pbvec.c:68)
    n1: 26198280 0x6B04CF: VecDuplicate (vector.c:399)
     n0: 26198280 in 3 places, all below massif's threshold (01.00%)
   n0: 14523048 in 1 place, below massif's threshold (01.00%)
 n0: 10878814 in 95 places, all below massif's threshold (01.00%)
#-----------
snapshot=40
#-----------
time=2068776186221
mem_heap_B=1919188393
mem_heap_extra_B=155015
mem_stacks_B=0
heap_tree=detailed
n2: 1919188393 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n11: 1908309579 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 303810000 0x81EE1C: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719)
   n2: 303810000 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664)
    n1: 301101800 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255)
     n1: 301101800 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959)
      n1: 301101800 0x955310: MatXAIJSetPreallocation (gcreate.c:252)
       n1: 301101800 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435)
        n1: 301101800 0x459619: DMCreateMatrix_Plex (plex.c:615)
         n1: 301101800 0x50F42F: DMCreateMatrix (dm.c:941)
          n0: 301101800 0x40B906: main (Nwtun.c:396)
    n0: 2708200 in 1 place, below massif's threshold (01.00%)
  n1: 303810000 0x81EDCA: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719)
   n2: 303810000 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664)
    n1: 301101800 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255)
     n1: 301101800 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959)
      n1: 301101800 0x955310: MatXAIJSetPreallocation (gcreate.c:252)
       n1: 301101800 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435)
        n1: 301101800 0x459619: DMCreateMatrix_Plex (plex.c:615)
         n1: 301101800 0x50F42F: DMCreateMatrix (dm.c:941)
          n0: 301101800 0x40B906: main (Nwtun.c:396)
    n0: 2708200 in 1 place, below massif's threshold (01.00%)
  n1: 303810000 0x9EC1BF: DMPlexPreallocateOperator (plexpreallocate.c:337)
   n1: 303810000 0x459619: DMCreateMatrix_Plex (plex.c:615)
    n1: 303810000 0x50F42F: DMCreateMatrix (dm.c:941)
     n0: 303810000 0x40B906: main (Nwtun.c:396)
  n2: 254884784 0x52EBD7: PetscSectionSetChart (vsectionis.c:389)
   n1: 198000728 0x9EB86B: DMPlexPreallocateOperator (plexpreallocate.c:291)
    n1: 198000728 0x459619: DMCreateMatrix_Plex (plex.c:615)
     n1: 198000728 0x50F42F: DMCreateMatrix (dm.c:941)
      n0: 198000728 0x40B906: main (Nwtun.c:396)
   n0: 56884056 in 17 places, all below massif's threshold (01.00%)
  n2: 254884784 0x52EC32: PetscSectionSetChart (vsectionis.c:389)
   n1: 198000728 0x9EB86B: DMPlexPreallocateOperator (plexpreallocate.c:291)
    n1: 198000728 0x459619: DMCreateMatrix_Plex (plex.c:615)
     n1: 198000728 0x50F42F: DMCreateMatrix (dm.c:941)
      n0: 198000728 0x40B906: main (Nwtun.c:396)
   n0: 56884056 in 17 places, all below massif's threshold (01.00%)
  n0: 227580779 in 236 places, all below massif's threshold (01.00%)
  n1: 84310296 0xC9786A: VecCreate_Seq (bvec3.c:38)
   n1: 84310296 0x6BF560: VecSetType (vecreg.c:53)
    n1: 84310296 0xC99F47: VecCreate_Standard (pbvec.c:269)
     n1: 84310296 0x6BF560: VecSetType (vecreg.c:53)
      n1: 84310296 0xA0773A: DMCreateLocalVector_Section_Private (dmi.c:65)
       n1: 84310296 0x976C9B: DMCreateLocalVector_Plex (plexcreate.c:931)
        n2: 84310296 0x50D220: DMCreateLocalVector (dm.c:709)
         n1: 67129728 0x417EA6: ConstructGeometryFVM (SetupMeshes.c:306)
          n0: 67129728 0x407898: main (Nwtun.c:60)
         n0: 17180568 in 2 places, all below massif's threshold (01.00%)
  n1: 44832536 0x462AB1: DMSetUp_Plex (plex.c:1583)
   n2: 44832536 0x50B88B: DMSetUp (dm.c:560)
    n1: 44832536 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534)
     n1: 44832536 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
      n1: 44832536 0x415800: CreateMesh (SetupMeshes.c:53)
       n0: 44832536 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
  n1: 44832536 0x462A1D: DMSetUp_Plex (plex.c:1582)
   n2: 44832536 0x50B88B: DMSetUp (dm.c:560)
    n1: 44832536 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534)
     n1: 44832536 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
      n1: 44832536 0x415800: CreateMesh (SetupMeshes.c:53)
       n0: 44832536 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
  n1: 44832536 0x462C3D: DMSetUp_Plex (plex.c:1587)
   n1: 44832536 0x50B88B: DMSetUp (dm.c:560)
    n1: 44832536 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534)
     n1: 44832536 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
      n1: 44832536 0x415800: CreateMesh (SetupMeshes.c:53)
       n0: 44832536 0x4076DF: main (Nwtun.c:55)
  n2: 40721328 0xC99701: VecCreate_MPI_Private (pbvec.c:201)
   n1: 26198280 0xC98DCF: VecDuplicate_MPI (pbvec.c:68)
    n1: 26198280 0x6B04CF: VecDuplicate (vector.c:399)
     n0: 26198280 in 3 places, all below massif's threshold (01.00%)
   n0: 14523048 in 1 place, below massif's threshold (01.00%)
 n0: 10878814 in 95 places, all below massif's threshold (01.00%)
#-----------
snapshot=41
#-----------
time=2085612748660
mem_heap_B=1254369333
mem_heap_extra_B=150667
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=42
#-----------
time=2099570407105
mem_heap_B=1281711289
mem_heap_extra_B=151679
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=43
#-----------
time=2111798447020
mem_heap_B=1310347781
mem_heap_extra_B=152587
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=44
#-----------
time=2130810038150
mem_heap_B=1310347885
mem_heap_extra_B=152603
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=45
#-----------
time=2137035244977
mem_heap_B=1540712799
mem_heap_extra_B=165705
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=46
#-----------
time=2189002748743
mem_heap_B=1540710923
mem_heap_extra_B=165669
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=47
#-----------
time=2190640466102
mem_heap_B=1540711699
mem_heap_extra_B=165653
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=48
#-----------
time=2255398115514
mem_heap_B=1540713443
mem_heap_extra_B=165685
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=49
#-----------
time=2256803909597
mem_heap_B=1645503790
mem_heap_extra_B=167378
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=50
#-----------
time=2256808280096
mem_heap_B=1645503766
mem_heap_extra_B=167362
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=51
#-----------
time=2256811949338
mem_heap_B=1645502870
mem_heap_extra_B=167330
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=52
#-----------
time=2256814355812
mem_heap_B=1645503766
mem_heap_extra_B=167362
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=53
#-----------
time=2256824000668
mem_heap_B=1645502022
mem_heap_extra_B=167330
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=54
#-----------
time=2256826285854
mem_heap_B=1645502894
mem_heap_extra_B=167346
mem_stacks_B=0
heap_tree=detailed
n2: 1645502894 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n22: 1634623016 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 303810000 0x81EE1C: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719)
   n2: 303810000 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664)
    n1: 301101800 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255)
     n1: 301101800 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959)
      n1: 301101800 0x955310: MatXAIJSetPreallocation (gcreate.c:252)
       n1: 301101800 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435)
        n1: 301101800 0x459619: DMCreateMatrix_Plex (plex.c:615)
         n1: 301101800 0x50F42F: DMCreateMatrix (dm.c:941)
          n0: 301101800 0x40B906: main (Nwtun.c:396)
    n0: 2708200 in 1 place, below massif's threshold (01.00%)
  n1: 303810000 0x81EDCA: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719)
   n2: 303810000 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664)
    n1: 301101800 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255)
     n1: 301101800 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959)
      n1: 301101800 0x955310: MatXAIJSetPreallocation (gcreate.c:252)
       n1: 301101800 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435)
        n1: 301101800 0x459619: DMCreateMatrix_Plex (plex.c:615)
         n1: 301101800 0x50F42F: DMCreateMatrix (dm.c:941)
          n0: 301101800 0x40B906: main (Nwtun.c:396)
    n0: 2708200 in 1 place, below massif's threshold (01.00%)
  n1: 151555096 0xC9786A: VecCreate_Seq (bvec3.c:38)
   n2: 151555096 0x6BF560: VecSetType (vecreg.c:53)
    n1: 151123616 0xC99F47: VecCreate_Standard (pbvec.c:269)
     n1: 151123616 0x6BF560: VecSetType (vecreg.c:53)
      n1: 151123616 0xA0773A: DMCreateLocalVector_Section_Private (dmi.c:65)
       n1: 151123616 0x976C9B: DMCreateLocalVector_Plex (plexcreate.c:931)
        n3: 151123616 0x50D220: DMCreateLocalVector (dm.c:709)
         n3: 76358080 0xA035E0: DMGetLocalVector (dmget.c:49)
          n1: 28634280 0x425614: CaculateLocalFunction_LS (SetupFunctions.c:742)
           n1: 28634280 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
            n1: 28634280 0x41E3AB: FormFunction (SetupFunctions.c:52)
             n1: 28634280 0xA8D3FC: SNESComputeFunction (snes.c:2033)
              n1: 28634280 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
               n1: 28634280 0xA9E3CC: SNESSolve (snes.c:3743)
                n1: 28634280 0x40FB9B: SolveTimeDependent (Nwtun.c:758)
                 n0: 28634280 0x40BC62: main (Nwtun.c:417)
          n1: 28634280 0x4257D8: CaculateLocalFunction_LS (SetupFunctions.c:748)
           n1: 28634280 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
            n1: 28634280 0x41E3AB: FormFunction (SetupFunctions.c:52)
             n1: 28634280 0xA8D3FC: SNESComputeFunction (snes.c:2033)
              n1: 28634280 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
               n1: 28634280 0xA9E3CC: SNESSolve (snes.c:3743)
                n1: 28634280 0x40FB9B: SolveTimeDependent (Nwtun.c:758)
                 n0: 28634280 0x40BC62: main (Nwtun.c:417)
          n0: 19089520 in 2 places, all below massif's threshold (01.00%)
         n1: 67129728 0x417EA6: ConstructGeometryFVM (SetupMeshes.c:306)
          n0: 67129728 0x407898: main (Nwtun.c:60)
         n0: 7635808 in 1 place, below massif's threshold (01.00%)
    n0: 431480 in 1 place, below massif's threshold (01.00%)
  n2: 111724968 0xC99701: VecCreate_MPI_Private (pbvec.c:201)
   n1: 61129320 0xC98DCF: VecDuplicate_MPI (pbvec.c:68)
    n2: 61129320 0x6B04CF: VecDuplicate (vector.c:399)
     n0: 43663800 in 6 places, all below massif's threshold (01.00%)
     n1: 17465520 0x6B434F: VecDuplicateVecs_Default (vector.c:827)
      n1: 17465520 0x6B11C4: VecDuplicateVecs (vector.c:473)
       n1: 17465520 0xAAFB50: SNESSetWorkVecs (snesut.c:562)
        n1: 17465520 0xAF8B57: SNESSetUp_NEWTONLS (ls.c:317)
         n1: 17465520 0xA9489F: SNESSetUp (snes.c:2616)
          n1: 17465520 0xA9E184: SNESSolve (snes.c:3731)
           n1: 17465520 0x40FB9B: SolveTimeDependent (Nwtun.c:758)
            n0: 17465520 0x40BC62: main (Nwtun.c:417)
   n1: 50595648 0xC99C3D: VecCreate_MPI (pbvec.c:244)
    n1: 50595648 0x6BF560: VecSetType (vecreg.c:53)
     n1: 50595648 0xC99FA5: VecCreate_Standard (pbvec.c:271)
      n2: 50595648 0x6BF560: VecSetType (vecreg.c:53)
       n1: 44805360 0xA07131: DMCreateGlobalVector_Section_Private (dmi.c:34)
        n1: 44805360 0x976932: DMCreateGlobalVector_Plex (plexcreate.c:917)
         n2: 44805360 0x50CE38: DMCreateGlobalVector (dm.c:681)
          n2: 36072600 0xA04005: DMGetGlobalVector (dmget.c:154)
           n1: 27339840 0x426EC6: ConstructCellCentriodGradient (SetupFunctions.c:855)
            n1: 27339840 0x42054B: FormTimeStepFunction (SetupFunctions.c:233)
             n1: 27339840 0x41E3AB: FormFunction (SetupFunctions.c:52)
              n1: 27339840 0xA8D3FC: SNESComputeFunction (snes.c:2033)
               n1: 27339840 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
                n1: 27339840 0xA9E3CC: SNESSolve (snes.c:3743)
                 n1: 27339840 0x40FB9B: SolveTimeDependent (Nwtun.c:758)
                  n0: 27339840 0x40BC62: main (Nwtun.c:417)
           n0: 8732760 in 1 place, below massif's threshold (01.00%)
          n0: 8732760 in 1 place, below massif's threshold (01.00%)
       n0: 5790288 in 2 places, all below massif's threshold (01.00%)
  n1: 104790400 0x905DD7: MatGetSubMatrix_MPIAIJ_All (mpiov.c:549)
   n1: 104790400 0x8DE981: MatGetSeqNonzeroStructure_MPIAIJ (mpiaij.c:3010)
    n1: 104790400 0x78FDBF: MatGetSeqNonzeroStructure (matrix.c:6487)
     n1: 104790400 0xCA31E2: MatColoringApply_SL (color.c:78)
      n1: 104790400 0x746C6C: MatColoringApply (matcoloring.c:379)
       n1: 104790400 0xAB0879: SNESComputeJacobianDefaultColor (snesj2.c:71)
        n1: 104790400 0x4349AD: FormJacobian (SetupJacobian.c:58)
         n1: 104790400 0xA8F850: SNESComputeJacobian (snes.c:2193)
          n1: 104790400 0xAF7C15: SNESSolve_NEWTONLS (ls.c:230)
           n1: 104790400 0xA9E3CC: SNESSolve (snes.c:3743)
            n1: 104790400 0x40FB9B: SolveTimeDependent (Nwtun.c:758)
             n0: 104790400 0x40BC62: main (Nwtun.c:417)
  n1: 75597040 0x521E45: DMCreateDefaultSF (dm.c:3075)
   n1: 75597040 0x520D8A: DMGetDefaultSF (dm.c:2991)
    n2: 75597040 0x5153DB: DMGlobalToLocalBegin (dm.c:1649)
     n1: 57268560 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
      n1: 57268560 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
       n1: 57268560 0x41E3AB: FormFunction (SetupFunctions.c:52)
        n1: 57268560 0xA8D3FC: SNESComputeFunction (snes.c:2033)
         n1: 57268560 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
          n1: 57268560 0xA9E3CC: SNESSolve (snes.c:3743)
           n1: 57268560 0x40FB9B: SolveTimeDependent (Nwtun.c:758)
            n0: 57268560 0x40BC62: main (Nwtun.c:417)
     n1: 18328480 0x411EF2: SetInitialCondition (Nwtun.c:938)
      n0: 18328480 0x40B823: main (Nwtun.c:385)
  n1: 58793008 0x52EBD7: PetscSectionSetChart (vsectionis.c:389)
   n0: 58793008 in 18 places, all below massif's threshold (01.00%)
  n1: 58793008 0x52EC32: PetscSectionSetChart (vsectionis.c:389)
   n0: 58793008 in 18 places, all below massif's threshold (01.00%)
  n1: 44832536 0x462A1D: DMSetUp_Plex (plex.c:1582)
   n2: 44832536 0x50B88B: DMSetUp (dm.c:560)
    n1: 44832536 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534)
     n1: 44832536 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
      n1: 44832536 0x415800: CreateMesh (SetupMeshes.c:53)
       n0: 44832536 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
  n1: 44832536 0x462C3D: DMSetUp_Plex (plex.c:1587)
   n1: 44832536 0x50B88B: DMSetUp (dm.c:560)
    n1: 44832536 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534)
     n1: 44832536 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
      n1: 44832536 0x415800: CreateMesh (SetupMeshes.c:53)
       n0: 44832536 0x4076DF: main (Nwtun.c:55)
  n1: 44832536 0x462AB1: DMSetUp_Plex (plex.c:1583)
   n2: 44832536 0x50B88B: DMSetUp (dm.c:560)
    n1: 44832536 0x4B9023: DMPlexConstructGhostCells_Internal (plexsubmesh.c:534)
     n1: 44832536 0x4BA10E: DMPlexConstructGhostCells (plexsubmesh.c:618)
      n1: 44832536 0x415800: CreateMesh (SetupMeshes.c:53)
       n0: 44832536 0x4076DF: main (Nwtun.c:55)
    n0: 0 in 1 place, below massif's threshold (01.00%)
  n1: 38949904 0xC861ED: PetscSFSetUp_Basic (sfbasic.c:557)
   n2: 38949904 0x67F6D7: PetscSFSetUp (sf.c:192)
    n2: 38949904 0x68739A: PetscSFBcastBegin (sf.c:925)
     n2: 37794200 0x51554F: DMGlobalToLocalBegin (dm.c:1656)
      n1: 28631040 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
       n1: 28631040 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
        n1: 28631040 0x41E3AB: FormFunction (SetupFunctions.c:52)
         n1: 28631040 0xA8D3FC: SNESComputeFunction (snes.c:2033)
          n1: 28631040 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
           n1: 28631040 0xA9E3CC: SNESSolve (snes.c:3743)
            n1: 28631040 0x40FB9B: SolveTimeDependent (Nwtun.c:758)
             n0: 28631040 0x40BC62: main (Nwtun.c:417)
      n0: 9163160 in 1 place, below massif's threshold (01.00%)
     n0: 1155704 in 3 places, all below massif's threshold (01.00%)
    n0: 0 in 2 places, all below massif's threshold (01.00%)
  n2: 38949904 0xC88E56: PetscSFBasicGetPack (sfbasic.c:779)
   n1: 38949904 0xC8AA7F: PetscSFBcastBegin_Basic (sfbasic.c:914)
    n2: 38949904 0x687407: PetscSFBcastBegin (sf.c:926)
     n2: 37794200 0x51554F: DMGlobalToLocalBegin (dm.c:1656)
      n1: 28631040 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
       n1: 28631040 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
        n1: 28631040 0x41E3AB: FormFunction (SetupFunctions.c:52)
         n1: 28631040 0xA8D3FC: SNESComputeFunction (snes.c:2033)
          n1: 28631040 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
           n1: 28631040 0xA9E3CC: SNESSolve (snes.c:3743)
            n1: 28631040 0x40FB9B: SolveTimeDependent (Nwtun.c:758)
             n0: 28631040 0x40BC62: main (Nwtun.c:417)
      n0: 9163160 in 1 place, below massif's threshold (01.00%)
     n0: 1155704 in 3 places, all below massif's threshold (01.00%)
   n0: 0 in 1 place, below massif's threshold (01.00%)
  n2: 38507800 0xC88EE6: PetscSFBasicGetPack (sfbasic.c:779)
   n1: 38507800 0xC8AA7F: PetscSFBcastBegin_Basic (sfbasic.c:914)
    n2: 38507800 0x687407: PetscSFBcastBegin (sf.c:926)
     n2: 37798520 0x51554F: DMGlobalToLocalBegin (dm.c:1656)
      n1: 28634280 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
       n1: 28634280 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
        n1: 28634280 0x41E3AB: FormFunction (SetupFunctions.c:52)
         n1: 28634280 0xA8D3FC: SNESComputeFunction (snes.c:2033)
          n1: 28634280 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
           n1: 28634280 0xA9E3CC: SNESSolve (snes.c:3743)
            n1: 28634280 0x40FB9B: SolveTimeDependent (Nwtun.c:758)
             n0: 28634280 0x40BC62: main (Nwtun.c:417)
      n0: 9164240 in 1 place, below massif's threshold (01.00%)
     n0: 709280 in 8 places, all below massif's threshold (01.00%)
   n0: 0 in 1 place, below massif's threshold (01.00%)
  n2: 38507800 0x6812ED: PetscSFSetGraph (sf.c:349)
   n1: 37798520 0x522622: DMCreateDefaultSF (dm.c:3116)
    n1: 37798520 0x520D8A: DMGetDefaultSF (dm.c:2991)
     n2: 37798520 0x5153DB: DMGlobalToLocalBegin (dm.c:1649)
      n1: 28634280 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
       n1: 28634280 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
        n1: 28634280 0x41E3AB: FormFunction (SetupFunctions.c:52)
         n1: 28634280 0xA8D3FC: SNESComputeFunction (snes.c:2033)
          n1: 28634280 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
           n1: 28634280 0xA9E3CC: SNESSolve (snes.c:3743)
            n1: 28634280 0x40FB9B: SolveTimeDependent (Nwtun.c:758)
             n0: 28634280 0x40BC62: main (Nwtun.c:417)
      n0: 9164240 in 1 place, below massif's threshold (01.00%)
   n0: 709280 in 7 places, all below massif's threshold (01.00%)
  n2: 38507800 0x681299: PetscSFSetGraph (sf.c:349)
   n1: 37798520 0x522622: DMCreateDefaultSF (dm.c:3116)
    n1: 37798520 0x520D8A: DMGetDefaultSF (dm.c:2991)
     n2: 37798520 0x5153DB: DMGlobalToLocalBegin (dm.c:1649)
      n1: 28634280 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
       n1: 28634280 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
        n1: 28634280 0x41E3AB: FormFunction (SetupFunctions.c:52)
         n1: 28634280 0xA8D3FC: SNESComputeFunction (snes.c:2033)
          n1: 28634280 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
           n1: 28634280 0xA9E3CC: SNESSolve (snes.c:3743)
            n1: 28634280 0x40FB9B: SolveTimeDependent (Nwtun.c:758)
             n0: 28634280 0x40BC62: main (Nwtun.c:417)
      n0: 9164240 in 1 place, below massif's threshold (01.00%)
   n0: 709280 in 7 places, all below massif's threshold (01.00%)
  n1: 37798520 0x521DAD: DMCreateDefaultSF (dm.c:3074)
   n1: 37798520 0x520D8A: DMGetDefaultSF (dm.c:2991)
    n2: 37798520 0x5153DB: DMGlobalToLocalBegin (dm.c:1649)
     n1: 28634280 0x42568B: CaculateLocalFunction_LS (SetupFunctions.c:743)
      n1: 28634280 0x420744: FormTimeStepFunction (SetupFunctions.c:247)
       n1: 28634280 0x41E3AB: FormFunction (SetupFunctions.c:52)
        n1: 28634280 0xA8D3FC: SNESComputeFunction (snes.c:2033)
         n1: 28634280 0xAF6DC2: SNESSolve_NEWTONLS (ls.c:174)
          n1: 28634280 0xA9E3CC: SNESSolve (snes.c:3743)
           n1: 28634280 0x40FB9B: SolveTimeDependent (Nwtun.c:758)
            n0: 28634280 0x40BC62: main (Nwtun.c:417)
     n0: 9164240 in 1 place, below massif's threshold (01.00%)
  n0: 30168064 in 290 places, all below massif's threshold (01.00%)
  n1: 17465536 0x81EE87: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719)
   n1: 17465536 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664)
    n0: 17465536 in 2 places, all below massif's threshold (01.00%)
  n1: 17465520 0x809EF6: MatMarkDiagonal_SeqAIJ (aij.c:1654)
   n1: 17465520 0x80490C: MatAssemblyEnd_SeqAIJ (aij.c:1018)
    n1: 17465520 0x7832CB: MatAssemblyEnd (matrix.c:4905)
     n0: 17465520 in 2 places, all below massif's threshold (01.00%)
  n1: 17465520 0x81EA9A: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3702)
   n1: 17465520 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664)
    n0: 17465520 in 2 places, all below massif's threshold (01.00%)
  n1: 17465520 0x81EAFA: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3702)
   n1: 17465520 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664)
    n0: 17465520 in 2 places, all below massif's threshold (01.00%)
 n0: 10879878 in 95 places, all below massif's threshold (01.00%)
#-----------
snapshot=55
#-----------
time=2256829364156
mem_heap_B=1645503742
mem_heap_extra_B=167346
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=56
#-----------
time=2256831333111
mem_heap_B=1645503254
mem_heap_extra_B=167330
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=57
#-----------
time=2257106618788
mem_heap_B=1750296811
mem_heap_extra_B=169525
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=58
#-----------
time=2257538879453
mem_heap_B=5501562411
mem_heap_extra_B=172165
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=59
#-----------
time=2257905647280
mem_heap_B=9252828027
mem_heap_extra_B=174813
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=60
#-----------
time=2259161343696
mem_heap_B=9252828097
mem_heap_extra_B=174831
mem_stacks_B=0
heap_tree=empty
#-----------
snapshot=61
#-----------
time=2259161345536
mem_heap_B=9252828097
mem_heap_extra_B=174831
mem_stacks_B=0
heap_tree=peak
n2: 9252828097 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
 n9: 9241949045 0x57877E: PetscMallocAlign (mal.c:27)
  n1: 3950285200 0x81EE1C: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719)
   n3: 3950285200 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664)
    n1: 3646475200 0x906416: MatGetSubMatrix_MPIAIJ_All (mpiov.c:571)
     n1: 3646475200 0x8DE981: MatGetSeqNonzeroStructure_MPIAIJ (mpiaij.c:3010)
      n1: 3646475200 0x78FDBF: MatGetSeqNonzeroStructure (matrix.c:6487)
       n1: 3646475200 0xCA31E2: MatColoringApply_SL (color.c:78)
        n1: 3646475200 0x746C6C: MatColoringApply (matcoloring.c:379)
         n1: 3646475200 0xAB0879: SNESComputeJacobianDefaultColor (snesj2.c:71)
          n1: 3646475200 0x4349AD: FormJacobian (SetupJacobian.c:58)
           n1: 3646475200 0xA8F850: SNESComputeJacobian (snes.c:2193)
            n1: 3646475200 0xAF7C15: SNESSolve_NEWTONLS (ls.c:230)
             n1: 3646475200 0xA9E3CC: SNESSolve (snes.c:3743)
              n1: 3646475200 0x40FB9B: SolveTimeDependent (Nwtun.c:758)
               n0: 3646475200 0x40BC62: main (Nwtun.c:417)
    n1: 301101800 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255)
     n1: 301101800 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959)
      n1: 301101800 0x955310: MatXAIJSetPreallocation (gcreate.c:252)
       n1: 301101800 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435)
        n1: 301101800 0x459619: DMCreateMatrix_Plex (plex.c:615)
         n1: 301101800 0x50F42F: DMCreateMatrix (dm.c:941)
          n0: 301101800 0x40B906: main (Nwtun.c:396)
    n0: 2708200 in 1 place, below massif's threshold (01.00%)
  n1: 3950285200 0x81EDCA: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719)
   n3: 3950285200 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664)
    n1: 3646475200 0x906416: MatGetSubMatrix_MPIAIJ_All (mpiov.c:571)
     n1: 3646475200 0x8DE981: MatGetSeqNonzeroStructure_MPIAIJ (mpiaij.c:3010)
      n1: 3646475200 0x78FDBF: MatGetSeqNonzeroStructure (matrix.c:6487)
       n1: 3646475200 0xCA31E2: MatColoringApply_SL (color.c:78)
        n1: 3646475200 0x746C6C: MatColoringApply (matcoloring.c:379)
         n1: 3646475200 0xAB0879: SNESComputeJacobianDefaultColor (snesj2.c:71)
          n1: 3646475200 0x4349AD: FormJacobian (SetupJacobian.c:58)
           n1: 3646475200 0xA8F850: SNESComputeJacobian (snes.c:2193)
            n1: 3646475200 0xAF7C15: SNESSolve_NEWTONLS (ls.c:230)
             n1: 3646475200 0xA9E3CC: SNESSolve (snes.c:3743)
              n1: 3646475200 0x40FB9B: SolveTimeDependent (Nwtun.c:758)
               n0: 3646475200 0x40BC62: main (Nwtun.c:417)
    n1: 301101800 0x8DFE55: MatMPIAIJSetPreallocation_MPIAIJ (mpiaij.c:3255)
     n1: 301101800 0x8E5C00: MatMPIAIJSetPreallocation (mpiaij.c:3959)
      n1: 301101800 0x955310: MatXAIJSetPreallocation (gcreate.c:252)
       n1: 301101800 0x9ED60A: DMPlexPreallocateOperator (plexpreallocate.c:435)
        n1: 301101800 0x459619: DMCreateMatrix_Plex (plex.c:615)
         n1: 301101800 0x50F42F: DMCreateMatrix (dm.c:941)
          n0: 301101800 0x40B906: main (Nwtun.c:396)
    n0: 2708200 in 1 place, below massif's threshold (01.00%)
  n0: 606540397 in 304 places, all below massif's threshold (01.00%)
  n1: 151555096 0xC9786A: VecCreate_Seq (bvec3.c:38)
   n2: 151555096 0x6BF560: VecSetType (vecreg.c:53)
    n1: 151123616 0xC99F47: VecCreate_Standard (pbvec.c:269)
     n1: 151123616 0x6BF560: VecSetType (vecreg.c:53)
      n1: 151123616 0xA0773A: DMCreateLocalVector_Section_Private (dmi.c:65)
       n1: 151123616 0x976C9B: DMCreateLocalVector_Plex (plexcreate.c:931)
        n1: 151123616 0x50D220: DMCreateLocalVector (dm.c:709)
         n0: 151123616 in 3 places, all below massif's threshold (01.00%)
    n0: 431480 in 1 place, below massif's threshold (01.00%)
  n1: 122255944 0x81EE87: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3719)
   n2: 122255944 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664)
    n1: 104790408 0x906416: MatGetSubMatrix_MPIAIJ_All (mpiov.c:571)
     n1: 104790408 0x8DE981: MatGetSeqNonzeroStructure_MPIAIJ (mpiaij.c:3010)
      n1: 104790408 0x78FDBF: MatGetSeqNonzeroStructure (matrix.c:6487)
       n1: 104790408 0xCA31E2: MatColoringApply_SL (color.c:78)
        n1: 104790408 0x746C6C: MatColoringApply (matcoloring.c:379)
         n1: 104790408 0xAB0879: SNESComputeJacobianDefaultColor (snesj2.c:71)
          n1: 104790408 0x4349AD: FormJacobian (SetupJacobian.c:58)
           n1: 104790408 0xA8F850: SNESComputeJacobian (snes.c:2193)
            n1: 104790408 0xAF7C15: SNESSolve_NEWTONLS (ls.c:230)
             n1: 104790408 0xA9E3CC: SNESSolve (snes.c:3743)
              n1: 104790408 0x40FB9B: SolveTimeDependent (Nwtun.c:758)
               n0: 104790408 0x40BC62: main (Nwtun.c:417)
    n0: 17465536 in 2 places, all below massif's threshold (01.00%)
  n1: 122255920 0x81EAFA: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3702)
   n2: 122255920 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664)
    n1: 104790400 0x906416: MatGetSubMatrix_MPIAIJ_All (mpiov.c:571)
     n1: 104790400 0x8DE981: MatGetSeqNonzeroStructure_MPIAIJ (mpiaij.c:3010)
      n1: 104790400 0x78FDBF: MatGetSeqNonzeroStructure (matrix.c:6487)
       n1: 104790400 0xCA31E2: MatColoringApply_SL (color.c:78)
        n1: 104790400 0x746C6C: MatColoringApply (matcoloring.c:379)
         n1: 104790400 0xAB0879: SNESComputeJacobianDefaultColor (snesj2.c:71)
          n1: 104790400 0x4349AD: FormJacobian (SetupJacobian.c:58)
           n1: 104790400 0xA8F850: SNESComputeJacobian (snes.c:2193)
            n1: 104790400 0xAF7C15: SNESSolve_NEWTONLS (ls.c:230)
             n1: 104790400 0xA9E3CC: SNESSolve (snes.c:3743)
              n1: 104790400 0x40FB9B: SolveTimeDependent (Nwtun.c:758)
               n0: 104790400 0x40BC62: main (Nwtun.c:417)
    n0: 17465520 in 2 places, all below massif's threshold (01.00%)
  n1: 122255920 0x81EA9A: MatSeqAIJSetPreallocation_SeqAIJ (aij.c:3702)
   n2: 122255920 0x81E4C1: MatSeqAIJSetPreallocation (aij.c:3664)
    n1: 104790400 0x906416: MatGetSubMatrix_MPIAIJ_All (mpiov.c:571)
     n1: 104790400 0x8DE981: MatGetSeqNonzeroStructure_MPIAIJ (mpiaij.c:3010)
      n1: 104790400 0x78FDBF: MatGetSeqNonzeroStructure (matrix.c:6487)
       n1: 104790400 0xCA31E2: MatColoringApply_SL (color.c:78)
        n1: 104790400 0x746C6C: MatColoringApply (matcoloring.c:379)
         n1: 104790400 0xAB0879: SNESComputeJacobianDefaultColor (snesj2.c:71)
          n1: 104790400 0x4349AD: FormJacobian (SetupJacobian.c:58)
           n1: 104790400 0xA8F850: SNESComputeJacobian (snes.c:2193)
            n1: 104790400 0xAF7C15: SNESSolve_NEWTONLS (ls.c:230)
             n1: 104790400 0xA9E3CC: SNESSolve (snes.c:3743)
              n1: 104790400 0x40FB9B: SolveTimeDependent (Nwtun.c:758)
               n0: 104790400 0x40BC62: main (Nwtun.c:417)
    n0: 17465520 in 2 places, all below massif's threshold (01.00%)
  n1: 111724968 0xC99701: VecCreate_MPI_Private (pbvec.c:201)
   n0: 111724968 in 2 places, all below massif's threshold (01.00%)
  n1: 104790400 0x905DD7: MatGetSubMatrix_MPIAIJ_All (mpiov.c:549)
   n1: 104790400 0x8DE981: MatGetSeqNonzeroStructure_MPIAIJ (mpiaij.c:3010)
    n1: 104790400 0x78FDBF: MatGetSeqNonzeroStructure (matrix.c:6487)
     n1: 104790400 0xCA31E2: MatColoringApply_SL (color.c:78)
      n1: 104790400 0x746C6C: MatColoringApply (matcoloring.c:379)
       n1: 104790400 0xAB0879: SNESComputeJacobianDefaultColor (snesj2.c:71)
        n1: 104790400 0x4349AD: FormJacobian (SetupJacobian.c:58)
         n1: 104790400 0xA8F850: SNESComputeJacobian (snes.c:2193)
          n1: 104790400 0xAF7C15: SNESSolve_NEWTONLS (ls.c:230)
           n1: 104790400 0xA9E3CC: SNESSolve (snes.c:3743)
            n1: 104790400 0x40FB9B: SolveTimeDependent (Nwtun.c:758)
             n0: 104790400 0x40BC62: main (Nwtun.c:417)
 n0: 10879052 in 95 places, all below massif's threshold (01.00%)


More information about the petsc-users mailing list