[petsc-users] Bad memory scaling with PETSc 3.10

Myriam Peyrounette myriam.peyrounette at idris.fr
Fri Mar 8 03:10:30 CST 2019


Hi all,

thanks a lot for all your suggestions. I followed Jed's advice and
focused on the smallest case (problem size = 1e5) to find the origin of
the memory gap. And I ran my jobs on a single core.

I first used a homemade script to plot memory and time consumption: see
mem_consumption.png and time_consumption.png attached. The steps
displayed correspond to checkpoints I defined in the main file (search
for keyword "STEP" in the attached main file if you need to locate them).

We can see that both the time and the memory needs increase while using
petsc 3.10. With regard to memory consumption, the memory gap (of
~135,000,000B) is not critical yet for such a problem size, but will be
for larger problems. According to these graphs, something clearly
happens while calling KSPSolve. The code also spends more time building
the matrix.

To dig deeper, I printed the LogView outputs (logView_petsc3XX.log). In
particular, we see that the memory distribution among the petsc object
types is very different depending on the petsc version. I highlighted
this in log_view_mem_distr.pdf by sorting the petsc object types
according to their memory use, and computing the difference between
petsc 3.10 and petsc 3.6  (column on the rigth). But I don't know how to
understand that...

Finally, I used Massif (massif_petscXX.out). The total memory gap of
~135,000,000B is verified. The outputs further indicate that most of the
memory space is required by DMCreateMatrix (1,174,436,836B). This value
is almost the same is both versions so I don't think this is the problem.  

You mentioned PtAP: MatPtAPSymbolic_SeqAIJ_SeqMAIJ needs 56,277,776B
with petsc 3.6, and 112,562,000B in petsc 3.10 (twice more). So it is a
good start but it does not correspond to the total memory gap. I will
try the option "-matptap via scalable" to see if there is an improvement.

I also find a gap at KSPSolve, which needs up to 173,253,112B with petsc
3.10 (line 461 in massif_petsc310.out), and no more than  146,825,952B
(line 1534 in massif_petsc36.out) with petsc 3.6. But again, the sum of
these gaps does not fit the total gap. 

I am not very familiar with massif, so maybe you'll see additional
relevant information?

Best,

Myriam


Le 03/06/19 à 05:35, Jed Brown a écrit :
> Myriam, in your first message, there was a significant (about 50%)
> increase in memory consumption already on 4 cores.  Before attacking
> scaling, it may be useful to trace memory usage for that base case.
> Even better if you can reduce to one process.  Anyway, I would start by
> running both cases with -log_view and looking at the memory summary.  I
> would then use Massif (the memory profiler/tracer component in Valgrind)
> to obtain stack traces for the large allocations.  Comparing those
> traces should help narrow down which part of the code has significantly
> different memory allocation behavior.  It might also point to the
> unacceptable memory consumption under weak scaling, but it's something
> we should try to fix.
>
> If I had to guess, it may be in intermediate data structures for the
> different PtAP algorithms in GAMG.  The option "-matptap_via scalable"
> may be helpful.
>
> "Smith, Barry F. via petsc-users" <petsc-users at mcs.anl.gov> writes:
>
>>    Myriam,
>>
>>     Sorry we have not been able to resolve this problem with memory scaling yet.
>>
>>     The best tool to determine the change in a code that results in large differences in a program's run is git bisect. Basically you tell git bisect 
>> the git commit of the code that is "good" and the git commit of the code that is "bad" and it gives you additional git commits for you to check your code on  each time telling git if it is "good" or "bad", eventually git bisect tells you exactly the git commit that "broke" the code. No guess work, no endless speculation. 
>>
>>     The draw back is that you have to ./configure && make PETSc for each "test" commit and then compile and run your code for that commit. I can understand if you have to run your code on 10,000 processes to check if it is "good" or "bad" that can be very daunting. But all I can suggest is to find a problem size that is manageable and do the git bisect process (yeah it may take several hours but that beats days of head banging).
>>
>>    Good luck,
>>
>>    Barry
>>
>>
>>> On Mar 5, 2019, at 12:42 PM, Matthew Knepley via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>>
>>> On Tue, Mar 5, 2019 at 11:53 AM Myriam Peyrounette <myriam.peyrounette at idris.fr> wrote:
>>> I used PCView to display the size of the linear system in each level of the MG. You'll find the outputs attached to this mail (zip file) for both the default threshold value and a value of 0.1, and for both 3.6 and 3.10 PETSc versions. 
>>>
>>> For convenience, I summarized the information in a graph, also attached (png file).
>>>
>>>
>>> Great! Can you draw lines for the different runs you did? My interpretation was that memory was increasing
>>> as you did larger runs, and that you though that was coming from GAMG. That means the curves should
>>> be pushed up for larger runs. Do you see that?
>>>
>>>   Thanks,
>>>
>>>     Matt 
>>> As you can see, there are slight differences between the two versions but none is critical, in my opinion. Do you see anything suspicious in the outputs?
>>>
>>> + I can't find the default threshold value. Do you know where I can find it?
>>>
>>> Thanks for the follow-up
>>>
>>> Myriam
>>>
>>>
>>> Le 03/05/19 à 14:06, Matthew Knepley a écrit :
>>>> On Tue, Mar 5, 2019 at 7:14 AM Myriam Peyrounette <myriam.peyrounette at idris.fr> wrote:
>>>> Hi Matt,
>>>>
>>>> I plotted the memory scalings using different threshold values. The two scalings are slightly translated (from -22 to -88 mB) but this gain is neglectable. The 3.6-scaling keeps being robust while the 3.10-scaling deteriorates.
>>>>
>>>> Do you have any other suggestion?
>>>>
>>>> Mark, what is the option she can give to output all the GAMG data?
>>>>
>>>> Also, run using -ksp_view. GAMG will report all the sizes of its grids, so it should be easy to see
>>>> if the coarse grid sizes are increasing, and also what the effect of the threshold value is.
>>>>
>>>>   Thanks,
>>>>
>>>>      Matt 
>>>> Thanks
>>>>
>>>> Myriam 
>>>>
>>>> Le 03/02/19 à 02:27, Matthew Knepley a écrit :
>>>>> On Fri, Mar 1, 2019 at 10:53 AM Myriam Peyrounette via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>>>> Hi,
>>>>>
>>>>> I used to run my code with PETSc 3.6. Since I upgraded the PETSc version
>>>>> to 3.10, this code has a bad memory scaling.
>>>>>
>>>>> To report this issue, I took the PETSc script ex42.c and slightly
>>>>> modified it so that the KSP and PC configurations are the same as in my
>>>>> code. In particular, I use a "personnalised" multi-grid method. The
>>>>> modifications are indicated by the keyword "TopBridge" in the attached
>>>>> scripts.
>>>>>
>>>>> To plot the memory (weak) scaling, I ran four calculations for each
>>>>> script with increasing problem sizes and computations cores:
>>>>>
>>>>> 1. 100,000 elts on 4 cores
>>>>> 2. 1 million elts on 40 cores
>>>>> 3. 10 millions elts on 400 cores
>>>>> 4. 100 millions elts on 4,000 cores
>>>>>
>>>>> The resulting graph is also attached. The scaling using PETSc 3.10
>>>>> clearly deteriorates for large cases, while the one using PETSc 3.6 is
>>>>> robust.
>>>>>
>>>>> After a few tests, I found that the scaling is mostly sensitive to the
>>>>> use of the AMG method for the coarse grid (line 1780 in
>>>>> main_ex42_petsc36.cc). In particular, the performance strongly
>>>>> deteriorates when commenting lines 1777 to 1790 (in main_ex42_petsc36.cc).
>>>>>
>>>>> Do you have any idea of what changed between version 3.6 and version
>>>>> 3.10 that may imply such degradation?
>>>>>
>>>>> I believe the default values for PCGAMG changed between versions. It sounds like the coarsening rate
>>>>> is not great enough, so that these grids are too large. This can be set using:
>>>>>
>>>>>   https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html
>>>>>
>>>>> There is some explanation of this effect on that page. Let us know if setting this does not correct the situation.
>>>>>
>>>>>   Thanks,
>>>>>
>>>>>      Matt
>>>>>  
>>>>> Let me know if you need further information.
>>>>>
>>>>> Best,
>>>>>
>>>>> Myriam Peyrounette
>>>>>
>>>>>
>>>>> -- 
>>>>> Myriam Peyrounette
>>>>> CNRS/IDRIS - HLST
>>>>> --
>>>>>
>>>>>
>>>>>
>>>>> -- 
>>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> https://www.cse.buffalo.edu/~knepley/
>>>> -- 
>>>> Myriam Peyrounette
>>>> CNRS/IDRIS - HLST
>>>> --
>>>>
>>>>
>>>>
>>>> -- 
>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>> -- 
>>> Myriam Peyrounette
>>> CNRS/IDRIS - HLST
>>> --
>>>
>>>
>>>
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/

-- 
Myriam Peyrounette
CNRS/IDRIS - HLST
--

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190308/99d1b272/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mem_consumption.png
Type: image/png
Size: 24499 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190308/99d1b272/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: time_consumption.png
Type: image/png
Size: 19484 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190308/99d1b272/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: main_petsc36.cc
Type: text/x-c++src
Size: 92061 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190308/99d1b272/attachment-0003.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: logView_petsc36.log
Type: text/x-log
Size: 18317 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190308/99d1b272/attachment-0004.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: logView_petsc310.log
Type: text/x-log
Size: 18288 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190308/99d1b272/attachment-0005.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: log_view_mem_distr.pdf
Type: application/pdf
Size: 20293 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190308/99d1b272/attachment-0001.pdf>
-------------- next part --------------
--------------------------------------------------------------------------------
Command:            ./ex42 -mx 48 -my 48 -mz 48 -stokes_ksp_monitor_blocks 0 -model 0 -levels 3
Massif arguments:   (none)
ms_print arguments: massif.out.33309
--------------------------------------------------------------------------------


    GB
1.547^                                                                       :
     |                                       @:::::::::::::@::::::::::::#:::::
     |                      @@@:@::::::::::::@             @            #    :
     |                    :@@@ :@            @             @            #    :
     |                  :::@@@ :@            @             @            #    :
     |   @@@@:::::::::::: :@@@ :@            @             @            #    :
     |   @   :    :    :: :@@@ :@            @             @            #    :
     |   @   :    :    :: :@@@ :@            @             @            #    :
     |   @   :    :    :: :@@@ :@            @             @            #    :
     |   @   :    :    :: :@@@ :@            @             @            #    :
     |   @   :    :    :: :@@@ :@            @             @            #    :
     |   @   :    :    :: :@@@ :@            @             @            #    :
     |@@@@   :    :    :: :@@@ :@            @             @            #    :
     |@  @   :    :    :: :@@@ :@            @             @            #    :
     |@  @   :    :    :: :@@@ :@            @             @            #    :
     |@  @   :    :    :: :@@@ :@            @             @            #    :
     |@  @   :    :    :: :@@@ :@            @             @            #    :
     |@  @   :    :    :: :@@@ :@            @             @            #    :
     |@  @   :    :    :: :@@@ :@            @             @            #    :
     |@  @   :    :    :: :@@@ :@            @             @            #    :
   0 +----------------------------------------------------------------------->Gi
     0                                                                   304.7

Number of snapshots: 82
 Detailed snapshots: [3, 5, 11, 37, 40, 43, 46, 50, 53, 57, 60, 64, 67 (peak), 77]

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
  0              0                0                0             0            0
  1     35,834,612       71,737,416       71,723,119        14,297            0
  2    327,760,951       71,715,056       71,700,775        14,281            0
  3    328,278,660       75,486,720       75,471,765        14,955            0
99.98% (75,471,765B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->88.82% (67,049,927B) 0x4F3E598: PetscMallocAlign (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| ->75.01% (56,623,104B) 0x417587: CellPropertiesCreate(_p_DM*, _p_CellProperties**) (main_petsc36.cc:75)
| | ->75.01% (56,623,104B) 0x404CBE: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:1860)
| |   ->75.01% (56,623,104B) 0x4048AA: main (main_petsc36.cc:2255)
| |     
| ->07.48% (5,647,152B) 0x4FF7CFF: VecCreate_Seq (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->07.48% (5,647,152B) 0x5008681: VecSetType (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->07.48% (5,647,152B) 0x4FF8EA3: VecCreate_Standard (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->07.48% (5,647,152B) 0x5008681: VecSetType (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->03.74% (2,823,576B) 0x53C0070: DMCreateGlobalVector_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       | ->03.74% (2,823,576B) 0x553D255: DMCreateGlobalVector (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   ->03.74% (2,823,576B) 0x53C5B4C: DMDASetUniformCoordinates (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     ->03.74% (2,823,576B) 0x404C6C: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:1854)
| |       |       ->03.74% (2,823,576B) 0x4048AA: main (main_petsc36.cc:2255)
| |       |         
| |       ->03.74% (2,823,576B) 0x53B5985: DMCreateLocalVector_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->03.74% (2,823,576B) 0x5542B05: DMCreateLocalVector (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |           ->03.74% (2,823,576B) 0x5543B56: DMGetCoordinatesLocal (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |             ->03.74% (2,823,576B) 0x404D00: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:1864)
| |               ->03.74% (2,823,576B) 0x4048AA: main (main_petsc36.cc:2255)
| |                 
| ->04.99% (3,764,783B) 0x53EE8A9: DMCreateMatrix_DA_3d_MPIAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->04.99% (3,764,783B) 0x53E4FB5: DMCreateMatrix_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->04.99% (3,764,783B) 0x553AA24: DMCreateMatrix (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->04.99% (3,764,783B) 0x40614B: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2028)
| |       ->04.99% (3,764,783B) 0x4048AA: main (main_petsc36.cc:2255)
| |         
| ->01.25% (941,192B) 0x53A4C31: DMSetUp_DA_3D (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->01.25% (941,192B) 0x53E0AB6: DMSetUp_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->01.25% (941,192B) 0x553972E: DMSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->01.25% (941,192B) 0x53AE57B: DMDACreate3d (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->01.25% (941,192B) in 2 places, all below massif's threshold (01.00%)
| |         
| ->00.10% (73,696B) in 1+ places, all below ms_print's threshold (01.00%)
| 
->11.12% (8,391,248B) 0xA051F04: MPIDI_CH3I_Seg_commit (in /gpfs4l/smplocal/intel/impi/4.1.1.036/intel64/lib/libmpi.so.4.1)
| ->11.12% (8,391,248B) 0xA05E0D2: MPID_nem_init_ckpt (in /gpfs4l/smplocal/intel/impi/4.1.1.036/intel64/lib/libmpi.so.4.1)
|   ->11.12% (8,391,248B) 0xA05FCA5: MPID_nem_init (in /gpfs4l/smplocal/intel/impi/4.1.1.036/intel64/lib/libmpi.so.4.1)
|     ->11.12% (8,391,248B) 0x9F1106E: MPIDI_CH3_Init (in /gpfs4l/smplocal/intel/impi/4.1.1.036/intel64/lib/libmpi.so.4.1)
|       ->11.12% (8,391,248B) 0xA04EBAB: MPID_Init (in /gpfs4l/smplocal/intel/impi/4.1.1.036/intel64/lib/libmpi.so.4.1)
|         ->11.12% (8,391,248B) 0xA02538D: MPIR_Init_thread (in /gpfs4l/smplocal/intel/impi/4.1.1.036/intel64/lib/libmpi.so.4.1)
|           ->11.12% (8,391,248B) 0xA024D98: PMPI_Init_thread (in /gpfs4l/smplocal/intel/impi/4.1.1.036/intel64/lib/libmpi.so.4.1)
|             ->11.12% (8,391,248B) 0x4F1D607: PetscInitialize (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
|               ->11.12% (8,391,248B) 0x404761: main (main_petsc36.cc:2242)
|                 
->00.04% (30,590B) in 1+ places, all below ms_print's threshold (01.00%)

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
  4  1,499,136,193       75,486,696       75,471,765        14,931            0
  5  1,516,554,677      666,470,400      666,454,974        15,426            0
100.00% (666,454,974B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->98.73% (658,033,144B) 0x4F3E598: PetscMallocAlign (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| ->88.11% (587,218,418B) 0x52B1D78: MatSeqAIJSetPreallocation_SeqAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->88.11% (587,218,418B) 0x52B3D19: MatSeqAIJSetPreallocation (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->88.11% (587,218,418B) 0x53EEE0C: DMCreateMatrix_DA_3d_MPIAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->88.11% (587,218,418B) 0x53E4FB5: DMCreateMatrix_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->88.11% (587,218,418B) 0x553AA24: DMCreateMatrix (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->88.11% (587,218,418B) 0x40614B: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2028)
| |           ->88.11% (587,218,418B) 0x4048AA: main (main_petsc36.cc:2255)
| |             
| ->08.50% (56,623,104B) 0x417587: CellPropertiesCreate(_p_DM*, _p_CellProperties**) (main_petsc36.cc:75)
| | ->08.50% (56,623,104B) 0x404CBE: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:1860)
| |   ->08.50% (56,623,104B) 0x4048AA: main (main_petsc36.cc:2255)
| |     
| ->02.13% (14,191,622B) in 69 places, all below massif's threshold (01.00%)
|   
->01.26% (8,391,248B) 0xA051F04: MPIDI_CH3I_Seg_commit (in /gpfs4l/smplocal/intel/impi/4.1.1.036/intel64/lib/libmpi.so.4.1)
| ->01.26% (8,391,248B) 0xA05E0D2: MPID_nem_init_ckpt (in /gpfs4l/smplocal/intel/impi/4.1.1.036/intel64/lib/libmpi.so.4.1)
|   ->01.26% (8,391,248B) 0xA05FCA5: MPID_nem_init (in /gpfs4l/smplocal/intel/impi/4.1.1.036/intel64/lib/libmpi.so.4.1)
|     ->01.26% (8,391,248B) 0x9F1106E: MPIDI_CH3_Init (in /gpfs4l/smplocal/intel/impi/4.1.1.036/intel64/lib/libmpi.so.4.1)
|       ->01.26% (8,391,248B) 0xA04EBAB: MPID_Init (in /gpfs4l/smplocal/intel/impi/4.1.1.036/intel64/lib/libmpi.so.4.1)
|         ->01.26% (8,391,248B) 0xA02538D: MPIR_Init_thread (in /gpfs4l/smplocal/intel/impi/4.1.1.036/intel64/lib/libmpi.so.4.1)
|           ->01.26% (8,391,248B) 0xA024D98: PMPI_Init_thread (in /gpfs4l/smplocal/intel/impi/4.1.1.036/intel64/lib/libmpi.so.4.1)
|             ->01.26% (8,391,248B) 0x4F1D607: PetscInitialize (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
|               ->01.26% (8,391,248B) 0x404761: main (main_petsc36.cc:2242)
|                 
->00.00% (30,582B) in 1+ places, all below ms_print's threshold (01.00%)

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
  6 16,261,614,710      662,705,608      662,690,191        15,417            0
  7 16,273,177,611      664,588,000      664,572,575        15,425            0
  8 16,453,080,876      666,470,408      666,454,963        15,445            0
  9 16,592,550,759      670,240,048      670,224,014        16,034            0
 10 17,763,405,488      670,240,024      670,224,014        16,010            0
 11 17,780,820,810    1,261,223,728    1,261,207,223        16,505            0
100.00% (1,261,207,223B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->99.33% (1,252,785,393B) 0x4F3E598: PetscMallocAlign (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| ->93.12% (1,174,436,836B) 0x52B1D78: MatSeqAIJSetPreallocation_SeqAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->93.12% (1,174,436,836B) 0x52B3D19: MatSeqAIJSetPreallocation (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->93.12% (1,174,436,836B) 0x53EEE0C: DMCreateMatrix_DA_3d_MPIAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->93.12% (1,174,436,836B) 0x53E4FB5: DMCreateMatrix_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->93.12% (1,174,436,836B) 0x553AA24: DMCreateMatrix (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->46.56% (587,218,418B) 0x40614B: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2028)
| |         | ->46.56% (587,218,418B) 0x4048AA: main (main_petsc36.cc:2255)
| |         |   
| |         ->46.56% (587,218,418B) 0x40616C: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2029)
| |           ->46.56% (587,218,418B) 0x4048AA: main (main_petsc36.cc:2255)
| |             
| ->04.49% (56,623,104B) 0x417587: CellPropertiesCreate(_p_DM*, _p_CellProperties**) (main_petsc36.cc:75)
| | ->04.49% (56,623,104B) 0x404CBE: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:1860)
| |   ->04.49% (56,623,104B) 0x4048AA: main (main_petsc36.cc:2255)
| |     
| ->01.72% (21,725,453B) in 72 places, all below massif's threshold (01.00%)
|   
->00.67% (8,421,830B) in 1+ places, all below ms_print's threshold (01.00%)

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 12 32,525,877,843    1,257,458,936    1,257,442,440        16,496            0
 13 32,717,337,805    1,261,223,736    1,261,207,212        16,524            0
 14 32,856,396,640    1,261,221,968    1,261,205,453        16,515            0
 15 32,930,640,810    1,268,778,168    1,268,761,389        16,779            0
 16 58,569,488,720    1,268,755,808    1,268,739,045        16,763            0
 17 80,399,203,865    1,268,755,808    1,268,739,045        16,763            0
 18 80,399,942,978    1,272,567,448    1,272,550,529        16,919            0
 19 81,763,811,939    1,272,545,088    1,272,528,185        16,903            0
 20 81,893,298,331    1,305,332,968    1,305,305,169        27,799            0
 21 85,163,552,026    1,356,110,968    1,356,077,261        33,707            0
 22 85,215,698,411    1,376,145,504    1,376,117,065        28,439            0
 23 93,174,834,293    1,375,145,464    1,375,117,035        28,429            0
 24 93,176,062,194    1,375,962,496    1,375,933,984        28,512            0
 25 93,593,897,813    1,382,701,952    1,382,673,404        28,548            0
 26 93,600,805,445    1,385,254,232    1,385,224,814        29,418            0
 27 94,611,634,258    1,385,051,600    1,385,022,202        29,398            0
 28 94,632,051,286    1,404,694,096    1,404,662,417        31,679            0
 29 94,883,287,158    1,404,702,920    1,404,671,205        31,715            0
 30 95,007,514,066    1,407,559,152    1,407,523,721        35,431            0
 31 95,167,535,043    1,405,947,776    1,405,915,815        31,961            0
 32 95,349,461,781    1,413,587,144    1,413,552,928        34,216            0
 33 95,639,044,739    1,405,736,760    1,405,704,162        32,598            0
 34 95,674,406,759    1,458,922,280    1,458,880,871        41,409            0
 35 96,600,728,596    1,458,922,544    1,458,881,127        41,417            0
 36 97,102,450,577    1,458,922,632    1,458,881,207        41,425            0
 37 97,105,663,401    1,492,821,480    1,492,779,339        42,141            0
100.00% (1,492,779,339B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->99.43% (1,484,354,449B) 0x4F3E598: PetscMallocAlign (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| ->79.06% (1,180,246,040B) 0x52B1D78: MatSeqAIJSetPreallocation_SeqAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->79.06% (1,180,246,040B) 0x52B3D19: MatSeqAIJSetPreallocation (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->78.67% (1,174,436,836B) 0x53EEE0C: DMCreateMatrix_DA_3d_MPIAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   | ->78.67% (1,174,436,836B) 0x53E4FB5: DMCreateMatrix_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   ->78.67% (1,174,436,836B) 0x553AA24: DMCreateMatrix (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |     ->39.34% (587,218,418B) 0x40614B: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2028)
| |   |     | ->39.34% (587,218,418B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |     |   
| |   |     ->39.34% (587,218,418B) 0x40616C: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2029)
| |   |       ->39.34% (587,218,418B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |         
| |   ->00.39% (5,809,204B) in 1+ places, all below ms_print's threshold (01.00%)
| |   
| ->07.18% (107,177,936B) 0x4FF7CFF: VecCreate_Seq (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->07.18% (107,177,936B) 0x5008681: VecSetType (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->04.78% (71,412,640B) 0x4FF1C5F: VecDuplicate_Seq (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   | ->04.78% (71,412,640B) 0x5007F69: VecDuplicate (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   ->04.71% (70,337,984B) 0x500832F: VecDuplicateVecs_Default (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   | ->04.71% (70,337,984B) 0x50050D5: VecDuplicateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   ->04.71% (70,337,984B) 0x570A400: KSPCreateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     ->02.27% (33,882,912B) 0x56AF984: KSPGMRESGetNewVectors (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     | ->02.27% (33,882,912B) 0x56AEFB0: KSPGMRESCycle (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   ->02.27% (33,882,912B) 0x56AEC7A: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |     ->02.27% (33,882,912B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       ->02.27% (33,882,912B) 0x56F6F73: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       | ->02.27% (33,882,912B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       |   ->02.27% (33,882,912B) 0x56818FA: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       |     ->02.27% (33,882,912B) 0x5682B76: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       |       ->02.27% (33,882,912B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       |         ->02.27% (33,882,912B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       |           ->02.27% (33,882,912B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       |             ->02.27% (33,882,912B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       |               ->02.27% (33,882,912B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |   |   |     |       |                 ->02.27% (33,882,912B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |   |     |       |                   
| |   |   |     |       ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   |     |       
| |   |   |     ->01.26% (18,825,120B) 0x56AE79B: KSPSetUp_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     | ->01.26% (18,825,120B) 0x5701286: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   ->01.26% (18,823,840B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | ->01.26% (18,823,840B) 0x56F6F73: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | | ->01.26% (18,823,840B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | |   ->01.26% (18,823,840B) 0x56818FA: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | |     ->01.26% (18,823,840B) 0x5682B76: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | |       ->01.26% (18,823,840B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | |         ->01.26% (18,823,840B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | |           ->01.26% (18,823,840B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | |             ->01.26% (18,823,840B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | |               ->01.26% (18,823,840B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |   |   |     |   | |                 ->01.26% (18,823,840B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |   |     |   | |                   
| |   |   |     |   | ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   |     |   | 
| |   |   |     |   ->00.00% (1,280B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   |     |   
| |   |   |     ->01.18% (17,629,952B) in 3 places, all below massif's threshold (01.00%)
| |   |   |       
| |   |   ->00.07% (1,074,656B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   
| |   ->02.40% (35,765,296B) 0x4FF8EA3: VecCreate_Standard (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->02.40% (35,765,296B) 0x5008681: VecSetType (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->02.14% (32,000,528B) 0x53C0070: DMCreateGlobalVector_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       | ->02.14% (32,000,528B) 0x553D255: DMCreateGlobalVector (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   ->01.26% (18,823,840B) 0x53BFE5F: VecDuplicate_MPI_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   | ->01.26% (18,823,840B) 0x5007F69: VecDuplicate (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |   ->01.26% (18,823,840B) 0x500832F: VecDuplicateVecs_Default (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |     ->01.26% (18,823,840B) 0x50050D5: VecDuplicateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |       ->01.26% (18,823,840B) 0x570A400: KSPCreateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |         ->01.26% (18,823,840B) 0x56AE79B: KSPSetUp_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |           ->01.26% (18,823,840B) 0x5701286: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |             ->01.26% (18,823,840B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |               ->01.26% (18,823,840B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |       |   |                 ->01.26% (18,823,840B) 0x4048AA: main (main_petsc36.cc:2255)
| |       |   |                   
| |       |   ->00.88% (13,176,688B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |   
| |       ->00.25% (3,764,768B) in 1+ places, all below ms_print's threshold (01.00%)
| |       
| ->03.79% (56,623,104B) 0x417587: CellPropertiesCreate(_p_DM*, _p_CellProperties**) (main_petsc36.cc:75)
| | ->03.79% (56,623,104B) 0x404CBE: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:1860)
| |   ->03.79% (56,623,104B) 0x4048AA: main (main_petsc36.cc:2255)
| |     
| ->03.77% (56,277,776B) 0x51F256F: MatPtAPSymbolic_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->03.77% (56,277,776B) 0x51F192E: MatPtAP_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->03.77% (56,277,776B) 0x509B68C: MatPtAP (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->03.77% (56,277,776B) 0x568798E: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->03.77% (56,277,776B) 0x5693D68: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->03.77% (56,277,776B) 0x57014F7: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |           ->03.77% (56,277,776B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |             ->03.77% (56,277,776B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |               ->03.77% (56,277,776B) 0x4048AA: main (main_petsc36.cc:2255)
| |                 
| ->02.27% (33,882,942B) 0x5307655: MatSOR_SeqAIJ_Inode (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->02.27% (33,882,942B) 0x508F2F7: MatSOR (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->02.27% (33,882,942B) 0x5603367: PCApply_SOR (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->02.27% (33,882,942B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->02.27% (33,882,942B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->02.27% (33,882,942B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |           ->02.27% (33,882,942B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |             ->02.27% (33,882,942B) 0x56F6F73: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |               ->02.27% (33,882,942B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                 ->02.27% (33,882,942B) 0x56818FA: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   ->02.27% (33,882,942B) 0x5682B76: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                     ->02.27% (33,882,942B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                       ->02.27% (33,882,942B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                         ->02.27% (33,882,942B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                           ->02.27% (33,882,942B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                             ->02.27% (33,882,942B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |                               ->02.27% (33,882,942B) 0x4048AA: main (main_petsc36.cc:2255)
| |                                 
| ->01.88% (28,138,888B) 0x51F2440: MatPtAPSymbolic_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->01.88% (28,138,888B) 0x51F192E: MatPtAP_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->01.88% (28,138,888B) 0x509B68C: MatPtAP (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->01.88% (28,138,888B) 0x568798E: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->01.88% (28,138,888B) 0x5693D68: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->01.88% (28,138,888B) 0x57014F7: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |           ->01.88% (28,138,888B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |             ->01.88% (28,138,888B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |               ->01.88% (28,138,888B) 0x4048AA: main (main_petsc36.cc:2255)
| |                 
| ->01.47% (22,007,763B) in 162 places, all below massif's threshold (01.00%)
|   
->00.56% (8,424,890B) in 1+ places, all below ms_print's threshold (01.00%)

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 38 101,238,965,269    1,496,588,096    1,496,545,902        42,194            0
 39 103,718,453,406    1,496,637,272    1,496,595,059        42,213            0
 40 103,903,351,997    1,508,176,464    1,508,131,617        44,847            0
100.00% (1,508,131,617B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->99.44% (1,499,673,831B) 0x4F3E598: PetscMallocAlign (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| ->78.26% (1,180,246,040B) 0x52B1D78: MatSeqAIJSetPreallocation_SeqAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->78.26% (1,180,246,040B) 0x52B3D19: MatSeqAIJSetPreallocation (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->77.87% (1,174,436,836B) 0x53EEE0C: DMCreateMatrix_DA_3d_MPIAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   | ->77.87% (1,174,436,836B) 0x53E4FB5: DMCreateMatrix_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   ->77.87% (1,174,436,836B) 0x553AA24: DMCreateMatrix (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |     ->38.94% (587,218,418B) 0x40614B: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2028)
| |   |     | ->38.94% (587,218,418B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |     |   
| |   |     ->38.94% (587,218,418B) 0x40616C: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2029)
| |   |       ->38.94% (587,218,418B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |         
| |   ->00.39% (5,809,204B) in 1+ places, all below ms_print's threshold (01.00%)
| |   
| ->07.82% (117,942,704B) 0x4FF7CFF: VecCreate_Seq (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->07.82% (117,942,704B) 0x5008681: VecSetType (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->05.45% (82,177,408B) 0x4FF1C5F: VecDuplicate_Seq (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   | ->05.45% (82,177,408B) 0x5007F69: VecDuplicate (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   ->05.38% (81,102,752B) 0x500832F: VecDuplicateVecs_Default (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   | ->05.38% (81,102,752B) 0x50050D5: VecDuplicateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   ->05.38% (81,102,752B) 0x570A400: KSPCreateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     ->02.79% (42,147,680B) 0x56AF984: KSPGMRESGetNewVectors (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     | ->02.79% (42,147,680B) 0x56AEFB0: KSPGMRESCycle (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   ->02.79% (42,147,680B) 0x56AEC7A: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |     ->02.79% (42,147,680B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       ->02.79% (42,147,680B) 0x56F6F73: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       | ->02.79% (42,147,680B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       |   ->02.79% (42,147,680B) 0x56818FA: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       |     ->02.50% (37,647,680B) 0x5682B76: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       |     | ->02.50% (37,647,680B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       |     |   ->02.50% (37,647,680B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       |     |     ->02.50% (37,647,680B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       |     |       ->02.50% (37,647,680B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       |     |         ->02.50% (37,647,680B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |   |   |     |       |     |           ->02.50% (37,647,680B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |   |     |       |     |             
| |   |   |     |       |     ->00.30% (4,500,000B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   |     |       |     
| |   |   |     |       ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   |     |       
| |   |   |     ->01.41% (21,325,120B) 0x56AE79B: KSPSetUp_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     | ->01.41% (21,325,120B) 0x5701286: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   ->01.41% (21,323,840B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | ->01.41% (21,323,840B) 0x56F6F73: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | | ->01.41% (21,323,840B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | |   ->01.41% (21,323,840B) 0x56818FA: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | |     ->01.25% (18,823,840B) 0x5682B76: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | |     | ->01.25% (18,823,840B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | |     |   ->01.25% (18,823,840B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | |     |     ->01.25% (18,823,840B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | |     |       ->01.25% (18,823,840B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | |     |         ->01.25% (18,823,840B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |   |   |     |   | |     |           ->01.25% (18,823,840B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |   |     |   | |     |             
| |   |   |     |   | |     ->00.17% (2,500,000B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   |     |   | |     
| |   |   |     |   | ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   |     |   | 
| |   |   |     |   ->00.00% (1,280B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   |     |   
| |   |   |     ->01.17% (17,629,952B) in 3 places, all below massif's threshold (01.00%)
| |   |   |       
| |   |   ->00.07% (1,074,656B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   
| |   ->02.37% (35,765,296B) 0x4FF8EA3: VecCreate_Standard (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->02.37% (35,765,296B) 0x5008681: VecSetType (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->02.12% (32,000,528B) 0x53C0070: DMCreateGlobalVector_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       | ->02.12% (32,000,528B) 0x553D255: DMCreateGlobalVector (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   ->01.25% (18,823,840B) 0x53BFE5F: VecDuplicate_MPI_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   | ->01.25% (18,823,840B) 0x5007F69: VecDuplicate (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |   ->01.25% (18,823,840B) 0x500832F: VecDuplicateVecs_Default (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |     ->01.25% (18,823,840B) 0x50050D5: VecDuplicateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |       ->01.25% (18,823,840B) 0x570A400: KSPCreateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |         ->01.25% (18,823,840B) 0x56AE79B: KSPSetUp_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |           ->01.25% (18,823,840B) 0x5701286: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |             ->01.25% (18,823,840B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |               ->01.25% (18,823,840B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |       |   |                 ->01.25% (18,823,840B) 0x4048AA: main (main_petsc36.cc:2255)
| |       |   |                   
| |       |   ->00.87% (13,176,688B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |   
| |       ->00.25% (3,764,768B) in 1+ places, all below ms_print's threshold (01.00%)
| |       
| ->03.75% (56,623,104B) 0x417587: CellPropertiesCreate(_p_DM*, _p_CellProperties**) (main_petsc36.cc:75)
| | ->03.75% (56,623,104B) 0x404CBE: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:1860)
| |   ->03.75% (56,623,104B) 0x4048AA: main (main_petsc36.cc:2255)
| |     
| ->03.73% (56,277,776B) 0x51F256F: MatPtAPSymbolic_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->03.73% (56,277,776B) 0x51F192E: MatPtAP_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->03.73% (56,277,776B) 0x509B68C: MatPtAP (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->03.73% (56,277,776B) 0x568798E: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->03.73% (56,277,776B) 0x5693D68: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->03.73% (56,277,776B) 0x57014F7: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |           ->03.73% (56,277,776B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |             ->03.73% (56,277,776B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |               ->03.73% (56,277,776B) 0x4048AA: main (main_petsc36.cc:2255)
| |                 
| ->02.54% (38,382,972B) 0x5307655: MatSOR_SeqAIJ_Inode (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->02.54% (38,382,972B) 0x508F2F7: MatSOR (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->02.54% (38,382,972B) 0x5603367: PCApply_SOR (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->02.54% (38,382,972B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->02.54% (38,382,972B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->02.54% (38,382,972B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |           ->02.54% (38,382,972B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |             ->02.54% (38,382,972B) 0x56F6F73: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |               ->02.54% (38,382,972B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                 ->02.54% (38,382,972B) 0x56818FA: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   ->02.25% (33,882,942B) 0x5682B76: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   | ->02.25% (33,882,942B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   ->02.25% (33,882,942B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |     ->02.25% (33,882,942B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |       ->02.25% (33,882,942B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |         ->02.25% (33,882,942B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |                   |           ->02.25% (33,882,942B) 0x4048AA: main (main_petsc36.cc:2255)
| |                   |             
| |                   ->00.30% (4,500,030B) in 1+ places, all below ms_print's threshold (01.00%)
| |                   
| ->01.87% (28,138,888B) 0x51F2440: MatPtAPSymbolic_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->01.87% (28,138,888B) 0x51F192E: MatPtAP_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->01.87% (28,138,888B) 0x509B68C: MatPtAP (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->01.87% (28,138,888B) 0x568798E: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->01.87% (28,138,888B) 0x5693D68: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->01.87% (28,138,888B) 0x57014F7: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |           ->01.87% (28,138,888B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |             ->01.87% (28,138,888B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |               ->01.87% (28,138,888B) 0x4048AA: main (main_petsc36.cc:2255)
| |                 
| ->01.46% (22,062,347B) in 164 places, all below massif's threshold (01.00%)
|   
->00.56% (8,457,786B) in 1+ places, all below ms_print's threshold (01.00%)

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 41 104,434,019,902    1,508,678,312    1,508,633,412        44,900            0
 42 104,751,975,500    1,508,694,584    1,508,649,673        44,911            0
 43 104,877,346,163    1,510,547,792    1,510,499,343        48,449            0
100.00% (1,510,499,343B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->99.44% (1,502,041,557B) 0x4F3E598: PetscMallocAlign (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| ->78.13% (1,180,246,040B) 0x52B1D78: MatSeqAIJSetPreallocation_SeqAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->78.13% (1,180,246,040B) 0x52B3D19: MatSeqAIJSetPreallocation (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->77.75% (1,174,436,836B) 0x53EEE0C: DMCreateMatrix_DA_3d_MPIAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   | ->77.75% (1,174,436,836B) 0x53E4FB5: DMCreateMatrix_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   ->77.75% (1,174,436,836B) 0x553AA24: DMCreateMatrix (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |     ->38.87% (587,218,418B) 0x40614B: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2028)
| |   |     | ->38.87% (587,218,418B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |     |   
| |   |     ->38.87% (587,218,418B) 0x40616C: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2029)
| |   |       ->38.87% (587,218,418B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |         
| |   ->00.38% (5,809,204B) in 1+ places, all below ms_print's threshold (01.00%)
| |   
| ->07.91% (119,527,984B) 0x4FF7CFF: VecCreate_Seq (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->07.91% (119,527,984B) 0x5008681: VecSetType (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->05.55% (83,762,688B) 0x4FF1C5F: VecDuplicate_Seq (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   | ->05.55% (83,762,688B) 0x5007F69: VecDuplicate (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   ->05.47% (82,688,032B) 0x500832F: VecDuplicateVecs_Default (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   | ->05.47% (82,688,032B) 0x50050D5: VecDuplicateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   ->05.47% (82,688,032B) 0x570A400: KSPCreateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     ->02.87% (43,371,200B) 0x56AF984: KSPGMRESGetNewVectors (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     | ->02.87% (43,371,200B) 0x56AEFB0: KSPGMRESCycle (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   ->02.87% (43,371,200B) 0x56AEC7A: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |     ->02.87% (43,371,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       ->02.87% (43,371,200B) 0x56F6F73: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       | ->02.87% (43,371,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       |   ->02.87% (43,371,200B) 0x56818FA: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       |     ->02.54% (38,350,720B) 0x5682B76: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       |     | ->02.54% (38,350,720B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       |     |   ->02.49% (37,647,680B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       |     |   | ->02.49% (37,647,680B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       |     |   |   ->02.49% (37,647,680B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       |     |   |     ->02.49% (37,647,680B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |   |   |     |       |     |   |       ->02.49% (37,647,680B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |   |     |       |     |   |         
| |   |   |     |       |     |   ->00.05% (703,040B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   |     |       |     |   
| |   |   |     |       |     ->00.33% (5,020,480B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   |     |       |     
| |   |   |     |       ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   |     |       
| |   |   |     ->01.44% (21,686,880B) 0x56AE79B: KSPSetUp_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     | ->01.44% (21,686,880B) 0x5701286: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   ->01.44% (21,685,600B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | ->01.44% (21,685,600B) 0x56F6F73: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | | ->01.44% (21,685,600B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | |   ->01.44% (21,685,600B) 0x56818FA: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | |     ->01.27% (19,175,360B) 0x5682B76: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | |     | ->01.27% (19,175,360B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | |     |   ->01.25% (18,823,840B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | |     |   | ->01.25% (18,823,840B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | |     |   |   ->01.25% (18,823,840B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | |     |   |     ->01.25% (18,823,840B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |   |   |     |   | |     |   |       ->01.25% (18,823,840B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |   |     |   | |     |   |         
| |   |   |     |   | |     |   ->00.02% (351,520B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   |     |   | |     |   
| |   |   |     |   | |     ->00.17% (2,510,240B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   |     |   | |     
| |   |   |     |   | ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   |     |   | 
| |   |   |     |   ->00.00% (1,280B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   |     |   
| |   |   |     ->01.17% (17,629,952B) in 3 places, all below massif's threshold (01.00%)
| |   |   |       
| |   |   ->00.07% (1,074,656B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   
| |   ->02.37% (35,765,296B) 0x4FF8EA3: VecCreate_Standard (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->02.37% (35,765,296B) 0x5008681: VecSetType (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->02.12% (32,000,528B) 0x53C0070: DMCreateGlobalVector_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       | ->02.12% (32,000,528B) 0x553D255: DMCreateGlobalVector (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   ->01.25% (18,823,840B) 0x53BFE5F: VecDuplicate_MPI_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   | ->01.25% (18,823,840B) 0x5007F69: VecDuplicate (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |   ->01.25% (18,823,840B) 0x500832F: VecDuplicateVecs_Default (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |     ->01.25% (18,823,840B) 0x50050D5: VecDuplicateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |       ->01.25% (18,823,840B) 0x570A400: KSPCreateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |         ->01.25% (18,823,840B) 0x56AE79B: KSPSetUp_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |           ->01.25% (18,823,840B) 0x5701286: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |             ->01.25% (18,823,840B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |               ->01.25% (18,823,840B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |       |   |                 ->01.25% (18,823,840B) 0x4048AA: main (main_petsc36.cc:2255)
| |       |   |                   
| |       |   ->00.87% (13,176,688B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |   
| |       ->00.25% (3,764,768B) in 1+ places, all below ms_print's threshold (01.00%)
| |       
| ->03.75% (56,623,104B) 0x417587: CellPropertiesCreate(_p_DM*, _p_CellProperties**) (main_petsc36.cc:75)
| | ->03.75% (56,623,104B) 0x404CBE: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:1860)
| |   ->03.75% (56,623,104B) 0x4048AA: main (main_petsc36.cc:2255)
| |     
| ->03.73% (56,277,776B) 0x51F256F: MatPtAPSymbolic_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->03.73% (56,277,776B) 0x51F192E: MatPtAP_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->03.73% (56,277,776B) 0x509B68C: MatPtAP (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->03.73% (56,277,776B) 0x568798E: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->03.73% (56,277,776B) 0x5693D68: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->03.73% (56,277,776B) 0x57014F7: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |           ->03.73% (56,277,776B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |             ->03.73% (56,277,776B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |               ->03.73% (56,277,776B) 0x4048AA: main (main_petsc36.cc:2255)
| |                 
| ->02.58% (39,034,200B) 0x5307655: MatSOR_SeqAIJ_Inode (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->02.58% (39,034,200B) 0x508F2F7: MatSOR (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->02.58% (39,034,200B) 0x5603367: PCApply_SOR (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->02.58% (39,034,200B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->02.58% (39,034,200B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->02.58% (39,034,200B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |           ->02.58% (39,034,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |             ->02.58% (39,034,200B) 0x56F6F73: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |               ->02.58% (39,034,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                 ->02.58% (39,034,200B) 0x56818FA: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   ->02.28% (34,515,708B) 0x5682B76: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   | ->02.28% (34,515,708B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   ->02.24% (33,882,942B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   | ->02.24% (33,882,942B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   |   ->02.24% (33,882,942B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   |     ->02.24% (33,882,942B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |                   |   |       ->02.24% (33,882,942B) 0x4048AA: main (main_petsc36.cc:2255)
| |                   |   |         
| |                   |   ->00.04% (632,766B) in 1+ places, all below ms_print's threshold (01.00%)
| |                   |   
| |                   ->00.30% (4,518,492B) in 1+ places, all below ms_print's threshold (01.00%)
| |                   
| ->01.86% (28,138,888B) 0x51F2440: MatPtAPSymbolic_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->01.86% (28,138,888B) 0x51F192E: MatPtAP_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->01.86% (28,138,888B) 0x509B68C: MatPtAP (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->01.86% (28,138,888B) 0x568798E: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->01.86% (28,138,888B) 0x5693D68: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->01.86% (28,138,888B) 0x57014F7: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |           ->01.86% (28,138,888B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |             ->01.86% (28,138,888B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |               ->01.86% (28,138,888B) 0x4048AA: main (main_petsc36.cc:2255)
| |                 
| ->01.47% (22,193,565B) in 189 places, all below massif's threshold (01.00%)
|   
->00.56% (8,457,786B) in 1+ places, all below ms_print's threshold (01.00%)

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 44 114,000,025,261    1,510,549,016    1,510,500,559        48,457            0
 45 120,086,777,602    1,510,549,104    1,510,500,639        48,465            0
 46 120,089,651,044    1,544,450,152    1,544,400,843        49,309            0
100.00% (1,544,400,843B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->99.45% (1,535,943,053B) 0x4F3E598: PetscMallocAlign (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| ->76.42% (1,180,246,040B) 0x52B1D78: MatSeqAIJSetPreallocation_SeqAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->76.42% (1,180,246,040B) 0x52B3D19: MatSeqAIJSetPreallocation (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->76.04% (1,174,436,836B) 0x53EEE0C: DMCreateMatrix_DA_3d_MPIAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   | ->76.04% (1,174,436,836B) 0x53E4FB5: DMCreateMatrix_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   ->76.04% (1,174,436,836B) 0x553AA24: DMCreateMatrix (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |     ->38.02% (587,218,418B) 0x40614B: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2028)
| |   |     | ->38.02% (587,218,418B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |     |   
| |   |     ->38.02% (587,218,418B) 0x40616C: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2029)
| |   |       ->38.02% (587,218,418B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |         
| |   ->00.38% (5,809,204B) in 1+ places, all below ms_print's threshold (01.00%)
| |   
| ->09.93% (153,410,896B) 0x4FF7CFF: VecCreate_Seq (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->09.93% (153,410,896B) 0x5008681: VecSetType (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->05.42% (83,762,688B) 0x4FF1C5F: VecDuplicate_Seq (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   | ->05.42% (83,762,688B) 0x5007F69: VecDuplicate (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   ->05.35% (82,688,032B) 0x500832F: VecDuplicateVecs_Default (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   | ->05.35% (82,688,032B) 0x50050D5: VecDuplicateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   ->05.35% (82,688,032B) 0x570A400: KSPCreateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     ->02.81% (43,371,200B) 0x56AF984: KSPGMRESGetNewVectors (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     | ->02.81% (43,371,200B) 0x56AEFB0: KSPGMRESCycle (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   ->02.81% (43,371,200B) 0x56AEC7A: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |     ->02.81% (43,371,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       ->02.81% (43,371,200B) 0x56F6F73: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       | ->02.81% (43,371,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       |   ->02.81% (43,371,200B) 0x56818FA: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       |     ->02.48% (38,350,720B) 0x5682B76: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       |     | ->02.48% (38,350,720B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       |     |   ->02.44% (37,647,680B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       |     |   | ->02.44% (37,647,680B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       |     |   |   ->02.44% (37,647,680B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |       |     |   |     ->02.44% (37,647,680B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |   |   |     |       |     |   |       ->02.44% (37,647,680B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |   |     |       |     |   |         
| |   |   |     |       |     |   ->00.05% (703,040B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   |     |       |     |   
| |   |   |     |       |     ->00.33% (5,020,480B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   |     |       |     
| |   |   |     |       ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   |     |       
| |   |   |     ->01.40% (21,686,880B) 0x56AE79B: KSPSetUp_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     | ->01.40% (21,686,880B) 0x5701286: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   ->01.40% (21,685,600B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | ->01.40% (21,685,600B) 0x56F6F73: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | | ->01.40% (21,685,600B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | |   ->01.40% (21,685,600B) 0x56818FA: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | |     ->01.24% (19,175,360B) 0x5682B76: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | |     | ->01.24% (19,175,360B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | |     |   ->01.22% (18,823,840B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | |     |   | ->01.22% (18,823,840B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | |     |   |   ->01.22% (18,823,840B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |     |   | |     |   |     ->01.22% (18,823,840B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |   |   |     |   | |     |   |       ->01.22% (18,823,840B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |   |     |   | |     |   |         
| |   |   |     |   | |     |   ->00.02% (351,520B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   |     |   | |     |   
| |   |   |     |   | |     ->00.16% (2,510,240B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   |     |   | |     
| |   |   |     |   | ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   |     |   | 
| |   |   |     |   ->00.00% (1,280B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   |     |   
| |   |   |     ->01.14% (17,629,952B) in 3 places, all below massif's threshold (01.00%)
| |   |   |       
| |   |   ->00.07% (1,074,656B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   
| |   ->04.51% (69,648,208B) 0x4FF8EA3: VecCreate_Standard (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->04.51% (69,648,208B) 0x5008681: VecSetType (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->04.27% (65,883,440B) 0x53C0070: DMCreateGlobalVector_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       | ->04.27% (65,883,440B) 0x553D255: DMCreateGlobalVector (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   ->03.41% (52,706,752B) 0x53BFE5F: VecDuplicate_MPI_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   | ->03.41% (52,706,752B) 0x5007F69: VecDuplicate (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |   ->03.41% (52,706,752B) 0x500832F: VecDuplicateVecs_Default (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |     ->03.41% (52,706,752B) 0x50050D5: VecDuplicateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |       ->03.41% (52,706,752B) 0x570A400: KSPCreateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |         ->02.19% (33,882,912B) 0x56AF984: KSPGMRESGetNewVectors (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |         | ->02.19% (33,882,912B) 0x56AEFB0: KSPGMRESCycle (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |         |   ->02.19% (33,882,912B) 0x56AEC7A: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |         |     ->02.19% (33,882,912B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |         |       ->02.19% (33,882,912B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |       |   |         |         ->02.19% (33,882,912B) 0x4048AA: main (main_petsc36.cc:2255)
| |       |   |         |           
| |       |   |         ->01.22% (18,823,840B) 0x56AE79B: KSPSetUp_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |           ->01.22% (18,823,840B) 0x5701286: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |             ->01.22% (18,823,840B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   |               ->01.22% (18,823,840B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |       |   |                 ->01.22% (18,823,840B) 0x4048AA: main (main_petsc36.cc:2255)
| |       |   |                   
| |       |   ->00.85% (13,176,688B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |   
| |       ->00.24% (3,764,768B) in 1+ places, all below ms_print's threshold (01.00%)
| |       
| ->03.67% (56,623,104B) 0x417587: CellPropertiesCreate(_p_DM*, _p_CellProperties**) (main_petsc36.cc:75)
| | ->03.67% (56,623,104B) 0x404CBE: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:1860)
| |   ->03.67% (56,623,104B) 0x4048AA: main (main_petsc36.cc:2255)
| |     
| ->03.64% (56,277,776B) 0x51F256F: MatPtAPSymbolic_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->03.64% (56,277,776B) 0x51F192E: MatPtAP_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->03.64% (56,277,776B) 0x509B68C: MatPtAP (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->03.64% (56,277,776B) 0x568798E: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->03.64% (56,277,776B) 0x5693D68: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->03.64% (56,277,776B) 0x57014F7: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |           ->03.64% (56,277,776B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |             ->03.64% (56,277,776B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |               ->03.64% (56,277,776B) 0x4048AA: main (main_petsc36.cc:2255)
| |                 
| ->02.53% (39,034,200B) 0x5307655: MatSOR_SeqAIJ_Inode (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->02.53% (39,034,200B) 0x508F2F7: MatSOR (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->02.53% (39,034,200B) 0x5603367: PCApply_SOR (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->02.53% (39,034,200B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->02.53% (39,034,200B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->02.53% (39,034,200B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |           ->02.53% (39,034,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |             ->02.53% (39,034,200B) 0x56F6F73: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |               ->02.53% (39,034,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                 ->02.53% (39,034,200B) 0x56818FA: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   ->02.23% (34,515,708B) 0x5682B76: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   | ->02.23% (34,515,708B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   ->02.19% (33,882,942B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   | ->02.19% (33,882,942B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   |   ->02.19% (33,882,942B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   |     ->02.19% (33,882,942B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |                   |   |       ->02.19% (33,882,942B) 0x4048AA: main (main_petsc36.cc:2255)
| |                   |   |         
| |                   |   ->00.04% (632,766B) in 1+ places, all below ms_print's threshold (01.00%)
| |                   |   
| |                   ->00.29% (4,518,492B) in 1+ places, all below ms_print's threshold (01.00%)
| |                   
| ->01.82% (28,138,888B) 0x51F2440: MatPtAPSymbolic_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->01.82% (28,138,888B) 0x51F192E: MatPtAP_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->01.82% (28,138,888B) 0x509B68C: MatPtAP (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->01.82% (28,138,888B) 0x568798E: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->01.82% (28,138,888B) 0x5693D68: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->01.82% (28,138,888B) 0x57014F7: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |           ->01.82% (28,138,888B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |             ->01.82% (28,138,888B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |               ->01.82% (28,138,888B) 0x4048AA: main (main_petsc36.cc:2255)
| |                 
| ->01.44% (22,212,149B) in 189 places, all below massif's threshold (01.00%)
|   
->00.55% (8,457,790B) in 1+ places, all below ms_print's threshold (01.00%)

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 47 120,090,364,653    1,548,217,088    1,548,167,663        49,425            0
 48 180,882,674,722    1,548,217,448    1,548,168,039        49,409            0
 49 180,883,391,417    1,555,750,808    1,555,701,207        49,601            0
 50 180,883,754,195    1,563,284,344    1,563,234,583        49,761            0
100.00% (1,563,234,583B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->99.46% (1,554,776,793B) 0x4F3E598: PetscMallocAlign (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| ->75.50% (1,180,246,040B) 0x52B1D78: MatSeqAIJSetPreallocation_SeqAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->75.50% (1,180,246,040B) 0x52B3D19: MatSeqAIJSetPreallocation (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->75.13% (1,174,436,836B) 0x53EEE0C: DMCreateMatrix_DA_3d_MPIAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   | ->75.13% (1,174,436,836B) 0x53E4FB5: DMCreateMatrix_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   ->75.13% (1,174,436,836B) 0x553AA24: DMCreateMatrix (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |     ->37.56% (587,218,418B) 0x40614B: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2028)
| |   |     | ->37.56% (587,218,418B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |     |   
| |   |     ->37.56% (587,218,418B) 0x40616C: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2029)
| |   |       ->37.56% (587,218,418B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |         
| |   ->00.37% (5,809,204B) in 1+ places, all below ms_print's threshold (01.00%)
| |   
| ->11.02% (172,234,736B) 0x4FF7CFF: VecCreate_Seq (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->11.02% (172,234,736B) 0x5008681: VecSetType (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->05.66% (88,472,048B) 0x4FF8EA3: VecCreate_Standard (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   | ->05.66% (88,472,048B) 0x5008681: VecSetType (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   ->05.42% (84,707,280B) 0x53C0070: DMCreateGlobalVector_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   | ->05.42% (84,707,280B) 0x553D255: DMCreateGlobalVector (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   ->04.58% (71,530,592B) 0x53BFE5F: VecDuplicate_MPI_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   | ->04.58% (71,530,592B) 0x5007F69: VecDuplicate (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |   ->04.58% (71,530,592B) 0x500832F: VecDuplicateVecs_Default (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |     ->04.58% (71,530,592B) 0x50050D5: VecDuplicateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |       ->04.58% (71,530,592B) 0x570A400: KSPCreateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |         ->03.37% (52,706,752B) 0x56AF984: KSPGMRESGetNewVectors (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |         | ->03.37% (52,706,752B) 0x56AEFB0: KSPGMRESCycle (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |         |   ->03.37% (52,706,752B) 0x56AEC7A: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |         |     ->03.37% (52,706,752B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |         |       ->03.37% (52,706,752B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |   |   |   |         |         ->03.37% (52,706,752B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |   |   |         |           
| |   |   |   |         ->01.20% (18,823,840B) 0x56AE79B: KSPSetUp_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |           ->01.20% (18,823,840B) 0x5701286: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |             ->01.20% (18,823,840B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |               ->01.20% (18,823,840B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |   |   |   |                 ->01.20% (18,823,840B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |   |   |                   
| |   |   |   ->00.84% (13,176,688B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   |   
| |   |   ->00.24% (3,764,768B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   
| |   ->05.36% (83,762,688B) 0x4FF1C5F: VecDuplicate_Seq (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->05.36% (83,762,688B) 0x5007F69: VecDuplicate (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->05.29% (82,688,032B) 0x500832F: VecDuplicateVecs_Default (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       | ->05.29% (82,688,032B) 0x50050D5: VecDuplicateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   ->05.29% (82,688,032B) 0x570A400: KSPCreateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     ->02.77% (43,371,200B) 0x56AF984: KSPGMRESGetNewVectors (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     | ->02.77% (43,371,200B) 0x56AEFB0: KSPGMRESCycle (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   ->02.77% (43,371,200B) 0x56AEC7A: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |     ->02.77% (43,371,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       ->02.77% (43,371,200B) 0x56F6F73: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       | ->02.77% (43,371,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |   ->02.77% (43,371,200B) 0x56818FA: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     ->02.45% (38,350,720B) 0x5682B76: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     | ->02.45% (38,350,720B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     |   ->02.41% (37,647,680B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     |   | ->02.41% (37,647,680B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     |   |   ->02.41% (37,647,680B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     |   |     ->02.41% (37,647,680B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |       |     |       |     |   |       ->02.41% (37,647,680B) 0x4048AA: main (main_petsc36.cc:2255)
| |       |     |       |     |   |         
| |       |     |       |     |   ->00.04% (703,040B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |       |     |   
| |       |     |       |     ->00.32% (5,020,480B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |       |     
| |       |     |       ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |       
| |       |     ->01.39% (21,686,880B) 0x56AE79B: KSPSetUp_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     | ->01.39% (21,686,880B) 0x5701286: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   ->01.39% (21,685,600B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | ->01.39% (21,685,600B) 0x56F6F73: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | | ->01.39% (21,685,600B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |   ->01.39% (21,685,600B) 0x56818FA: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     ->01.23% (19,175,360B) 0x5682B76: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     | ->01.23% (19,175,360B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     |   ->01.20% (18,823,840B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     |   | ->01.20% (18,823,840B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     |   |   ->01.20% (18,823,840B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     |   |     ->01.20% (18,823,840B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |       |     |   | |     |   |       ->01.20% (18,823,840B) 0x4048AA: main (main_petsc36.cc:2255)
| |       |     |   | |     |   |         
| |       |     |   | |     |   ->00.02% (351,520B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |   | |     |   
| |       |     |   | |     ->00.16% (2,510,240B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |   | |     
| |       |     |   | ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |   | 
| |       |     |   ->00.00% (1,280B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |   
| |       |     ->01.13% (17,629,952B) in 3 places, all below massif's threshold (01.00%)
| |       |       
| |       ->00.07% (1,074,656B) in 1+ places, all below ms_print's threshold (01.00%)
| |       
| ->03.62% (56,623,104B) 0x417587: CellPropertiesCreate(_p_DM*, _p_CellProperties**) (main_petsc36.cc:75)
| | ->03.62% (56,623,104B) 0x404CBE: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:1860)
| |   ->03.62% (56,623,104B) 0x4048AA: main (main_petsc36.cc:2255)
| |     
| ->03.60% (56,277,776B) 0x51F256F: MatPtAPSymbolic_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->03.60% (56,277,776B) 0x51F192E: MatPtAP_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->03.60% (56,277,776B) 0x509B68C: MatPtAP (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->03.60% (56,277,776B) 0x568798E: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->03.60% (56,277,776B) 0x5693D68: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->03.60% (56,277,776B) 0x57014F7: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |           ->03.60% (56,277,776B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |             ->03.60% (56,277,776B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |               ->03.60% (56,277,776B) 0x4048AA: main (main_petsc36.cc:2255)
| |                 
| ->02.50% (39,034,200B) 0x5307655: MatSOR_SeqAIJ_Inode (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->02.50% (39,034,200B) 0x508F2F7: MatSOR (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->02.50% (39,034,200B) 0x5603367: PCApply_SOR (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->02.50% (39,034,200B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->02.50% (39,034,200B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->02.50% (39,034,200B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |           ->02.50% (39,034,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |             ->02.50% (39,034,200B) 0x56F6F73: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |               ->02.50% (39,034,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                 ->02.50% (39,034,200B) 0x56818FA: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   ->02.21% (34,515,708B) 0x5682B76: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   | ->02.21% (34,515,708B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   ->02.17% (33,882,942B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   | ->02.17% (33,882,942B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   |   ->02.17% (33,882,942B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   |     ->02.17% (33,882,942B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |                   |   |       ->02.17% (33,882,942B) 0x4048AA: main (main_petsc36.cc:2255)
| |                   |   |         
| |                   |   ->00.04% (632,766B) in 1+ places, all below ms_print's threshold (01.00%)
| |                   |   
| |                   ->00.29% (4,518,492B) in 1+ places, all below ms_print's threshold (01.00%)
| |                   
| ->01.80% (28,138,888B) 0x51F2440: MatPtAPSymbolic_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->01.80% (28,138,888B) 0x51F192E: MatPtAP_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->01.80% (28,138,888B) 0x509B68C: MatPtAP (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->01.80% (28,138,888B) 0x568798E: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->01.80% (28,138,888B) 0x5693D68: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->01.80% (28,138,888B) 0x57014F7: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |           ->01.80% (28,138,888B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |             ->01.80% (28,138,888B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |               ->01.80% (28,138,888B) 0x4048AA: main (main_petsc36.cc:2255)
| |                 
| ->01.42% (22,222,049B) in 189 places, all below massif's threshold (01.00%)
|   
->00.54% (8,457,790B) in 1+ places, all below ms_print's threshold (01.00%)

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 51 180,884,467,808    1,567,051,280    1,567,001,403        49,877            0
 52 180,885,185,402    1,574,584,928    1,574,534,867        50,061            0
 53 180,885,548,180    1,582,118,464    1,582,068,243        50,221            0
100.00% (1,582,068,243B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->99.46% (1,573,610,453B) 0x4F3E598: PetscMallocAlign (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| ->74.60% (1,180,246,040B) 0x52B1D78: MatSeqAIJSetPreallocation_SeqAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->74.60% (1,180,246,040B) 0x52B3D19: MatSeqAIJSetPreallocation (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->74.23% (1,174,436,836B) 0x53EEE0C: DMCreateMatrix_DA_3d_MPIAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   | ->74.23% (1,174,436,836B) 0x53E4FB5: DMCreateMatrix_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   ->74.23% (1,174,436,836B) 0x553AA24: DMCreateMatrix (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |     ->37.12% (587,218,418B) 0x40614B: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2028)
| |   |     | ->37.12% (587,218,418B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |     |   
| |   |     ->37.12% (587,218,418B) 0x40616C: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2029)
| |   |       ->37.12% (587,218,418B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |         
| |   ->00.37% (5,809,204B) in 1+ places, all below ms_print's threshold (01.00%)
| |   
| ->12.08% (191,058,576B) 0x4FF7CFF: VecCreate_Seq (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->12.08% (191,058,576B) 0x5008681: VecSetType (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->06.78% (107,295,888B) 0x4FF8EA3: VecCreate_Standard (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   | ->06.78% (107,295,888B) 0x5008681: VecSetType (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   ->06.54% (103,531,120B) 0x53C0070: DMCreateGlobalVector_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   | ->06.54% (103,531,120B) 0x553D255: DMCreateGlobalVector (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   ->05.71% (90,354,432B) 0x53BFE5F: VecDuplicate_MPI_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   | ->05.71% (90,354,432B) 0x5007F69: VecDuplicate (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |   ->05.71% (90,354,432B) 0x500832F: VecDuplicateVecs_Default (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |     ->05.71% (90,354,432B) 0x50050D5: VecDuplicateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |       ->05.71% (90,354,432B) 0x570A400: KSPCreateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |         ->04.52% (71,530,592B) 0x56AF984: KSPGMRESGetNewVectors (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |         | ->04.52% (71,530,592B) 0x56AEFB0: KSPGMRESCycle (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |         |   ->04.52% (71,530,592B) 0x56AEC7A: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |         |     ->04.52% (71,530,592B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |         |       ->04.52% (71,530,592B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |   |   |   |         |         ->04.52% (71,530,592B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |   |   |         |           
| |   |   |   |         ->01.19% (18,823,840B) 0x56AE79B: KSPSetUp_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |           ->01.19% (18,823,840B) 0x5701286: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |             ->01.19% (18,823,840B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |               ->01.19% (18,823,840B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |   |   |   |                 ->01.19% (18,823,840B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |   |   |                   
| |   |   |   ->00.83% (13,176,688B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   |   
| |   |   ->00.24% (3,764,768B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   
| |   ->05.29% (83,762,688B) 0x4FF1C5F: VecDuplicate_Seq (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->05.29% (83,762,688B) 0x5007F69: VecDuplicate (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->05.23% (82,688,032B) 0x500832F: VecDuplicateVecs_Default (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       | ->05.23% (82,688,032B) 0x50050D5: VecDuplicateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   ->05.23% (82,688,032B) 0x570A400: KSPCreateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     ->02.74% (43,371,200B) 0x56AF984: KSPGMRESGetNewVectors (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     | ->02.74% (43,371,200B) 0x56AEFB0: KSPGMRESCycle (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   ->02.74% (43,371,200B) 0x56AEC7A: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |     ->02.74% (43,371,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       ->02.74% (43,371,200B) 0x56F6F73: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       | ->02.74% (43,371,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |   ->02.74% (43,371,200B) 0x56818FA: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     ->02.42% (38,350,720B) 0x5682B76: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     | ->02.42% (38,350,720B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     |   ->02.38% (37,647,680B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     |   | ->02.38% (37,647,680B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     |   |   ->02.38% (37,647,680B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     |   |     ->02.38% (37,647,680B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |       |     |       |     |   |       ->02.38% (37,647,680B) 0x4048AA: main (main_petsc36.cc:2255)
| |       |     |       |     |   |         
| |       |     |       |     |   ->00.04% (703,040B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |       |     |   
| |       |     |       |     ->00.32% (5,020,480B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |       |     
| |       |     |       ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |       
| |       |     ->01.37% (21,686,880B) 0x56AE79B: KSPSetUp_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     | ->01.37% (21,686,880B) 0x5701286: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   ->01.37% (21,685,600B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | ->01.37% (21,685,600B) 0x56F6F73: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | | ->01.37% (21,685,600B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |   ->01.37% (21,685,600B) 0x56818FA: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     ->01.21% (19,175,360B) 0x5682B76: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     | ->01.21% (19,175,360B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     |   ->01.19% (18,823,840B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     |   | ->01.19% (18,823,840B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     |   |   ->01.19% (18,823,840B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     |   |     ->01.19% (18,823,840B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |       |     |   | |     |   |       ->01.19% (18,823,840B) 0x4048AA: main (main_petsc36.cc:2255)
| |       |     |   | |     |   |         
| |       |     |   | |     |   ->00.02% (351,520B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |   | |     |   
| |       |     |   | |     ->00.16% (2,510,240B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |   | |     
| |       |     |   | ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |   | 
| |       |     |   ->00.00% (1,280B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |   
| |       |     ->01.11% (17,629,952B) in 3 places, all below massif's threshold (01.00%)
| |       |       
| |       ->00.07% (1,074,656B) in 1+ places, all below ms_print's threshold (01.00%)
| |       
| ->03.58% (56,623,104B) 0x417587: CellPropertiesCreate(_p_DM*, _p_CellProperties**) (main_petsc36.cc:75)
| | ->03.58% (56,623,104B) 0x404CBE: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:1860)
| |   ->03.58% (56,623,104B) 0x4048AA: main (main_petsc36.cc:2255)
| |     
| ->03.56% (56,277,776B) 0x51F256F: MatPtAPSymbolic_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->03.56% (56,277,776B) 0x51F192E: MatPtAP_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->03.56% (56,277,776B) 0x509B68C: MatPtAP (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->03.56% (56,277,776B) 0x568798E: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->03.56% (56,277,776B) 0x5693D68: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->03.56% (56,277,776B) 0x57014F7: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |           ->03.56% (56,277,776B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |             ->03.56% (56,277,776B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |               ->03.56% (56,277,776B) 0x4048AA: main (main_petsc36.cc:2255)
| |                 
| ->02.47% (39,034,200B) 0x5307655: MatSOR_SeqAIJ_Inode (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->02.47% (39,034,200B) 0x508F2F7: MatSOR (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->02.47% (39,034,200B) 0x5603367: PCApply_SOR (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->02.47% (39,034,200B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->02.47% (39,034,200B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->02.47% (39,034,200B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |           ->02.47% (39,034,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |             ->02.47% (39,034,200B) 0x56F6F73: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |               ->02.47% (39,034,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                 ->02.47% (39,034,200B) 0x56818FA: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   ->02.18% (34,515,708B) 0x5682B76: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   | ->02.18% (34,515,708B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   ->02.14% (33,882,942B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   | ->02.14% (33,882,942B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   |   ->02.14% (33,882,942B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   |     ->02.14% (33,882,942B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |                   |   |       ->02.14% (33,882,942B) 0x4048AA: main (main_petsc36.cc:2255)
| |                   |   |         
| |                   |   ->00.04% (632,766B) in 1+ places, all below ms_print's threshold (01.00%)
| |                   |   
| |                   ->00.29% (4,518,492B) in 1+ places, all below ms_print's threshold (01.00%)
| |                   
| ->01.78% (28,138,888B) 0x51F2440: MatPtAPSymbolic_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->01.78% (28,138,888B) 0x51F192E: MatPtAP_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->01.78% (28,138,888B) 0x509B68C: MatPtAP (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->01.78% (28,138,888B) 0x568798E: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->01.78% (28,138,888B) 0x5693D68: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->01.78% (28,138,888B) 0x57014F7: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |           ->01.78% (28,138,888B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |             ->01.78% (28,138,888B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |               ->01.78% (28,138,888B) 0x4048AA: main (main_petsc36.cc:2255)
| |                 
| ->01.41% (22,231,869B) in 189 places, all below massif's threshold (01.00%)
|   
->00.53% (8,457,790B) in 1+ places, all below ms_print's threshold (01.00%)

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 54 180,886,261,793    1,585,885,400    1,585,835,063        50,337            0
 55 241,478,135,876    1,585,885,776    1,585,835,439        50,337            0
 56 241,478,852,571    1,593,419,136    1,593,368,607        50,529            0
 57 241,479,215,349    1,600,952,672    1,600,901,983        50,689            0
100.00% (1,600,901,983B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->99.47% (1,592,444,193B) 0x4F3E598: PetscMallocAlign (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| ->73.72% (1,180,246,040B) 0x52B1D78: MatSeqAIJSetPreallocation_SeqAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->73.72% (1,180,246,040B) 0x52B3D19: MatSeqAIJSetPreallocation (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->73.36% (1,174,436,836B) 0x53EEE0C: DMCreateMatrix_DA_3d_MPIAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   | ->73.36% (1,174,436,836B) 0x53E4FB5: DMCreateMatrix_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   ->73.36% (1,174,436,836B) 0x553AA24: DMCreateMatrix (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |     ->36.68% (587,218,418B) 0x40614B: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2028)
| |   |     | ->36.68% (587,218,418B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |     |   
| |   |     ->36.68% (587,218,418B) 0x40616C: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2029)
| |   |       ->36.68% (587,218,418B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |         
| |   ->00.36% (5,809,204B) in 1+ places, all below ms_print's threshold (01.00%)
| |   
| ->13.11% (209,882,416B) 0x4FF7CFF: VecCreate_Seq (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->13.11% (209,882,416B) 0x5008681: VecSetType (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->07.88% (126,119,728B) 0x4FF8EA3: VecCreate_Standard (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   | ->07.88% (126,119,728B) 0x5008681: VecSetType (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   ->07.64% (122,354,960B) 0x53C0070: DMCreateGlobalVector_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   | ->07.64% (122,354,960B) 0x553D255: DMCreateGlobalVector (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   ->06.82% (109,178,272B) 0x53BFE5F: VecDuplicate_MPI_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   | ->06.82% (109,178,272B) 0x5007F69: VecDuplicate (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |   ->06.82% (109,178,272B) 0x500832F: VecDuplicateVecs_Default (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |     ->06.82% (109,178,272B) 0x50050D5: VecDuplicateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |       ->06.82% (109,178,272B) 0x570A400: KSPCreateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |         ->05.64% (90,354,432B) 0x56AF984: KSPGMRESGetNewVectors (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |         | ->05.64% (90,354,432B) 0x56AEFB0: KSPGMRESCycle (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |         |   ->05.64% (90,354,432B) 0x56AEC7A: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |         |     ->05.64% (90,354,432B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |         |       ->05.64% (90,354,432B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |   |   |   |         |         ->05.64% (90,354,432B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |   |   |         |           
| |   |   |   |         ->01.18% (18,823,840B) 0x56AE79B: KSPSetUp_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |           ->01.18% (18,823,840B) 0x5701286: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |             ->01.18% (18,823,840B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |               ->01.18% (18,823,840B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |   |   |   |                 ->01.18% (18,823,840B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |   |   |                   
| |   |   |   ->00.82% (13,176,688B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   |   
| |   |   ->00.24% (3,764,768B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   
| |   ->05.23% (83,762,688B) 0x4FF1C5F: VecDuplicate_Seq (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->05.23% (83,762,688B) 0x5007F69: VecDuplicate (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->05.16% (82,688,032B) 0x500832F: VecDuplicateVecs_Default (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       | ->05.16% (82,688,032B) 0x50050D5: VecDuplicateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   ->05.16% (82,688,032B) 0x570A400: KSPCreateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     ->02.71% (43,371,200B) 0x56AF984: KSPGMRESGetNewVectors (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     | ->02.71% (43,371,200B) 0x56AEFB0: KSPGMRESCycle (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   ->02.71% (43,371,200B) 0x56AEC7A: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |     ->02.71% (43,371,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       ->02.71% (43,371,200B) 0x56F6F73: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       | ->02.71% (43,371,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |   ->02.71% (43,371,200B) 0x56818FA: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     ->02.40% (38,350,720B) 0x5682B76: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     | ->02.40% (38,350,720B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     |   ->02.35% (37,647,680B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     |   | ->02.35% (37,647,680B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     |   |   ->02.35% (37,647,680B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     |   |     ->02.35% (37,647,680B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |       |     |       |     |   |       ->02.35% (37,647,680B) 0x4048AA: main (main_petsc36.cc:2255)
| |       |     |       |     |   |         
| |       |     |       |     |   ->00.04% (703,040B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |       |     |   
| |       |     |       |     ->00.31% (5,020,480B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |       |     
| |       |     |       ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |       
| |       |     ->01.35% (21,686,880B) 0x56AE79B: KSPSetUp_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     | ->01.35% (21,686,880B) 0x5701286: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   ->01.35% (21,685,600B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | ->01.35% (21,685,600B) 0x56F6F73: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | | ->01.35% (21,685,600B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |   ->01.35% (21,685,600B) 0x56818FA: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     ->01.20% (19,175,360B) 0x5682B76: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     | ->01.20% (19,175,360B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     |   ->01.18% (18,823,840B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     |   | ->01.18% (18,823,840B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     |   |   ->01.18% (18,823,840B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     |   |     ->01.18% (18,823,840B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |       |     |   | |     |   |       ->01.18% (18,823,840B) 0x4048AA: main (main_petsc36.cc:2255)
| |       |     |   | |     |   |         
| |       |     |   | |     |   ->00.02% (351,520B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |   | |     |   
| |       |     |   | |     ->00.16% (2,510,240B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |   | |     
| |       |     |   | ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |   | 
| |       |     |   ->00.00% (1,280B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |   
| |       |     ->01.10% (17,629,952B) in 3 places, all below massif's threshold (01.00%)
| |       |       
| |       ->00.07% (1,074,656B) in 1+ places, all below ms_print's threshold (01.00%)
| |       
| ->03.54% (56,623,104B) 0x417587: CellPropertiesCreate(_p_DM*, _p_CellProperties**) (main_petsc36.cc:75)
| | ->03.54% (56,623,104B) 0x404CBE: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:1860)
| |   ->03.54% (56,623,104B) 0x4048AA: main (main_petsc36.cc:2255)
| |     
| ->03.52% (56,277,776B) 0x51F256F: MatPtAPSymbolic_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->03.52% (56,277,776B) 0x51F192E: MatPtAP_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->03.52% (56,277,776B) 0x509B68C: MatPtAP (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->03.52% (56,277,776B) 0x568798E: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->03.52% (56,277,776B) 0x5693D68: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->03.52% (56,277,776B) 0x57014F7: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |           ->03.52% (56,277,776B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |             ->03.52% (56,277,776B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |               ->03.52% (56,277,776B) 0x4048AA: main (main_petsc36.cc:2255)
| |                 
| ->02.44% (39,034,200B) 0x5307655: MatSOR_SeqAIJ_Inode (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->02.44% (39,034,200B) 0x508F2F7: MatSOR (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->02.44% (39,034,200B) 0x5603367: PCApply_SOR (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->02.44% (39,034,200B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->02.44% (39,034,200B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->02.44% (39,034,200B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |           ->02.44% (39,034,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |             ->02.44% (39,034,200B) 0x56F6F73: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |               ->02.44% (39,034,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                 ->02.44% (39,034,200B) 0x56818FA: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   ->02.16% (34,515,708B) 0x5682B76: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   | ->02.16% (34,515,708B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   ->02.12% (33,882,942B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   | ->02.12% (33,882,942B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   |   ->02.12% (33,882,942B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   |     ->02.12% (33,882,942B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |                   |   |       ->02.12% (33,882,942B) 0x4048AA: main (main_petsc36.cc:2255)
| |                   |   |         
| |                   |   ->00.04% (632,766B) in 1+ places, all below ms_print's threshold (01.00%)
| |                   |   
| |                   ->00.28% (4,518,492B) in 1+ places, all below ms_print's threshold (01.00%)
| |                   
| ->01.76% (28,138,888B) 0x51F2440: MatPtAPSymbolic_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->01.76% (28,138,888B) 0x51F192E: MatPtAP_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->01.76% (28,138,888B) 0x509B68C: MatPtAP (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->01.76% (28,138,888B) 0x568798E: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->01.76% (28,138,888B) 0x5693D68: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->01.76% (28,138,888B) 0x57014F7: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |           ->01.76% (28,138,888B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |             ->01.76% (28,138,888B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |               ->01.76% (28,138,888B) 0x4048AA: main (main_petsc36.cc:2255)
| |                 
| ->01.39% (22,241,769B) in 189 places, all below massif's threshold (01.00%)
|   
->00.53% (8,457,790B) in 1+ places, all below ms_print's threshold (01.00%)

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 58 241,479,928,962    1,604,719,608    1,604,668,803        50,805            0
 59 241,480,646,554    1,612,253,256    1,612,202,267        50,989            0
 60 241,481,009,330    1,619,786,792    1,619,735,643        51,149            0
100.00% (1,619,735,643B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->99.47% (1,611,277,853B) 0x4F3E598: PetscMallocAlign (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| ->72.86% (1,180,246,040B) 0x52B1D78: MatSeqAIJSetPreallocation_SeqAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->72.86% (1,180,246,040B) 0x52B3D19: MatSeqAIJSetPreallocation (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->72.51% (1,174,436,836B) 0x53EEE0C: DMCreateMatrix_DA_3d_MPIAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   | ->72.51% (1,174,436,836B) 0x53E4FB5: DMCreateMatrix_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   ->72.51% (1,174,436,836B) 0x553AA24: DMCreateMatrix (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |     ->36.25% (587,218,418B) 0x40614B: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2028)
| |   |     | ->36.25% (587,218,418B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |     |   
| |   |     ->36.25% (587,218,418B) 0x40616C: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2029)
| |   |       ->36.25% (587,218,418B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |         
| |   ->00.36% (5,809,204B) in 1+ places, all below ms_print's threshold (01.00%)
| |   
| ->14.12% (228,706,256B) 0x4FF7CFF: VecCreate_Seq (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->14.12% (228,706,256B) 0x5008681: VecSetType (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->08.95% (144,943,568B) 0x4FF8EA3: VecCreate_Standard (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   | ->08.95% (144,943,568B) 0x5008681: VecSetType (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   ->08.72% (141,178,800B) 0x53C0070: DMCreateGlobalVector_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   | ->08.72% (141,178,800B) 0x553D255: DMCreateGlobalVector (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   ->07.90% (128,002,112B) 0x53BFE5F: VecDuplicate_MPI_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   | ->07.90% (128,002,112B) 0x5007F69: VecDuplicate (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |   ->07.90% (128,002,112B) 0x500832F: VecDuplicateVecs_Default (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |     ->07.90% (128,002,112B) 0x50050D5: VecDuplicateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |       ->07.90% (128,002,112B) 0x570A400: KSPCreateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |         ->06.74% (109,178,272B) 0x56AF984: KSPGMRESGetNewVectors (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |         | ->06.74% (109,178,272B) 0x56AEFB0: KSPGMRESCycle (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |         |   ->06.74% (109,178,272B) 0x56AEC7A: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |         |     ->06.74% (109,178,272B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |         |       ->06.74% (109,178,272B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |   |   |   |         |         ->06.74% (109,178,272B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |   |   |         |           
| |   |   |   |         ->01.16% (18,823,840B) 0x56AE79B: KSPSetUp_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |           ->01.16% (18,823,840B) 0x5701286: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |             ->01.16% (18,823,840B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |               ->01.16% (18,823,840B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |   |   |   |                 ->01.16% (18,823,840B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |   |   |                   
| |   |   |   ->00.81% (13,176,688B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   |   
| |   |   ->00.23% (3,764,768B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   
| |   ->05.17% (83,762,688B) 0x4FF1C5F: VecDuplicate_Seq (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->05.17% (83,762,688B) 0x5007F69: VecDuplicate (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->05.10% (82,688,032B) 0x500832F: VecDuplicateVecs_Default (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       | ->05.10% (82,688,032B) 0x50050D5: VecDuplicateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   ->05.10% (82,688,032B) 0x570A400: KSPCreateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     ->02.68% (43,371,200B) 0x56AF984: KSPGMRESGetNewVectors (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     | ->02.68% (43,371,200B) 0x56AEFB0: KSPGMRESCycle (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   ->02.68% (43,371,200B) 0x56AEC7A: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |     ->02.68% (43,371,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       ->02.68% (43,371,200B) 0x56F6F73: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       | ->02.68% (43,371,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |   ->02.68% (43,371,200B) 0x56818FA: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     ->02.37% (38,350,720B) 0x5682B76: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     | ->02.37% (38,350,720B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     |   ->02.32% (37,647,680B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     |   | ->02.32% (37,647,680B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     |   |   ->02.32% (37,647,680B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     |   |     ->02.32% (37,647,680B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |       |     |       |     |   |       ->02.32% (37,647,680B) 0x4048AA: main (main_petsc36.cc:2255)
| |       |     |       |     |   |         
| |       |     |       |     |   ->00.04% (703,040B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |       |     |   
| |       |     |       |     ->00.31% (5,020,480B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |       |     
| |       |     |       ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |       
| |       |     ->01.34% (21,686,880B) 0x56AE79B: KSPSetUp_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     | ->01.34% (21,686,880B) 0x5701286: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   ->01.34% (21,685,600B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | ->01.34% (21,685,600B) 0x56F6F73: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | | ->01.34% (21,685,600B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |   ->01.34% (21,685,600B) 0x56818FA: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     ->01.18% (19,175,360B) 0x5682B76: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     | ->01.18% (19,175,360B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     |   ->01.16% (18,823,840B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     |   | ->01.16% (18,823,840B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     |   |   ->01.16% (18,823,840B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     |   |     ->01.16% (18,823,840B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |       |     |   | |     |   |       ->01.16% (18,823,840B) 0x4048AA: main (main_petsc36.cc:2255)
| |       |     |   | |     |   |         
| |       |     |   | |     |   ->00.02% (351,520B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |   | |     |   
| |       |     |   | |     ->00.15% (2,510,240B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |   | |     
| |       |     |   | ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |   | 
| |       |     |   ->00.00% (1,280B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |   
| |       |     ->01.09% (17,629,952B) in 3 places, all below massif's threshold (01.00%)
| |       |       
| |       ->00.07% (1,074,656B) in 1+ places, all below ms_print's threshold (01.00%)
| |       
| ->03.50% (56,623,104B) 0x417587: CellPropertiesCreate(_p_DM*, _p_CellProperties**) (main_petsc36.cc:75)
| | ->03.50% (56,623,104B) 0x404CBE: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:1860)
| |   ->03.50% (56,623,104B) 0x4048AA: main (main_petsc36.cc:2255)
| |     
| ->03.47% (56,277,776B) 0x51F256F: MatPtAPSymbolic_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->03.47% (56,277,776B) 0x51F192E: MatPtAP_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->03.47% (56,277,776B) 0x509B68C: MatPtAP (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->03.47% (56,277,776B) 0x568798E: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->03.47% (56,277,776B) 0x5693D68: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->03.47% (56,277,776B) 0x57014F7: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |           ->03.47% (56,277,776B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |             ->03.47% (56,277,776B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |               ->03.47% (56,277,776B) 0x4048AA: main (main_petsc36.cc:2255)
| |                 
| ->02.41% (39,034,200B) 0x5307655: MatSOR_SeqAIJ_Inode (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->02.41% (39,034,200B) 0x508F2F7: MatSOR (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->02.41% (39,034,200B) 0x5603367: PCApply_SOR (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->02.41% (39,034,200B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->02.41% (39,034,200B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->02.41% (39,034,200B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |           ->02.41% (39,034,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |             ->02.41% (39,034,200B) 0x56F6F73: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |               ->02.41% (39,034,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                 ->02.41% (39,034,200B) 0x56818FA: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   ->02.13% (34,515,708B) 0x5682B76: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   | ->02.13% (34,515,708B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   ->02.09% (33,882,942B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   | ->02.09% (33,882,942B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   |   ->02.09% (33,882,942B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   |     ->02.09% (33,882,942B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |                   |   |       ->02.09% (33,882,942B) 0x4048AA: main (main_petsc36.cc:2255)
| |                   |   |         
| |                   |   ->00.04% (632,766B) in 1+ places, all below ms_print's threshold (01.00%)
| |                   |   
| |                   ->00.28% (4,518,492B) in 1+ places, all below ms_print's threshold (01.00%)
| |                   
| ->01.74% (28,138,888B) 0x51F2440: MatPtAPSymbolic_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->01.74% (28,138,888B) 0x51F192E: MatPtAP_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->01.74% (28,138,888B) 0x509B68C: MatPtAP (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->01.74% (28,138,888B) 0x568798E: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->01.74% (28,138,888B) 0x5693D68: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->01.74% (28,138,888B) 0x57014F7: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |           ->01.74% (28,138,888B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |             ->01.74% (28,138,888B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |               ->01.74% (28,138,888B) 0x4048AA: main (main_petsc36.cc:2255)
| |                 
| ->01.37% (22,251,589B) in 189 places, all below massif's threshold (01.00%)
|   
->00.52% (8,457,790B) in 1+ places, all below ms_print's threshold (01.00%)

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 61 241,481,722,939    1,623,553,728    1,623,502,463        51,265            0
 62 302,564,355,117    1,623,554,104    1,623,502,839        51,265            0
 63 302,565,071,808    1,631,087,464    1,631,036,007        51,457            0
 64 302,565,434,584    1,638,621,000    1,638,569,383        51,617            0
100.00% (1,638,569,383B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->99.48% (1,630,111,593B) 0x4F3E598: PetscMallocAlign (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| ->72.03% (1,180,246,040B) 0x52B1D78: MatSeqAIJSetPreallocation_SeqAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->72.03% (1,180,246,040B) 0x52B3D19: MatSeqAIJSetPreallocation (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->71.67% (1,174,436,836B) 0x53EEE0C: DMCreateMatrix_DA_3d_MPIAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   | ->71.67% (1,174,436,836B) 0x53E4FB5: DMCreateMatrix_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   ->71.67% (1,174,436,836B) 0x553AA24: DMCreateMatrix (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |     ->35.84% (587,218,418B) 0x40614B: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2028)
| |   |     | ->35.84% (587,218,418B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |     |   
| |   |     ->35.84% (587,218,418B) 0x40616C: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2029)
| |   |       ->35.84% (587,218,418B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |         
| |   ->00.35% (5,809,204B) in 1+ places, all below ms_print's threshold (01.00%)
| |   
| ->15.11% (247,530,096B) 0x4FF7CFF: VecCreate_Seq (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->15.11% (247,530,096B) 0x5008681: VecSetType (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->09.99% (163,767,408B) 0x4FF8EA3: VecCreate_Standard (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   | ->09.99% (163,767,408B) 0x5008681: VecSetType (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   ->09.76% (160,002,640B) 0x53C0070: DMCreateGlobalVector_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   | ->09.76% (160,002,640B) 0x553D255: DMCreateGlobalVector (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   ->08.96% (146,825,952B) 0x53BFE5F: VecDuplicate_MPI_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   | ->08.96% (146,825,952B) 0x5007F69: VecDuplicate (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |   ->08.96% (146,825,952B) 0x500832F: VecDuplicateVecs_Default (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |     ->08.96% (146,825,952B) 0x50050D5: VecDuplicateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |       ->08.96% (146,825,952B) 0x570A400: KSPCreateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |         ->07.81% (128,002,112B) 0x56AF984: KSPGMRESGetNewVectors (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |         | ->07.81% (128,002,112B) 0x56AEFB0: KSPGMRESCycle (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |         |   ->07.81% (128,002,112B) 0x56AEC7A: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |         |     ->07.81% (128,002,112B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |         |       ->07.81% (128,002,112B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |   |   |   |         |         ->07.81% (128,002,112B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |   |   |         |           
| |   |   |   |         ->01.15% (18,823,840B) 0x56AE79B: KSPSetUp_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |           ->01.15% (18,823,840B) 0x5701286: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |             ->01.15% (18,823,840B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |               ->01.15% (18,823,840B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |   |   |   |                 ->01.15% (18,823,840B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |   |   |                   
| |   |   |   ->00.80% (13,176,688B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   |   
| |   |   ->00.23% (3,764,768B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   
| |   ->05.11% (83,762,688B) 0x4FF1C5F: VecDuplicate_Seq (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->05.11% (83,762,688B) 0x5007F69: VecDuplicate (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->05.05% (82,688,032B) 0x500832F: VecDuplicateVecs_Default (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       | ->05.05% (82,688,032B) 0x50050D5: VecDuplicateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   ->05.05% (82,688,032B) 0x570A400: KSPCreateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     ->02.65% (43,371,200B) 0x56AF984: KSPGMRESGetNewVectors (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     | ->02.65% (43,371,200B) 0x56AEFB0: KSPGMRESCycle (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   ->02.65% (43,371,200B) 0x56AEC7A: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |     ->02.65% (43,371,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       ->02.65% (43,371,200B) 0x56F6F73: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       | ->02.65% (43,371,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |   ->02.65% (43,371,200B) 0x56818FA: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     ->02.34% (38,350,720B) 0x5682B76: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     | ->02.34% (38,350,720B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     |   ->02.30% (37,647,680B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     |   | ->02.30% (37,647,680B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     |   |   ->02.30% (37,647,680B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     |   |     ->02.30% (37,647,680B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |       |     |       |     |   |       ->02.30% (37,647,680B) 0x4048AA: main (main_petsc36.cc:2255)
| |       |     |       |     |   |         
| |       |     |       |     |   ->00.04% (703,040B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |       |     |   
| |       |     |       |     ->00.31% (5,020,480B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |       |     
| |       |     |       ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |       
| |       |     ->01.32% (21,686,880B) 0x56AE79B: KSPSetUp_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     | ->01.32% (21,686,880B) 0x5701286: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   ->01.32% (21,685,600B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | ->01.32% (21,685,600B) 0x56F6F73: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | | ->01.32% (21,685,600B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |   ->01.32% (21,685,600B) 0x56818FA: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     ->01.17% (19,175,360B) 0x5682B76: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     | ->01.17% (19,175,360B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     |   ->01.15% (18,823,840B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     |   | ->01.15% (18,823,840B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     |   |   ->01.15% (18,823,840B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     |   |     ->01.15% (18,823,840B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |       |     |   | |     |   |       ->01.15% (18,823,840B) 0x4048AA: main (main_petsc36.cc:2255)
| |       |     |   | |     |   |         
| |       |     |   | |     |   ->00.02% (351,520B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |   | |     |   
| |       |     |   | |     ->00.15% (2,510,240B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |   | |     
| |       |     |   | ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |   | 
| |       |     |   ->00.00% (1,280B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |   
| |       |     ->01.08% (17,629,952B) in 3 places, all below massif's threshold (01.00%)
| |       |       
| |       ->00.07% (1,074,656B) in 1+ places, all below ms_print's threshold (01.00%)
| |       
| ->03.46% (56,623,104B) 0x417587: CellPropertiesCreate(_p_DM*, _p_CellProperties**) (main_petsc36.cc:75)
| | ->03.46% (56,623,104B) 0x404CBE: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:1860)
| |   ->03.46% (56,623,104B) 0x4048AA: main (main_petsc36.cc:2255)
| |     
| ->03.43% (56,277,776B) 0x51F256F: MatPtAPSymbolic_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->03.43% (56,277,776B) 0x51F192E: MatPtAP_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->03.43% (56,277,776B) 0x509B68C: MatPtAP (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->03.43% (56,277,776B) 0x568798E: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->03.43% (56,277,776B) 0x5693D68: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->03.43% (56,277,776B) 0x57014F7: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |           ->03.43% (56,277,776B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |             ->03.43% (56,277,776B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |               ->03.43% (56,277,776B) 0x4048AA: main (main_petsc36.cc:2255)
| |                 
| ->02.38% (39,034,200B) 0x5307655: MatSOR_SeqAIJ_Inode (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->02.38% (39,034,200B) 0x508F2F7: MatSOR (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->02.38% (39,034,200B) 0x5603367: PCApply_SOR (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->02.38% (39,034,200B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->02.38% (39,034,200B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->02.38% (39,034,200B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |           ->02.38% (39,034,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |             ->02.38% (39,034,200B) 0x56F6F73: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |               ->02.38% (39,034,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                 ->02.38% (39,034,200B) 0x56818FA: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   ->02.11% (34,515,708B) 0x5682B76: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   | ->02.11% (34,515,708B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   ->02.07% (33,882,942B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   | ->02.07% (33,882,942B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   |   ->02.07% (33,882,942B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   |     ->02.07% (33,882,942B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |                   |   |       ->02.07% (33,882,942B) 0x4048AA: main (main_petsc36.cc:2255)
| |                   |   |         
| |                   |   ->00.04% (632,766B) in 1+ places, all below ms_print's threshold (01.00%)
| |                   |   
| |                   ->00.28% (4,518,492B) in 1+ places, all below ms_print's threshold (01.00%)
| |                   
| ->01.72% (28,138,888B) 0x51F2440: MatPtAPSymbolic_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->01.72% (28,138,888B) 0x51F192E: MatPtAP_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->01.72% (28,138,888B) 0x509B68C: MatPtAP (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->01.72% (28,138,888B) 0x568798E: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->01.72% (28,138,888B) 0x5693D68: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->01.72% (28,138,888B) 0x57014F7: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |           ->01.72% (28,138,888B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |             ->01.72% (28,138,888B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |               ->01.72% (28,138,888B) 0x4048AA: main (main_petsc36.cc:2255)
| |                 
| ->01.36% (22,261,489B) in 189 places, all below massif's threshold (01.00%)
|   
->00.52% (8,457,790B) in 1+ places, all below ms_print's threshold (01.00%)

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 65 302,566,148,193    1,642,387,936    1,642,336,203        51,733            0
 66 302,566,865,783    1,649,921,584    1,649,869,667        51,917            0
 67 302,567,228,559    1,657,455,120    1,657,403,043        52,077            0
100.00% (1,657,403,043B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->99.49% (1,648,945,253B) 0x4F3E598: PetscMallocAlign (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| ->71.21% (1,180,246,040B) 0x52B1D78: MatSeqAIJSetPreallocation_SeqAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->71.21% (1,180,246,040B) 0x52B3D19: MatSeqAIJSetPreallocation (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->70.86% (1,174,436,836B) 0x53EEE0C: DMCreateMatrix_DA_3d_MPIAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   | ->70.86% (1,174,436,836B) 0x53E4FB5: DMCreateMatrix_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   ->70.86% (1,174,436,836B) 0x553AA24: DMCreateMatrix (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |     ->35.43% (587,218,418B) 0x40614B: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2028)
| |   |     | ->35.43% (587,218,418B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |     |   
| |   |     ->35.43% (587,218,418B) 0x40616C: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2029)
| |   |       ->35.43% (587,218,418B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |         
| |   ->00.35% (5,809,204B) in 1+ places, all below ms_print's threshold (01.00%)
| |   
| ->16.07% (266,353,936B) 0x4FF7CFF: VecCreate_Seq (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->16.07% (266,353,936B) 0x5008681: VecSetType (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->11.02% (182,591,248B) 0x4FF8EA3: VecCreate_Standard (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   | ->11.02% (182,591,248B) 0x5008681: VecSetType (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   ->10.79% (178,826,480B) 0x53C0070: DMCreateGlobalVector_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   | ->10.79% (178,826,480B) 0x553D255: DMCreateGlobalVector (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   ->09.99% (165,649,792B) 0x53BFE5F: VecDuplicate_MPI_DA (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   | ->09.99% (165,649,792B) 0x5007F69: VecDuplicate (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |   ->09.99% (165,649,792B) 0x500832F: VecDuplicateVecs_Default (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |     ->09.99% (165,649,792B) 0x50050D5: VecDuplicateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |       ->09.99% (165,649,792B) 0x570A400: KSPCreateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |         ->08.86% (146,825,952B) 0x56AF984: KSPGMRESGetNewVectors (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |         | ->08.86% (146,825,952B) 0x56AEFB0: KSPGMRESCycle (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |         |   ->08.86% (146,825,952B) 0x56AEC7A: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |         |     ->08.86% (146,825,952B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |         |       ->08.86% (146,825,952B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |   |   |   |         |         ->08.86% (146,825,952B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |   |   |         |           
| |   |   |   |         ->01.14% (18,823,840B) 0x56AE79B: KSPSetUp_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |           ->01.14% (18,823,840B) 0x5701286: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |             ->01.14% (18,823,840B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   |   |   |               ->01.14% (18,823,840B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |   |   |   |                 ->01.14% (18,823,840B) 0x4048AA: main (main_petsc36.cc:2255)
| |   |   |   |                   
| |   |   |   ->00.79% (13,176,688B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   |   
| |   |   ->00.23% (3,764,768B) in 1+ places, all below ms_print's threshold (01.00%)
| |   |   
| |   ->05.05% (83,762,688B) 0x4FF1C5F: VecDuplicate_Seq (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->05.05% (83,762,688B) 0x5007F69: VecDuplicate (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->04.99% (82,688,032B) 0x500832F: VecDuplicateVecs_Default (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       | ->04.99% (82,688,032B) 0x50050D5: VecDuplicateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |   ->04.99% (82,688,032B) 0x570A400: KSPCreateVecs (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     ->02.62% (43,371,200B) 0x56AF984: KSPGMRESGetNewVectors (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     | ->02.62% (43,371,200B) 0x56AEFB0: KSPGMRESCycle (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   ->02.62% (43,371,200B) 0x56AEC7A: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |     ->02.62% (43,371,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       ->02.62% (43,371,200B) 0x56F6F73: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       | ->02.62% (43,371,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |   ->02.62% (43,371,200B) 0x56818FA: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     ->02.31% (38,350,720B) 0x5682B76: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     | ->02.31% (38,350,720B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     |   ->02.27% (37,647,680B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     |   | ->02.27% (37,647,680B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     |   |   ->02.27% (37,647,680B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |       |     |   |     ->02.27% (37,647,680B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |       |     |       |     |   |       ->02.27% (37,647,680B) 0x4048AA: main (main_petsc36.cc:2255)
| |       |     |       |     |   |         
| |       |     |       |     |   ->00.04% (703,040B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |       |     |   
| |       |     |       |     ->00.30% (5,020,480B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |       |     
| |       |     |       ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |       
| |       |     ->01.31% (21,686,880B) 0x56AE79B: KSPSetUp_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     | ->01.31% (21,686,880B) 0x5701286: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   ->01.31% (21,685,600B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | ->01.31% (21,685,600B) 0x56F6F73: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | | ->01.31% (21,685,600B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |   ->01.31% (21,685,600B) 0x56818FA: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     ->01.16% (19,175,360B) 0x5682B76: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     | ->01.16% (19,175,360B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     |   ->01.14% (18,823,840B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     |   | ->01.14% (18,823,840B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     |   |   ->01.14% (18,823,840B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       |     |   | |     |   |     ->01.14% (18,823,840B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |       |     |   | |     |   |       ->01.14% (18,823,840B) 0x4048AA: main (main_petsc36.cc:2255)
| |       |     |   | |     |   |         
| |       |     |   | |     |   ->00.02% (351,520B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |   | |     |   
| |       |     |   | |     ->00.15% (2,510,240B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |   | |     
| |       |     |   | ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |   | 
| |       |     |   ->00.00% (1,280B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |     |   
| |       |     ->01.06% (17,629,952B) in 3 places, all below massif's threshold (01.00%)
| |       |       
| |       ->00.06% (1,074,656B) in 1+ places, all below ms_print's threshold (01.00%)
| |       
| ->03.42% (56,623,104B) 0x417587: CellPropertiesCreate(_p_DM*, _p_CellProperties**) (main_petsc36.cc:75)
| | ->03.42% (56,623,104B) 0x404CBE: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:1860)
| |   ->03.42% (56,623,104B) 0x4048AA: main (main_petsc36.cc:2255)
| |     
| ->03.40% (56,277,776B) 0x51F256F: MatPtAPSymbolic_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->03.40% (56,277,776B) 0x51F192E: MatPtAP_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->03.40% (56,277,776B) 0x509B68C: MatPtAP (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->03.40% (56,277,776B) 0x568798E: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->03.40% (56,277,776B) 0x5693D68: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->03.40% (56,277,776B) 0x57014F7: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |           ->03.40% (56,277,776B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |             ->03.40% (56,277,776B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |               ->03.40% (56,277,776B) 0x4048AA: main (main_petsc36.cc:2255)
| |                 
| ->02.36% (39,034,200B) 0x5307655: MatSOR_SeqAIJ_Inode (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->02.36% (39,034,200B) 0x508F2F7: MatSOR (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->02.36% (39,034,200B) 0x5603367: PCApply_SOR (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->02.36% (39,034,200B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->02.36% (39,034,200B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->02.36% (39,034,200B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |           ->02.36% (39,034,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |             ->02.36% (39,034,200B) 0x56F6F73: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |               ->02.36% (39,034,200B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                 ->02.36% (39,034,200B) 0x56818FA: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   ->02.08% (34,515,708B) 0x5682B76: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   | ->02.08% (34,515,708B) 0x5698127: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   ->02.04% (33,882,942B) 0x570D0E0: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   | ->02.04% (33,882,942B) 0x56AEC65: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   |   ->02.04% (33,882,942B) 0x56FDF2E: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |                   |   |     ->02.04% (33,882,942B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |                   |   |       ->02.04% (33,882,942B) 0x4048AA: main (main_petsc36.cc:2255)
| |                   |   |         
| |                   |   ->00.04% (632,766B) in 1+ places, all below ms_print's threshold (01.00%)
| |                   |   
| |                   ->00.27% (4,518,492B) in 1+ places, all below ms_print's threshold (01.00%)
| |                   
| ->01.70% (28,138,888B) 0x51F2440: MatPtAPSymbolic_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| | ->01.70% (28,138,888B) 0x51F192E: MatPtAP_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |   ->01.70% (28,138,888B) 0x509B68C: MatPtAP (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |     ->01.70% (28,138,888B) 0x568798E: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |       ->01.70% (28,138,888B) 0x5693D68: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |         ->01.70% (28,138,888B) 0x57014F7: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |           ->01.70% (28,138,888B) 0x56FDBFB: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
| |             ->01.70% (28,138,888B) 0x4069AB: solve_stokes_3d_coupled(int, int, int) (main_petsc36.cc:2133)
| |               ->01.70% (28,138,888B) 0x4048AA: main (main_petsc36.cc:2255)
| |                 
| ->01.34% (22,271,309B) in 189 places, all below massif's threshold (01.00%)
|   
->00.51% (8,457,790B) in 1+ places, all below ms_print's threshold (01.00%)

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 68 302,567,942,168    1,661,222,056    1,661,169,863        52,193            0
 69 327,133,263,563    1,661,222,384    1,661,170,189        52,195            0
 70 327,134,801,224    1,661,225,528    1,661,172,873        52,655            0
 71 327,135,484,200    1,661,226,648    1,661,173,805        52,843            0
 72 327,136,002,907    1,320,812,376    1,320,775,719        36,657            0
 73 327,136,533,387        8,543,760        8,531,809        11,951            0
 74 327,137,076,493        8,543,856        8,531,825        12,031            0
 75 327,137,613,732        8,543,856        8,531,825        12,031            0
 76 327,138,154,123        8,543,856        8,531,825        12,031            0
 77 327,138,673,343        8,543,856        8,531,825        12,031            0
99.86% (8,531,825B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->98.21% (8,391,248B) 0xA051F04: MPIDI_CH3I_Seg_commit (in /gpfs4l/smplocal/intel/impi/4.1.1.036/intel64/lib/libmpi.so.4.1)
| ->98.21% (8,391,248B) 0xA05E0D2: MPID_nem_init_ckpt (in /gpfs4l/smplocal/intel/impi/4.1.1.036/intel64/lib/libmpi.so.4.1)
|   ->98.21% (8,391,248B) 0xA05FCA5: MPID_nem_init (in /gpfs4l/smplocal/intel/impi/4.1.1.036/intel64/lib/libmpi.so.4.1)
|     ->98.21% (8,391,248B) 0x9F1106E: MPIDI_CH3_Init (in /gpfs4l/smplocal/intel/impi/4.1.1.036/intel64/lib/libmpi.so.4.1)
|       ->98.21% (8,391,248B) 0xA04EBAB: MPID_Init (in /gpfs4l/smplocal/intel/impi/4.1.1.036/intel64/lib/libmpi.so.4.1)
|         ->98.21% (8,391,248B) 0xA02538D: MPIR_Init_thread (in /gpfs4l/smplocal/intel/impi/4.1.1.036/intel64/lib/libmpi.so.4.1)
|           ->98.21% (8,391,248B) 0xA024D98: PMPI_Init_thread (in /gpfs4l/smplocal/intel/impi/4.1.1.036/intel64/lib/libmpi.so.4.1)
|             ->98.21% (8,391,248B) 0x4F1D607: PetscInitialize (in /gpfs4l/smplocal/pub/PETSc/3.6.2/ada-real/lib/libpetsc.so.3.6.2)
|               ->98.21% (8,391,248B) 0x404761: main (main_petsc36.cc:2242)
|                 
->01.65% (140,577B) in 80 places, all below massif's threshold (01.00%)
  
--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 78 327,139,191,175        8,543,832        8,531,817        12,015            0
 79 327,139,714,049        8,543,856        8,531,825        12,031            0
 80 327,140,232,002        8,543,832        8,531,817        12,015            0
 81 327,140,755,852           49,600           47,540         2,060            0
-------------- next part --------------
--------------------------------------------------------------------------------
Command:            ./ex42 -mx 48 -my 48 -mz 48 -stokes_ksp_monitor_blocks 0 -model 0 -levels 3
Massif arguments:   (none)
ms_print arguments: massif.out.47240
--------------------------------------------------------------------------------


    GB
1.674^                                                                    ::: 
     |                                                 :::::::@:::::::::##::: 
     |                                 :::::::@:::::::::: ::::@: : :::::# ::: 
     |                             ::::::::: :@::::: :::: ::::@: : :::::# ::: 
     |                          :::::: ::::: :@::::: :::: ::::@: : :::::# ::: 
     |                        ::: :::: ::::: :@::::: :::: ::::@: : :::::# ::: 
     |   :::::::::::::::::::::::: :::: ::::: :@::::: :::: ::::@: : :::::# ::: 
     |   :  :        :       :::: :::: ::::: :@::::: :::: ::::@: : :::::# ::: 
     |   :  :        :       :::: :::: ::::: :@::::: :::: ::::@: : :::::# ::: 
     |   :  :        :       :::: :::: ::::: :@::::: :::: ::::@: : :::::# ::: 
     |   :  :        :       :::: :::: ::::: :@::::: :::: ::::@: : :::::# ::: 
     |   :  :        :       :::: :::: ::::: :@::::: :::: ::::@: : :::::# ::: 
     |   :  :        :       :::: :::: ::::: :@::::: :::: ::::@: : :::::# ::: 
     |::::  :        :       :::: :::: ::::: :@::::: :::: ::::@: : :::::# ::: 
     |:  :  :        :       :::: :::: ::::: :@::::: :::: ::::@: : :::::# ::: 
     |:  :  :        :       :::: :::: ::::: :@::::: :::: ::::@: : :::::# ::: 
     |:  :  :        :       :::: :::: ::::: :@::::: :::: ::::@: : :::::# ::: 
     |:  :  :        :       :::: :::: ::::: :@::::: :::: ::::@: : :::::# ::: 
     |:  :  :        :       :::: :::: ::::: :@::::: :::: ::::@: : :::::# ::: 
     |:  :  :        :       :::: :::: ::::: :@::::: :::: ::::@: : :::::# ::: 
   0 +----------------------------------------------------------------------->Ti
     0                                                                   1.100

Number of snapshots: 75
 Detailed snapshots: [22, 36, 45 (peak), 54, 64, 74]

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
  0              0                0                0             0            0
  1  4,980,608,775      661,628,576      661,566,530        62,046            0
  2 53,659,329,974      661,628,576      661,566,530        62,046            0
  3 58,343,759,694    1,258,402,064    1,258,334,906        67,158            0
  4 107,022,473,279    1,258,386,616    1,258,319,478        67,138            0
  5 107,808,983,916    1,269,736,560    1,269,669,042        67,518            0
  6 262,297,899,894    1,269,712,584    1,269,645,086        67,498            0
  7 262,297,915,219    1,269,736,560    1,269,669,042        67,518            0
  8 395,956,080,886    1,269,712,584    1,269,645,086        67,498            0
  9 410,013,623,726    1,379,141,240    1,379,051,510        89,730            0
 10 436,485,513,646    1,379,894,488    1,379,804,718        89,770            0
 11 444,589,783,484    1,455,207,200    1,455,111,954        95,246            0
 12 471,061,673,275    1,455,960,448    1,455,865,162        95,286            0
 13 490,475,678,488    1,593,525,776    1,593,399,454       126,322            0
 14 513,251,901,897    1,608,152,744    1,608,018,094       134,650            0
 15 531,294,375,247    1,608,155,584    1,608,020,914       134,670            0
 16 558,853,194,437    1,645,938,808    1,645,802,718       136,090            0
 17 576,440,011,933    1,645,938,808    1,645,802,718       136,090            0
 18 600,336,201,777    1,645,938,808    1,645,802,718       136,090            0
 19 614,937,032,875    1,645,938,808    1,645,802,718       136,090            0
 20 637,484,776,453    1,645,938,808    1,645,802,718       136,090            0
 21 659,407,063,651    1,645,938,808    1,645,802,718       136,090            0
 22 676,265,624,554    1,668,604,104    1,668,467,230       136,874            0
99.99% (1,668,467,230B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->99.73% (1,664,055,560B) 0x4FFCEA1: PetscMallocAlign (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
| ->99.73% (1,664,055,560B) 0x50006E8: PetscTrMallocDefault (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   ->99.62% (1,662,324,180B) 0x4FFE767: PetscMallocA (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | ->70.82% (1,181,786,648B) 0x583AC5B: MatSeqAIJSetPreallocation_SeqAIJ (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | ->70.73% (1,180,265,248B) 0x5839FF5: MatSeqAIJSetPreallocation (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | ->70.38% (1,174,446,424B) 0x5B6327B: DMCreateMatrix_DA_3d_MPIAIJ (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | | ->70.38% (1,174,446,424B) 0x5B5951F: DMCreateMatrix_DA (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | |   ->70.38% (1,174,446,424B) 0x5EBC894: DMCreateMatrix (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | |     ->35.19% (587,223,212B) 0x417497: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2026)
|   | | | |     | ->35.19% (587,223,212B) 0x419ADD: main (main_petsc310.cc:2254)
|   | | | |     |   
|   | | | |     ->35.19% (587,223,212B) 0x41750B: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2027)
|   | | | |       ->35.19% (587,223,212B) 0x419ADD: main (main_petsc310.cc:2254)
|   | | | |         
|   | | | ->00.35% (5,818,824B) in 1+ places, all below ms_print's threshold (01.00%)
|   | | | 
|   | | ->00.09% (1,521,400B) in 1+ places, all below ms_print's threshold (01.00%)
|   | | 
|   | ->10.84% (180,804,084B) 0x52A9E6E: VecCreate_Seq (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | ->10.84% (180,804,084B) 0x52EC6A1: VecSetType (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   ->05.81% (96,905,208B) 0x52C2880: VecCreate_Standard (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   | ->05.81% (96,905,208B) 0x52EC6A1: VecSetType (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   ->05.58% (93,138,836B) 0x5B0C608: DMCreateGlobalVector_DA (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   | ->05.58% (93,138,836B) 0x5EB8BE6: DMCreateGlobalVector (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   ->04.74% (79,093,812B) 0x5B0BD6D: VecDuplicate_MPI_DA (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   | ->04.74% (79,093,812B) 0x52DBF2E: VecDuplicate (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |   ->04.74% (79,093,812B) 0x52E058A: VecDuplicateVecs_Default (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |     ->04.74% (79,093,812B) 0x52DCD2D: VecDuplicateVecs (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |       ->04.74% (79,093,812B) 0x635A9B1: KSPCreateVecs (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |         ->03.61% (60,261,952B) 0x62C4F9F: KSPGMRESGetNewVectors (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |         | ->03.61% (60,261,952B) 0x62C19CD: KSPGMRESCycle (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |         |   ->03.61% (60,261,952B) 0x62C2540: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |         |     ->03.61% (60,261,952B) 0x633504F: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |         |       ->03.61% (60,261,952B) 0x4184F1: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2130)
|   | |   |   |   |         |         ->03.61% (60,261,952B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |   |   |   |         |           
|   | |   |   |   |         ->01.13% (18,831,860B) 0x62C0C8D: KSPSetUp_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |           ->01.13% (18,831,860B) 0x633021B: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |             ->01.13% (18,831,860B) 0x633452D: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |               ->01.13% (18,831,860B) 0x4184F1: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2130)
|   | |   |   |   |                 ->01.13% (18,831,860B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |   |   |   |                   
|   | |   |   |   ->00.84% (14,045,024B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |   |   |   
|   | |   |   ->00.23% (3,766,372B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |   |   
|   | |   ->05.03% (83,898,876B) 0x529E375: VecDuplicate_Seq (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |     ->05.03% (83,898,876B) 0x52DBF2E: VecDuplicate (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       ->04.96% (82,814,820B) 0x52E058A: VecDuplicateVecs_Default (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       | ->04.96% (82,814,820B) 0x52DCD2D: VecDuplicateVecs (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |   ->04.96% (82,814,820B) 0x635A9B1: KSPCreateVecs (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     ->02.60% (43,434,400B) 0x62C4F9F: KSPGMRESGetNewVectors (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     | ->02.60% (43,434,400B) 0x62C19CD: KSPGMRESCycle (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |   ->02.60% (43,434,400B) 0x62C2540: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     ->02.60% (43,434,400B) 0x633504F: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |       ->02.60% (43,434,400B) 0x62A6082: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |       | ->02.60% (43,434,400B) 0x633504F: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |       |   ->02.60% (43,434,400B) 0x61990DE: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |       |     ->02.30% (38,382,800B) 0x619E2A5: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |       |     | ->02.30% (38,382,800B) 0x61E665F: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |       |     |   ->02.26% (37,663,720B) 0x635E4EB: KSP_PCApply (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |       |     |   | ->02.26% (37,663,720B) 0x635F3D2: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |       |     |   |   ->02.26% (37,663,720B) 0x62C24DD: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |       |     |   |     ->02.26% (37,663,720B) 0x633504F: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |       |     |   |       ->02.26% (37,663,720B) 0x4184F1: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2130)
|   | |       |     |       |     |   |         ->02.26% (37,663,720B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |       |     |       |     |   |           
|   | |       |     |       |     |   ->00.04% (719,080B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |       |     |       |     |   
|   | |       |     |       |     ->00.30% (5,051,600B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |       |     |       |     
|   | |       |     |       ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |       |     |       
|   | |       |     ->01.30% (21,717,200B) 0x62C0C8D: KSPSetUp_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     | ->01.30% (21,717,200B) 0x633021B: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |   ->01.30% (21,717,200B) 0x633452D: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     ->01.30% (21,717,200B) 0x62A6082: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     | ->01.30% (21,717,200B) 0x633504F: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     |   ->01.30% (21,717,200B) 0x61990DE: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     |     ->01.15% (19,191,400B) 0x619E2A5: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     |     | ->01.15% (19,191,400B) 0x61E665F: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     |     |   ->01.13% (18,831,860B) 0x635E4EB: KSP_PCApply (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     |     |   | ->01.13% (18,831,860B) 0x635F3D2: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     |     |   |   ->01.13% (18,831,860B) 0x62C24DD: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     |     |   |     ->01.13% (18,831,860B) 0x633504F: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     |     |   |       ->01.13% (18,831,860B) 0x4184F1: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2130)
|   | |       |     |     |     |   |         ->01.13% (18,831,860B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |       |     |     |     |   |           
|   | |       |     |     |     |   ->00.02% (359,540B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |       |     |     |     |   
|   | |       |     |     |     ->00.15% (2,525,800B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |       |     |     |     
|   | |       |     |     ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |       |     |     
|   | |       |     ->01.06% (17,663,220B) in 3 places, all below massif's threshold (01.00%)
|   | |       |       
|   | |       ->00.06% (1,084,056B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |       
|   | ->06.75% (112,562,000B) 0x56C4881: MatPtAPSymbolic_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | ->06.75% (112,562,000B) 0x56C5A3B: MatPtAP_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   ->06.75% (112,562,000B) 0x541F3A9: MatPtAP (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |     ->06.75% (112,562,000B) 0x5433AEC: MatGalerkin (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       ->03.37% (56,281,000B) 0x61A23A0: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       | ->03.37% (56,281,000B) 0x61ED10C: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |   ->03.37% (56,281,000B) 0x6330A89: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     ->03.37% (56,281,000B) 0x633452D: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |       ->03.37% (56,281,000B) 0x4184F1: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2130)
|   | |       |         ->03.37% (56,281,000B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |       |           
|   | |       ->03.37% (56,281,000B) 0x61A244E: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |         ->03.37% (56,281,000B) 0x61ED10C: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |           ->03.37% (56,281,000B) 0x6330A89: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |             ->03.37% (56,281,000B) 0x633452D: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |               ->03.37% (56,281,000B) 0x4184F1: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2130)
|   | |                 ->03.37% (56,281,000B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |                   
|   | ->03.39% (56,624,708B) 0x403FE0: CellPropertiesCreate(_p_DM*, _p_CellProperties**) (main_petsc310.cc:75)
|   | | ->03.39% (56,624,708B) 0x415FE6: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:1858)
|   | |   ->03.39% (56,624,708B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |     
|   | ->03.37% (56,284,240B) 0x56C46D6: MatPtAPSymbolic_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | ->03.37% (56,284,240B) 0x56C5A3B: MatPtAP_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   ->03.37% (56,284,240B) 0x541F3A9: MatPtAP (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |     ->03.37% (56,284,240B) 0x5433AEC: MatGalerkin (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       ->01.69% (28,142,120B) 0x61A23A0: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       | ->01.69% (28,142,120B) 0x61ED10C: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |   ->01.69% (28,142,120B) 0x6330A89: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     ->01.69% (28,142,120B) 0x633452D: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |       ->01.69% (28,142,120B) 0x4184F1: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2130)
|   | |       |         ->01.69% (28,142,120B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |       |           
|   | |       ->01.69% (28,142,120B) 0x61A244E: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |         ->01.69% (28,142,120B) 0x61ED10C: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |           ->01.69% (28,142,120B) 0x6330A89: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |             ->01.69% (28,142,120B) 0x633452D: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |               ->01.69% (28,142,120B) 0x4184F1: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2130)
|   | |                 ->01.69% (28,142,120B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |                   
|   | ->02.34% (39,052,464B) 0x58AA01F: MatSOR_SeqAIJ_Inode (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | ->02.34% (39,052,464B) 0x53E6A04: MatSOR (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   ->02.34% (39,052,464B) 0x60260A2: PCApply_SOR (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |     ->02.34% (39,052,464B) 0x61E665F: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       ->02.34% (39,052,464B) 0x635E4EB: KSP_PCApply (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |         ->02.34% (39,052,464B) 0x635F3D2: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |           ->02.34% (39,052,464B) 0x62C24DD: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |             ->02.34% (39,052,464B) 0x633504F: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |               ->02.34% (39,052,464B) 0x62A6082: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |                 ->02.34% (39,052,464B) 0x633504F: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |                   ->02.34% (39,052,464B) 0x61990DE: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |                     ->02.07% (34,525,272B) 0x619E2A5: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |                     | ->02.07% (34,525,272B) 0x61E665F: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |                     |   ->02.03% (33,887,724B) 0x635E4EB: KSP_PCApply (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |                     |   | ->02.03% (33,887,724B) 0x635F3D2: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |                     |   |   ->02.03% (33,887,724B) 0x62C24DD: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |                     |   |     ->02.03% (33,887,724B) 0x633504F: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |                     |   |       ->02.03% (33,887,724B) 0x4184F1: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2130)
|   | |                     |   |         ->02.03% (33,887,724B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |                     |   |           
|   | |                     |   ->00.04% (637,548B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |                     |   
|   | |                     ->00.27% (4,527,192B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |                     
|   | ->02.11% (35,210,036B) in 189 places, all below massif's threshold (01.00%)
|   |   
|   ->00.10% (1,731,380B) in 1+ places, all below ms_print's threshold (01.00%)
|   
->00.26% (4,411,670B) in 1+ places, all below ms_print's threshold (01.00%)

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 23 688,869,472,856    1,683,722,032    1,683,584,522       137,510            0
 24 706,846,779,285    1,683,722,032    1,683,584,522       137,510            0
 25 734,772,531,658    1,683,722,032    1,683,584,522       137,510            0
 26 753,089,886,037    1,683,722,032    1,683,584,522       137,510            0
 27 770,117,739,409    1,683,722,032    1,683,584,522       137,510            0
 28 794,894,660,656    1,683,722,032    1,683,584,522       137,510            0
 29 808,203,400,071    1,683,722,032    1,683,584,522       137,510            0
 30 826,475,260,753    1,721,505,256    1,721,366,326       138,930            0
 31 851,341,932,673    1,721,505,256    1,721,366,326       138,930            0
 32 878,754,667,235    1,721,505,256    1,721,366,326       138,930            0
 33 892,704,562,674    1,721,505,256    1,721,366,326       138,930            0
 34 907,360,934,944    1,721,505,256    1,721,366,326       138,930            0
 35 934,847,264,351    1,721,505,256    1,721,366,326       138,930            0
 36 955,723,849,888    1,744,170,552    1,744,030,838       139,714            0
99.99% (1,744,030,838B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->99.74% (1,739,619,168B) 0x4FFCEA1: PetscMallocAlign (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
| ->99.74% (1,739,619,168B) 0x50006E8: PetscTrMallocDefault (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   ->99.64% (1,737,887,788B) 0x4FFE767: PetscMallocA (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | ->67.76% (1,181,786,648B) 0x583AC5B: MatSeqAIJSetPreallocation_SeqAIJ (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | ->67.67% (1,180,265,248B) 0x5839FF5: MatSeqAIJSetPreallocation (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | ->67.34% (1,174,446,424B) 0x5B6327B: DMCreateMatrix_DA_3d_MPIAIJ (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | | ->67.34% (1,174,446,424B) 0x5B5951F: DMCreateMatrix_DA (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | |   ->67.34% (1,174,446,424B) 0x5EBC894: DMCreateMatrix (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | |     ->33.67% (587,223,212B) 0x417497: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2026)
|   | | | |     | ->33.67% (587,223,212B) 0x419ADD: main (main_petsc310.cc:2254)
|   | | | |     |   
|   | | | |     ->33.67% (587,223,212B) 0x41750B: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2027)
|   | | | |       ->33.67% (587,223,212B) 0x419ADD: main (main_petsc310.cc:2254)
|   | | | |         
|   | | | ->00.33% (5,818,824B) in 1+ places, all below ms_print's threshold (01.00%)
|   | | | 
|   | | ->00.09% (1,521,400B) in 1+ places, all below ms_print's threshold (01.00%)
|   | | 
|   | ->14.69% (256,131,524B) 0x52A9E6E: VecCreate_Seq (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | ->14.69% (256,131,524B) 0x52EC6A1: VecSetType (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   ->09.87% (172,232,648B) 0x52C2880: VecCreate_Standard (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   | ->09.87% (172,232,648B) 0x52EC6A1: VecSetType (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   ->09.66% (168,466,276B) 0x5B0C608: DMCreateGlobalVector_DA (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   | ->09.66% (168,466,276B) 0x5EB8BE6: DMCreateGlobalVector (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   ->08.85% (154,421,252B) 0x5B0BD6D: VecDuplicate_MPI_DA (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   | ->08.85% (154,421,252B) 0x52DBF2E: VecDuplicate (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |   ->08.85% (154,421,252B) 0x52E058A: VecDuplicateVecs_Default (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |     ->08.85% (154,421,252B) 0x52DCD2D: VecDuplicateVecs (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |       ->08.85% (154,421,252B) 0x635A9B1: KSPCreateVecs (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |         ->07.77% (135,589,392B) 0x62C4F9F: KSPGMRESGetNewVectors (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |         | ->07.77% (135,589,392B) 0x62C19CD: KSPGMRESCycle (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |         |   ->07.77% (135,589,392B) 0x62C2540: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |         |     ->07.77% (135,589,392B) 0x633504F: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |         |       ->07.77% (135,589,392B) 0x4184F1: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2130)
|   | |   |   |   |         |         ->07.77% (135,589,392B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |   |   |   |         |           
|   | |   |   |   |         ->01.08% (18,831,860B) 0x62C0C8D: KSPSetUp_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |           ->01.08% (18,831,860B) 0x633021B: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |             ->01.08% (18,831,860B) 0x633452D: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |               ->01.08% (18,831,860B) 0x4184F1: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2130)
|   | |   |   |   |                 ->01.08% (18,831,860B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |   |   |   |                   
|   | |   |   |   ->00.81% (14,045,024B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |   |   |   
|   | |   |   ->00.22% (3,766,372B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |   |   
|   | |   ->04.81% (83,898,876B) 0x529E375: VecDuplicate_Seq (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |     ->04.81% (83,898,876B) 0x52DBF2E: VecDuplicate (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       ->04.75% (82,814,820B) 0x52E058A: VecDuplicateVecs_Default (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       | ->04.75% (82,814,820B) 0x52DCD2D: VecDuplicateVecs (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |   ->04.75% (82,814,820B) 0x635A9B1: KSPCreateVecs (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     ->02.49% (43,434,400B) 0x62C4F9F: KSPGMRESGetNewVectors (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     | ->02.49% (43,434,400B) 0x62C19CD: KSPGMRESCycle (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |   ->02.49% (43,434,400B) 0x62C2540: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     ->02.49% (43,434,400B) 0x633504F: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |       ->02.49% (43,434,400B) 0x62A6082: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |       | ->02.49% (43,434,400B) 0x633504F: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |       |   ->02.49% (43,434,400B) 0x61990DE: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |       |     ->02.20% (38,382,800B) 0x619E2A5: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |       |     | ->02.20% (38,382,800B) 0x61E665F: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |       |     |   ->02.16% (37,663,720B) 0x635E4EB: KSP_PCApply (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |       |     |   | ->02.16% (37,663,720B) 0x635F3D2: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |       |     |   |   ->02.16% (37,663,720B) 0x62C24DD: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |       |     |   |     ->02.16% (37,663,720B) 0x633504F: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |       |     |   |       ->02.16% (37,663,720B) 0x4184F1: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2130)
|   | |       |     |       |     |   |         ->02.16% (37,663,720B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |       |     |       |     |   |           
|   | |       |     |       |     |   ->00.04% (719,080B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |       |     |       |     |   
|   | |       |     |       |     ->00.29% (5,051,600B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |       |     |       |     
|   | |       |     |       ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |       |     |       
|   | |       |     ->01.25% (21,717,200B) 0x62C0C8D: KSPSetUp_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     | ->01.25% (21,717,200B) 0x633021B: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |   ->01.25% (21,717,200B) 0x633452D: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     ->01.25% (21,717,200B) 0x62A6082: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     | ->01.25% (21,717,200B) 0x633504F: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     |   ->01.25% (21,717,200B) 0x61990DE: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     |     ->01.10% (19,191,400B) 0x619E2A5: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     |     | ->01.10% (19,191,400B) 0x61E665F: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     |     |   ->01.08% (18,831,860B) 0x635E4EB: KSP_PCApply (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     |     |   | ->01.08% (18,831,860B) 0x635F3D2: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     |     |   |   ->01.08% (18,831,860B) 0x62C24DD: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     |     |   |     ->01.08% (18,831,860B) 0x633504F: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     |     |   |       ->01.08% (18,831,860B) 0x4184F1: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2130)
|   | |       |     |     |     |   |         ->01.08% (18,831,860B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |       |     |     |     |   |           
|   | |       |     |     |     |   ->00.02% (359,540B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |       |     |     |     |   
|   | |       |     |     |     ->00.14% (2,525,800B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |       |     |     |     
|   | |       |     |     ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |       |     |     
|   | |       |     ->01.01% (17,663,220B) in 3 places, all below massif's threshold (01.00%)
|   | |       |       
|   | |       ->00.06% (1,084,056B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |       
|   | ->06.45% (112,562,000B) 0x56C4881: MatPtAPSymbolic_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | ->06.45% (112,562,000B) 0x56C5A3B: MatPtAP_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   ->06.45% (112,562,000B) 0x541F3A9: MatPtAP (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |     ->06.45% (112,562,000B) 0x5433AEC: MatGalerkin (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       ->03.23% (56,281,000B) 0x61A23A0: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       | ->03.23% (56,281,000B) 0x61ED10C: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |   ->03.23% (56,281,000B) 0x6330A89: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     ->03.23% (56,281,000B) 0x633452D: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |       ->03.23% (56,281,000B) 0x4184F1: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2130)
|   | |       |         ->03.23% (56,281,000B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |       |           
|   | |       ->03.23% (56,281,000B) 0x61A244E: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |         ->03.23% (56,281,000B) 0x61ED10C: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |           ->03.23% (56,281,000B) 0x6330A89: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |             ->03.23% (56,281,000B) 0x633452D: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |               ->03.23% (56,281,000B) 0x4184F1: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2130)
|   | |                 ->03.23% (56,281,000B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |                   
|   | ->03.25% (56,624,708B) 0x403FE0: CellPropertiesCreate(_p_DM*, _p_CellProperties**) (main_petsc310.cc:75)
|   | | ->03.25% (56,624,708B) 0x415FE6: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:1858)
|   | |   ->03.25% (56,624,708B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |     
|   | ->03.23% (56,284,240B) 0x56C46D6: MatPtAPSymbolic_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | ->03.23% (56,284,240B) 0x56C5A3B: MatPtAP_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   ->03.23% (56,284,240B) 0x541F3A9: MatPtAP (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |     ->03.23% (56,284,240B) 0x5433AEC: MatGalerkin (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       ->01.61% (28,142,120B) 0x61A23A0: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       | ->01.61% (28,142,120B) 0x61ED10C: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |   ->01.61% (28,142,120B) 0x6330A89: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     ->01.61% (28,142,120B) 0x633452D: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |       ->01.61% (28,142,120B) 0x4184F1: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2130)
|   | |       |         ->01.61% (28,142,120B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |       |           
|   | |       ->01.61% (28,142,120B) 0x61A244E: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |         ->01.61% (28,142,120B) 0x61ED10C: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |           ->01.61% (28,142,120B) 0x6330A89: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |             ->01.61% (28,142,120B) 0x633452D: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |               ->01.61% (28,142,120B) 0x4184F1: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2130)
|   | |                 ->01.61% (28,142,120B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |                   
|   | ->02.24% (39,052,464B) 0x58AA01F: MatSOR_SeqAIJ_Inode (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | ->02.24% (39,052,464B) 0x53E6A04: MatSOR (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   ->02.24% (39,052,464B) 0x60260A2: PCApply_SOR (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |     ->02.24% (39,052,464B) 0x61E665F: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       ->02.24% (39,052,464B) 0x635E4EB: KSP_PCApply (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |         ->02.24% (39,052,464B) 0x635F3D2: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |           ->02.24% (39,052,464B) 0x62C24DD: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |             ->02.24% (39,052,464B) 0x633504F: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |               ->02.24% (39,052,464B) 0x62A6082: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |                 ->02.24% (39,052,464B) 0x633504F: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |                   ->02.24% (39,052,464B) 0x61990DE: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |                     ->01.98% (34,525,272B) 0x619E2A5: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |                     | ->01.98% (34,525,272B) 0x61E665F: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |                     |   ->01.94% (33,887,724B) 0x635E4EB: KSP_PCApply (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |                     |   | ->01.94% (33,887,724B) 0x635F3D2: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |                     |   |   ->01.94% (33,887,724B) 0x62C24DD: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |                     |   |     ->01.94% (33,887,724B) 0x633504F: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |                     |   |       ->01.94% (33,887,724B) 0x4184F1: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2130)
|   | |                     |   |         ->01.94% (33,887,724B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |                     |   |           
|   | |                     |   ->00.04% (637,548B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |                     |   
|   | |                     ->00.26% (4,527,192B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |                     
|   | ->02.03% (35,446,204B) in 189 places, all below massif's threshold (01.00%)
|   |   
|   ->00.10% (1,731,380B) in 1+ places, all below ms_print's threshold (01.00%)
|   
->00.25% (4,411,670B) in 1+ places, all below ms_print's threshold (01.00%)

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 37 969,591,761,903    1,759,288,480    1,759,148,130       140,350            0
 38 991,106,632,404    1,759,288,480    1,759,148,130       140,350            0
 39 1,005,161,800,597    1,759,288,480    1,759,148,130       140,350            0
 40 1,025,190,917,071    1,759,288,480    1,759,148,130       140,350            0
 41 1,047,466,381,537    1,759,288,480    1,759,148,130       140,350            0
 42 1,061,578,439,350    1,759,288,480    1,759,148,130       140,350            0
 43 1,086,296,641,845    1,759,288,480    1,759,148,130       140,350            0
 44 1,102,529,172,628    1,759,288,480    1,759,148,130       140,350            0
 45 1,123,472,773,295    1,781,953,776    1,781,812,642       141,134            0
99.99% (1,781,812,642B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->99.74% (1,777,400,972B) 0x4FFCEA1: PetscMallocAlign (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
| ->99.74% (1,777,400,972B) 0x50006E8: PetscTrMallocDefault (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   ->99.65% (1,775,669,592B) 0x4FFE767: PetscMallocA (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | ->66.32% (1,181,786,648B) 0x583AC5B: MatSeqAIJSetPreallocation_SeqAIJ (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | ->66.23% (1,180,265,248B) 0x5839FF5: MatSeqAIJSetPreallocation (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | ->65.91% (1,174,446,424B) 0x5B6327B: DMCreateMatrix_DA_3d_MPIAIJ (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | | ->65.91% (1,174,446,424B) 0x5B5951F: DMCreateMatrix_DA (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | |   ->65.91% (1,174,446,424B) 0x5EBC894: DMCreateMatrix (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | |     ->32.95% (587,223,212B) 0x417497: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2026)
|   | | | |     | ->32.95% (587,223,212B) 0x419ADD: main (main_petsc310.cc:2254)
|   | | | |     |   
|   | | | |     ->32.95% (587,223,212B) 0x41750B: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2027)
|   | | | |       ->32.95% (587,223,212B) 0x419ADD: main (main_petsc310.cc:2254)
|   | | | |         
|   | | | ->00.33% (5,818,824B) in 1+ places, all below ms_print's threshold (01.00%)
|   | | | 
|   | | ->00.09% (1,521,400B) in 1+ places, all below ms_print's threshold (01.00%)
|   | | 
|   | ->16.49% (293,795,244B) 0x52A9E6E: VecCreate_Seq (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | ->16.49% (293,795,244B) 0x52EC6A1: VecSetType (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   ->11.78% (209,896,368B) 0x52C2880: VecCreate_Standard (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   | ->11.78% (209,896,368B) 0x52EC6A1: VecSetType (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   ->11.57% (206,129,996B) 0x5B0C608: DMCreateGlobalVector_DA (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   | ->11.57% (206,129,996B) 0x5EB8BE6: DMCreateGlobalVector (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   ->10.78% (192,084,972B) 0x5B0BD6D: VecDuplicate_MPI_DA (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   | ->10.78% (192,084,972B) 0x52DBF2E: VecDuplicate (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |   ->10.78% (192,084,972B) 0x52E058A: VecDuplicateVecs_Default (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |     ->10.78% (192,084,972B) 0x52DCD2D: VecDuplicateVecs (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |       ->10.78% (192,084,972B) 0x635A9B1: KSPCreateVecs (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |         ->09.72% (173,253,112B) 0x62C4F9F: KSPGMRESGetNewVectors (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |         | ->09.72% (173,253,112B) 0x62C19CD: KSPGMRESCycle (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |         |   ->09.72% (173,253,112B) 0x62C2540: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |         |     ->09.72% (173,253,112B) 0x633504F: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |         |       ->09.72% (173,253,112B) 0x4184F1: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2130)
|   | |   |   |   |         |         ->09.72% (173,253,112B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |   |   |   |         |           
|   | |   |   |   |         ->01.06% (18,831,860B) 0x62C0C8D: KSPSetUp_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |           ->01.06% (18,831,860B) 0x633021B: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |             ->01.06% (18,831,860B) 0x633452D: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   |   |   |               ->01.06% (18,831,860B) 0x4184F1: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2130)
|   | |   |   |   |                 ->01.06% (18,831,860B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |   |   |   |                   
|   | |   |   |   ->00.79% (14,045,024B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |   |   |   
|   | |   |   ->00.21% (3,766,372B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |   |   
|   | |   ->04.71% (83,898,876B) 0x529E375: VecDuplicate_Seq (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |     ->04.71% (83,898,876B) 0x52DBF2E: VecDuplicate (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       ->04.65% (82,814,820B) 0x52E058A: VecDuplicateVecs_Default (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       | ->04.65% (82,814,820B) 0x52DCD2D: VecDuplicateVecs (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |   ->04.65% (82,814,820B) 0x635A9B1: KSPCreateVecs (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     ->02.44% (43,434,400B) 0x62C4F9F: KSPGMRESGetNewVectors (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     | ->02.44% (43,434,400B) 0x62C19CD: KSPGMRESCycle (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |   ->02.44% (43,434,400B) 0x62C2540: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     ->02.44% (43,434,400B) 0x633504F: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |       ->02.44% (43,434,400B) 0x62A6082: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |       | ->02.44% (43,434,400B) 0x633504F: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |       |   ->02.44% (43,434,400B) 0x61990DE: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |       |     ->02.15% (38,382,800B) 0x619E2A5: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |       |     | ->02.15% (38,382,800B) 0x61E665F: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |       |     |   ->02.11% (37,663,720B) 0x635E4EB: KSP_PCApply (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |       |     |   | ->02.11% (37,663,720B) 0x635F3D2: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |       |     |   |   ->02.11% (37,663,720B) 0x62C24DD: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |       |     |   |     ->02.11% (37,663,720B) 0x633504F: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |       |     |   |       ->02.11% (37,663,720B) 0x4184F1: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2130)
|   | |       |     |       |     |   |         ->02.11% (37,663,720B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |       |     |       |     |   |           
|   | |       |     |       |     |   ->00.04% (719,080B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |       |     |       |     |   
|   | |       |     |       |     ->00.28% (5,051,600B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |       |     |       |     
|   | |       |     |       ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |       |     |       
|   | |       |     ->01.22% (21,717,200B) 0x62C0C8D: KSPSetUp_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     | ->01.22% (21,717,200B) 0x633021B: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |   ->01.22% (21,717,200B) 0x633452D: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     ->01.22% (21,717,200B) 0x62A6082: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     | ->01.22% (21,717,200B) 0x633504F: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     |   ->01.22% (21,717,200B) 0x61990DE: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     |     ->01.08% (19,191,400B) 0x619E2A5: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     |     | ->01.08% (19,191,400B) 0x61E665F: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     |     |   ->01.06% (18,831,860B) 0x635E4EB: KSP_PCApply (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     |     |   | ->01.06% (18,831,860B) 0x635F3D2: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     |     |   |   ->01.06% (18,831,860B) 0x62C24DD: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     |     |   |     ->01.06% (18,831,860B) 0x633504F: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     |     |     |   |       ->01.06% (18,831,860B) 0x4184F1: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2130)
|   | |       |     |     |     |   |         ->01.06% (18,831,860B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |       |     |     |     |   |           
|   | |       |     |     |     |   ->00.02% (359,540B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |       |     |     |     |   
|   | |       |     |     |     ->00.14% (2,525,800B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |       |     |     |     
|   | |       |     |     ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |       |     |     
|   | |       |     ->00.99% (17,663,220B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |       |     
|   | |       ->00.06% (1,084,056B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |       
|   | ->06.32% (112,562,000B) 0x56C4881: MatPtAPSymbolic_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | ->06.32% (112,562,000B) 0x56C5A3B: MatPtAP_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   ->06.32% (112,562,000B) 0x541F3A9: MatPtAP (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |     ->06.32% (112,562,000B) 0x5433AEC: MatGalerkin (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       ->03.16% (56,281,000B) 0x61A23A0: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       | ->03.16% (56,281,000B) 0x61ED10C: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |   ->03.16% (56,281,000B) 0x6330A89: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     ->03.16% (56,281,000B) 0x633452D: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |       ->03.16% (56,281,000B) 0x4184F1: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2130)
|   | |       |         ->03.16% (56,281,000B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |       |           
|   | |       ->03.16% (56,281,000B) 0x61A244E: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |         ->03.16% (56,281,000B) 0x61ED10C: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |           ->03.16% (56,281,000B) 0x6330A89: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |             ->03.16% (56,281,000B) 0x633452D: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |               ->03.16% (56,281,000B) 0x4184F1: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2130)
|   | |                 ->03.16% (56,281,000B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |                   
|   | ->03.18% (56,624,708B) 0x403FE0: CellPropertiesCreate(_p_DM*, _p_CellProperties**) (main_petsc310.cc:75)
|   | | ->03.18% (56,624,708B) 0x415FE6: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:1858)
|   | |   ->03.18% (56,624,708B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |     
|   | ->03.16% (56,284,240B) 0x56C46D6: MatPtAPSymbolic_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | ->03.16% (56,284,240B) 0x56C5A3B: MatPtAP_SeqAIJ_SeqMAIJ (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   ->03.16% (56,284,240B) 0x541F3A9: MatPtAP (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |     ->03.16% (56,284,240B) 0x5433AEC: MatGalerkin (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       ->01.58% (28,142,120B) 0x61A23A0: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       | ->01.58% (28,142,120B) 0x61ED10C: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |   ->01.58% (28,142,120B) 0x6330A89: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |     ->01.58% (28,142,120B) 0x633452D: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       |       ->01.58% (28,142,120B) 0x4184F1: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2130)
|   | |       |         ->01.58% (28,142,120B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |       |           
|   | |       ->01.58% (28,142,120B) 0x61A244E: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |         ->01.58% (28,142,120B) 0x61ED10C: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |           ->01.58% (28,142,120B) 0x6330A89: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |             ->01.58% (28,142,120B) 0x633452D: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |               ->01.58% (28,142,120B) 0x4184F1: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2130)
|   | |                 ->01.58% (28,142,120B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |                   
|   | ->02.19% (39,052,464B) 0x58AA01F: MatSOR_SeqAIJ_Inode (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | ->02.19% (39,052,464B) 0x53E6A04: MatSOR (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   ->02.19% (39,052,464B) 0x60260A2: PCApply_SOR (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |     ->02.19% (39,052,464B) 0x61E665F: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |       ->02.19% (39,052,464B) 0x635E4EB: KSP_PCApply (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |         ->02.19% (39,052,464B) 0x635F3D2: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |           ->02.19% (39,052,464B) 0x62C24DD: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |             ->02.19% (39,052,464B) 0x633504F: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |               ->02.19% (39,052,464B) 0x62A6082: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |                 ->02.19% (39,052,464B) 0x633504F: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |                   ->02.19% (39,052,464B) 0x61990DE: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |                     ->01.94% (34,525,272B) 0x619E2A5: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |                     | ->01.94% (34,525,272B) 0x61E665F: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |                     |   ->01.90% (33,887,724B) 0x635E4EB: KSP_PCApply (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |                     |   | ->01.90% (33,887,724B) 0x635F3D2: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |                     |   |   ->01.90% (33,887,724B) 0x62C24DD: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |                     |   |     ->01.90% (33,887,724B) 0x633504F: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |                     |   |       ->01.90% (33,887,724B) 0x4184F1: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2130)
|   | |                     |   |         ->01.90% (33,887,724B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |                     |   |           
|   | |                     |   ->00.04% (637,548B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |                     |   
|   | |                     ->00.25% (4,527,192B) in 1+ places, all below ms_print's threshold (01.00%)
|   | |                     
|   | ->02.00% (35,564,288B) in 189 places, all below massif's threshold (01.00%)
|   |   
|   ->00.10% (1,731,380B) in 1+ places, all below ms_print's threshold (01.00%)
|   
->00.25% (4,411,670B) in 1+ places, all below ms_print's threshold (01.00%)

--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 46 1,144,977,466,286    1,797,071,704    1,796,929,934       141,770            0
 47 1,169,482,873,650    1,797,071,704    1,796,929,934       141,770            0
 48 1,187,701,811,363    1,797,071,704    1,796,929,934       141,770            0
 49 1,209,131,205,775        5,474,208        5,419,554        54,654            0
 50 1,209,131,221,293        5,388,512        5,334,898        53,614            0
 51 1,209,131,236,638        5,281,384        5,229,070        52,314            0
 52 1,209,131,252,176        5,169,320        5,118,366        50,954            0
 53 1,209,131,267,534        5,099,704        5,049,606        50,098            0
 54 1,209,131,282,926        4,994,488        4,945,670        48,818            0
99.02% (4,945,670B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->55.11% (2,752,512B) 0xB6B4A75: MPIDI_RMA_init (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
| ->55.11% (2,752,512B) 0xB69B950: MPID_Init (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
|   ->55.11% (2,752,512B) 0xB5031EE: MPIR_Init_thread (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
|     ->55.11% (2,752,512B) 0xB5036EE: PMPI_Init_thread (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
|       ->55.11% (2,752,512B) 0x4FAB272: PetscInitialize (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|         ->55.11% (2,752,512B) 0x41983C: main (main_petsc310.cc:2241)
|           
->26.24% (1,310,720B) 0xB6B4CFD: MPIDI_RMA_init (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
| ->26.24% (1,310,720B) 0xB69B950: MPID_Init (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
|   ->26.24% (1,310,720B) 0xB5031EE: MPIR_Init_thread (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
|     ->26.24% (1,310,720B) 0xB5036EE: PMPI_Init_thread (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
|       ->26.24% (1,310,720B) 0x4FAB272: PetscInitialize (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|         ->26.24% (1,310,720B) 0x41983C: main (main_petsc310.cc:2241)
|           
->10.74% (536,488B) 0x4FFCEA1: PetscMallocAlign (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
| ->10.74% (536,488B) 0x50006E8: PetscTrMallocDefault (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   ->10.51% (524,852B) 0x4FFE767: PetscMallocA (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | ->08.51% (425,240B) 0x4FDB939: PetscStrallocpy (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | ->07.24% (361,772B) 0x4F662CD: PetscEventRegLogRegister (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | ->07.24% (361,772B) 0x4F3968E: PetscLogEventRegister (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | |   ->07.24% (361,772B) in 247 places, all below massif's threshold (01.00%)
|   | | |     
|   | | ->01.27% (63,468B) in 32 places, all below massif's threshold (01.00%)
|   | |   
|   | ->01.99% (99,612B) in 203 places, all below massif's threshold (01.00%)
|   |   
|   ->00.23% (11,636B) in 1+ places, all below ms_print's threshold (01.00%)
|   
->04.79% (239,180B) in 130 places, all below massif's threshold (01.00%)
| 
->01.12% (55,933B) 0xC2A1ED0: strdup (in /lib64/libc-2.12.so)
| ->01.12% (55,933B) in 100 places, all below massif's threshold (01.00%)
|   
->01.02% (50,837B) 0xB6E2FC3: hwloc_tma_malloc (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
  ->01.02% (50,837B) in 8 places, all below massif's threshold (01.00%)
    
--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 55 1,209,131,298,332        4,889,160        4,841,622        47,538            0
 56 1,209,131,313,702        4,784,120        4,737,862        46,258            0
 57 1,209,131,329,062        4,679,160        4,634,182        44,978            0
 58 1,209,131,344,556        4,585,648        4,541,730        43,918            0
 59 1,209,131,372,522        4,497,744        4,454,235        43,509            0
 60 1,209,131,391,226        4,483,776        4,440,702        43,074            0
 61 1,209,131,406,563        4,481,160        4,438,490        42,670            0
 62 1,209,131,422,011        4,459,456        4,420,562        38,894            0
 63 1,209,131,437,338        4,438,256        4,403,151        35,105            0
 64 1,209,131,452,709        4,416,680        4,385,319        31,361            0
99.29% (4,385,319B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->62.32% (2,752,512B) 0xB6B4A75: MPIDI_RMA_init (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
| ->62.32% (2,752,512B) 0xB69B950: MPID_Init (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
|   ->62.32% (2,752,512B) 0xB5031EE: MPIR_Init_thread (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
|     ->62.32% (2,752,512B) 0xB5036EE: PMPI_Init_thread (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
|       ->62.32% (2,752,512B) 0x4FAB272: PetscInitialize (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|         ->62.32% (2,752,512B) 0x41983C: main (main_petsc310.cc:2241)
|           
->29.68% (1,310,720B) 0xB6B4CFD: MPIDI_RMA_init (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
| ->29.68% (1,310,720B) 0xB69B950: MPID_Init (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
|   ->29.68% (1,310,720B) 0xB5031EE: MPIR_Init_thread (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
|     ->29.68% (1,310,720B) 0xB5036EE: PMPI_Init_thread (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
|       ->29.68% (1,310,720B) 0x4FAB272: PetscInitialize (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|         ->29.68% (1,310,720B) 0x41983C: main (main_petsc310.cc:2241)
|           
->06.03% (266,288B) in 132 places, all below massif's threshold (01.00%)
| 
->01.26% (55,799B) 0xC2A1ED0: strdup (in /lib64/libc-2.12.so)
  ->01.02% (45,156B) 0xB6D6187: MPIR_T_CVAR_REGISTER_impl (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
  | ->01.02% (45,156B) in 235 places, all below massif's threshold (01.00%)
  |   
  ->00.24% (10,643B) in 1+ places, all below ms_print's threshold (01.00%)
  
--------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
--------------------------------------------------------------------------------
 65 1,209,131,468,036        4,395,496        4,367,908        27,588            0
 66 1,209,131,483,425        4,374,144        4,350,321        23,823            0
 67 1,209,131,498,780        4,352,664        4,332,601        20,063            0
 68 1,209,131,515,035          266,744          249,681        17,063            0
 69 1,209,131,530,396          211,640          200,353        11,287            0
 70 1,209,131,545,804          143,072          136,226         6,846            0
 71 1,209,131,561,148          133,480          128,378         5,102            0
 72 1,209,131,576,478          122,528          119,356         3,172            0
 73 1,209,131,659,641          107,416          105,622         1,794            0
 74 1,209,131,709,690           88,184           86,649         1,535            0
98.26% (86,649B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->49.90% (44,008B) 0x4FFCEA1: PetscMallocAlign (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
| ->49.90% (44,008B) 0x50006E8: PetscTrMallocDefault (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   ->49.90% (44,008B) 0x4FFE767: PetscMallocA (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | ->22.19% (19,568B) 0x4FDB939: PetscStrallocpy (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | ->11.13% (9,816B) 0x500DF1D: PetscFunctionListAdd_Private (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | ->11.13% (9,816B) 0x4F8255C: PetscObjectComposeFunction_Petsc (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | | ->11.13% (9,816B) 0x4F83ACF: PetscObjectComposeFunction_Private (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | |   ->03.71% (3,272B) 0x50D123D: PetscViewerCreate_ASCII (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | |   | ->03.71% (3,272B) 0x50E29F2: PetscViewerSetType (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | |   |   ->03.71% (3,272B) 0x50D4E82: PetscViewerASCIIOpen (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | |   |     ->01.86% (1,636B) 0x418A0C: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2169)
|   | | | |   |     | ->01.86% (1,636B) 0x419ADD: main (main_petsc310.cc:2254)
|   | | | |   |     |   
|   | | | |   |     ->01.86% (1,636B) 0x418AE3: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2178)
|   | | | |   |       ->01.86% (1,636B) 0x419ADD: main (main_petsc310.cc:2254)
|   | | | |   |         
|   | | | |   ->03.71% (3,272B) 0x50D12A7: PetscViewerCreate_ASCII (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | |   | ->03.71% (3,272B) 0x50E29F2: PetscViewerSetType (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | |   |   ->03.71% (3,272B) 0x50D4E82: PetscViewerASCIIOpen (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | |   |     ->01.86% (1,636B) 0x418A0C: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2169)
|   | | | |   |     | ->01.86% (1,636B) 0x419ADD: main (main_petsc310.cc:2254)
|   | | | |   |     |   
|   | | | |   |     ->01.86% (1,636B) 0x418AE3: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2178)
|   | | | |   |       ->01.86% (1,636B) 0x419ADD: main (main_petsc310.cc:2254)
|   | | | |   |         
|   | | | |   ->03.71% (3,272B) 0x50D1311: PetscViewerCreate_ASCII (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | |   | ->03.71% (3,272B) 0x50E29F2: PetscViewerSetType (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | |   |   ->03.71% (3,272B) 0x50D4E82: PetscViewerASCIIOpen (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | |   |     ->01.86% (1,636B) 0x418A0C: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2169)
|   | | | |   |     | ->01.86% (1,636B) 0x419ADD: main (main_petsc310.cc:2254)
|   | | | |   |     |   
|   | | | |   |     ->01.86% (1,636B) 0x418AE3: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2178)
|   | | | |   |       ->01.86% (1,636B) 0x419ADD: main (main_petsc310.cc:2254)
|   | | | |   |         
|   | | | |   ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
|   | | | |   
|   | | | ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
|   | | | 
|   | | ->03.71% (3,272B) 0x500DBCE: PetscFunctionListAdd_Private (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | ->03.71% (3,272B) 0x4F8255C: PetscObjectComposeFunction_Petsc (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | | ->03.71% (3,272B) 0x4F83ACF: PetscObjectComposeFunction_Private (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | |   ->03.71% (3,272B) 0x50D11D3: PetscViewerCreate_ASCII (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | |   | ->03.71% (3,272B) 0x50E29F2: PetscViewerSetType (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | |   |   ->03.71% (3,272B) 0x50D4E82: PetscViewerASCIIOpen (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | |   |     ->01.86% (1,636B) 0x418A0C: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2169)
|   | | | |   |     | ->01.86% (1,636B) 0x419ADD: main (main_petsc310.cc:2254)
|   | | | |   |     |   
|   | | | |   |     ->01.86% (1,636B) 0x418AE3: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2178)
|   | | | |   |       ->01.86% (1,636B) 0x419ADD: main (main_petsc310.cc:2254)
|   | | | |   |         
|   | | | |   ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
|   | | | |   
|   | | | ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
|   | | | 
|   | | ->03.67% (3,240B) 0x4F74319: PetscObjectChangeTypeName (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | ->03.67% (3,240B) 0x50E2995: PetscViewerSetType (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | | ->03.67% (3,240B) 0x50D4E82: PetscViewerASCIIOpen (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | |   ->01.84% (1,620B) 0x418A0C: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2169)
|   | | | |   | ->01.84% (1,620B) 0x419ADD: main (main_petsc310.cc:2254)
|   | | | |   |   
|   | | | |   ->01.84% (1,620B) 0x418AE3: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2178)
|   | | | |     ->01.84% (1,620B) 0x419ADD: main (main_petsc310.cc:2254)
|   | | | |       
|   | | | ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
|   | | | 
|   | | ->03.67% (3,240B) 0x50CFAFF: PetscViewerFileSetName_ASCII (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | ->03.67% (3,240B) 0x50CED56: PetscViewerFileSetName (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | |   ->03.67% (3,240B) 0x50D4EEF: PetscViewerASCIIOpen (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | |     ->01.84% (1,620B) 0x418A0C: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2169)
|   | | |     | ->01.84% (1,620B) 0x419ADD: main (main_petsc310.cc:2254)
|   | | |     |   
|   | | |     ->01.84% (1,620B) 0x418AE3: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2178)
|   | | |       ->01.84% (1,620B) 0x419ADD: main (main_petsc310.cc:2254)
|   | | |         
|   | | ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
|   | | 
|   | ->11.13% (9,816B) 0x500DEB6: PetscFunctionListAdd_Private (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | ->11.13% (9,816B) 0x4F8255C: PetscObjectComposeFunction_Petsc (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | ->11.13% (9,816B) 0x4F83ACF: PetscObjectComposeFunction_Private (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | |   ->03.71% (3,272B) 0x50D123D: PetscViewerCreate_ASCII (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | |   | ->03.71% (3,272B) 0x50E29F2: PetscViewerSetType (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | |   |   ->03.71% (3,272B) 0x50D4E82: PetscViewerASCIIOpen (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | |   |     ->01.86% (1,636B) 0x418A0C: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2169)
|   | | |   |     | ->01.86% (1,636B) 0x419ADD: main (main_petsc310.cc:2254)
|   | | |   |     |   
|   | | |   |     ->01.86% (1,636B) 0x418AE3: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2178)
|   | | |   |       ->01.86% (1,636B) 0x419ADD: main (main_petsc310.cc:2254)
|   | | |   |         
|   | | |   ->03.71% (3,272B) 0x50D12A7: PetscViewerCreate_ASCII (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | |   | ->03.71% (3,272B) 0x50E29F2: PetscViewerSetType (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | |   |   ->03.71% (3,272B) 0x50D4E82: PetscViewerASCIIOpen (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | |   |     ->01.86% (1,636B) 0x418A0C: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2169)
|   | | |   |     | ->01.86% (1,636B) 0x419ADD: main (main_petsc310.cc:2254)
|   | | |   |     |   
|   | | |   |     ->01.86% (1,636B) 0x418AE3: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2178)
|   | | |   |       ->01.86% (1,636B) 0x419ADD: main (main_petsc310.cc:2254)
|   | | |   |         
|   | | |   ->03.71% (3,272B) 0x50D1311: PetscViewerCreate_ASCII (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | |   | ->03.71% (3,272B) 0x50E29F2: PetscViewerSetType (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | |   |   ->03.71% (3,272B) 0x50D4E82: PetscViewerASCIIOpen (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | |   |     ->01.86% (1,636B) 0x418A0C: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2169)
|   | | |   |     | ->01.86% (1,636B) 0x419ADD: main (main_petsc310.cc:2254)
|   | | |   |     |   
|   | | |   |     ->01.86% (1,636B) 0x418AE3: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2178)
|   | | |   |       ->01.86% (1,636B) 0x419ADD: main (main_petsc310.cc:2254)
|   | | |   |         
|   | | |   ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
|   | | |   
|   | | ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
|   | | 
|   | ->05.34% (4,712B) 0x50E20A8: PetscViewerCreate (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | ->05.34% (4,712B) 0x50D4E1C: PetscViewerASCIIOpen (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   ->02.67% (2,356B) 0x418A0C: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2169)
|   | |   | ->02.67% (2,356B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |   |   
|   | |   ->02.67% (2,356B) 0x418AE3: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2178)
|   | |     ->02.67% (2,356B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |       
|   | ->03.86% (3,400B) 0x50D1059: PetscViewerCreate_ASCII (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | ->03.86% (3,400B) 0x50E29F2: PetscViewerSetType (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |   ->03.86% (3,400B) 0x50D4E82: PetscViewerASCIIOpen (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | |     ->01.93% (1,700B) 0x418A0C: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2169)
|   | |     | ->01.93% (1,700B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |     |   
|   | |     ->01.93% (1,700B) 0x418AE3: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2178)
|   | |       ->01.93% (1,700B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |         
|   | ->03.71% (3,272B) 0x500DB67: PetscFunctionListAdd_Private (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | ->03.71% (3,272B) 0x4F8255C: PetscObjectComposeFunction_Petsc (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | | ->03.71% (3,272B) 0x4F83ACF: PetscObjectComposeFunction_Private (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | |   ->03.71% (3,272B) 0x50D11D3: PetscViewerCreate_ASCII (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | |   | ->03.71% (3,272B) 0x50E29F2: PetscViewerSetType (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | |   |   ->03.71% (3,272B) 0x50D4E82: PetscViewerASCIIOpen (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | |   |     ->01.86% (1,636B) 0x418A0C: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2169)
|   | | |   |     | ->01.86% (1,636B) 0x419ADD: main (main_petsc310.cc:2254)
|   | | |   |     |   
|   | | |   |     ->01.86% (1,636B) 0x418AE3: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2178)
|   | | |   |       ->01.86% (1,636B) 0x419ADD: main (main_petsc310.cc:2254)
|   | | |   |         
|   | | |   ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
|   | | |   
|   | | ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
|   | | 
|   | ->03.67% (3,240B) 0x50D4F74: PetscViewerASCIIOpen (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|   | | ->01.84% (1,620B) 0x418A0C: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2169)
|   | | | ->01.84% (1,620B) 0x419ADD: main (main_petsc310.cc:2254)
|   | | |   
|   | | ->01.84% (1,620B) 0x418AE3: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2178)
|   | |   ->01.84% (1,620B) 0x419ADD: main (main_petsc310.cc:2254)
|   | |     
|   | ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
|   | 
|   ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
|   
->38.03% (33,536B) 0x8C869BB: mkl_serv_allocate (in /gpfs4l/smplocal/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64/libmkl_core.so)
| ->37.30% (32,896B) 0x90EA3BD: mkl_lapack_dgehrd (in /gpfs4l/smplocal/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64/libmkl_core.so)
| | ->37.30% (32,896B) 0x90E3F92: mkl_lapack_dgeev (in /gpfs4l/smplocal/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64/libmkl_core.so)
| |   ->37.30% (32,896B) 0x7F41809: DGEEV (in /gpfs4l/smplocal/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64/libmkl_intel_lp64.so)
| |     ->37.30% (32,896B) 0x62CE421: KSPComputeEigenvalues_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
| |       ->37.30% (32,896B) 0x632E270: KSPComputeEigenvalues (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
| |         ->37.30% (32,896B) 0x62A54C2: KSPChebyshevComputeExtremeEigenvalues_Private (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
| |           ->37.30% (32,896B) 0x62A64CB: KSPSolve_Chebyshev (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
| |             ->37.30% (32,896B) 0x633504F: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
| |               ->37.30% (32,896B) 0x61990DE: PCMGMCycle_Private (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
| |                 ->37.30% (32,896B) 0x619E2A5: PCApply_MG (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
| |                   ->37.30% (32,896B) 0x61E665F: PCApply (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
| |                     ->37.30% (32,896B) 0x635E4EB: KSP_PCApply (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
| |                       ->37.30% (32,896B) 0x635F3D2: KSPInitialResidual (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
| |                         ->37.30% (32,896B) 0x62C24DD: KSPSolve_GMRES (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
| |                           ->37.30% (32,896B) 0x633504F: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
| |                             ->37.30% (32,896B) 0x4184F1: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2130)
| |                               ->37.30% (32,896B) 0x419ADD: main (main_petsc310.cc:2254)
| |                                 
| ->00.73% (640B) in 1+ places, all below ms_print's threshold (01.00%)
| 
->02.73% (2,406B) 0x400B13D: _dl_new_object (in /lib64/ld-2.12.so)
| ->02.73% (2,406B) 0x400745C: _dl_map_object_from_fd (in /lib64/ld-2.12.so)
|   ->02.73% (2,406B) 0x4008660: _dl_map_object (in /lib64/ld-2.12.so)
|     ->02.73% (2,406B) 0x4012EF3: dl_open_worker (in /lib64/ld-2.12.so)
|       ->02.73% (2,406B) 0x400E5E4: _dl_catch_error (in /lib64/ld-2.12.so)
|         ->02.73% (2,406B) 0x4012998: _dl_open (in /lib64/ld-2.12.so)
|           ->01.39% (1,228B) 0xAFB4F64: dlopen_doit (in /lib64/libdl-2.12.so)
|           | ->01.39% (1,228B) 0x400E5E4: _dl_catch_error (in /lib64/ld-2.12.so)
|           |   ->01.39% (1,228B) 0xAFB529A: _dlerror_run (in /lib64/libdl-2.12.so)
|           |     ->01.39% (1,228B) 0xAFB4EDF: dlopen@@GLIBC_2.2.5 (in /lib64/libdl-2.12.so)
|           |       ->01.39% (1,228B) 0x8C83F19: mkl_serv_load_dll (in /gpfs4l/smplocal/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64/libmkl_core.so)
|           |         ->01.39% (1,228B) 0x8CD46DC: mkl_blas_dnrm2 (in /gpfs4l/smplocal/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64/libmkl_core.so)
|           |           ->01.39% (1,228B) 0x918D552: mkl_lapack_dlarfg (in /gpfs4l/smplocal/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64/libmkl_core.so)
|           |             ->01.39% (1,228B) 0x90FF89F: mkl_lapack_dgeqr2 (in /gpfs4l/smplocal/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64/libmkl_core.so)
|           |               ->01.39% (1,228B) 0x958109D: mkl_lapack_xdgeqrf (in /gpfs4l/smplocal/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64/libmkl_core.so)
|           |                 ->01.39% (1,228B) 0x8714BB6: mkl_lapack_dgeqrf (in /gpfs4l/smplocal/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64/libmkl_sequential.so)
|           |                   ->01.39% (1,228B) 0x7F37955: DGEQRF (in /gpfs4l/smplocal/intel/composer_xe_2013_sp1.1.106/mkl/lib/intel64/libmkl_intel_lp64.so)
|           |                     ->01.39% (1,228B) 0x60F2D40: formProl0 (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|           |                       ->01.39% (1,228B) 0x60F74C2: PCGAMGProlongator_AGG (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|           |                         ->01.39% (1,228B) 0x60DFD80: PCSetUp_GAMG (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|           |                           ->01.39% (1,228B) 0x61ED10C: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|           |                             ->01.39% (1,228B) 0x6330A89: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|           |                               ->01.39% (1,228B) 0x61A45F2: PCSetUp_MG (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|           |                                 ->01.39% (1,228B) 0x61ED10C: PCSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|           |                                   ->01.39% (1,228B) 0x6330A89: KSPSetUp (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|           |                                     ->01.39% (1,228B) 0x633452D: KSPSolve (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|           |                                       ->01.39% (1,228B) 0x4184F1: solve_stokes_3d_coupled(int, int, int) (main_petsc310.cc:2130)
|           |                                         ->01.39% (1,228B) 0x419ADD: main (main_petsc310.cc:2254)
|           |                                           
|           ->01.34% (1,178B) 0xC347EFE: do_dlopen (in /lib64/libc-2.12.so)
|             ->01.34% (1,178B) 0x400E5E4: _dl_catch_error (in /lib64/ld-2.12.so)
|               ->01.34% (1,178B) 0xC348055: __libc_dlopen_mode (in /lib64/libc-2.12.so)
|                 ->01.34% (1,178B) 0xC31D571: __nss_lookup_function (in /lib64/libc-2.12.so)
|                   ->01.34% (1,178B) 0xC31D5DA: __nss_lookup (in /lib64/libc-2.12.so)
|                     ->01.34% (1,178B) 0xC324DEE: gethostbyname_r@@GLIBC_2.2.5 (in /lib64/libc-2.12.so)
|                       ->01.34% (1,178B) 0xC324491: gethostbyname (in /lib64/libc-2.12.so)
|                         ->01.34% (1,178B) 0xB6BC049: MPIDU_CH3U_GetSockInterfaceAddr (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
|                           ->01.34% (1,178B) 0xB6BA105: MPIDI_CH3U_Get_business_card_sock (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
|                             ->01.34% (1,178B) 0xB6B92D0: MPIDI_CH3U_Init_sock (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
|                               ->01.34% (1,178B) 0xB6C3B8C: MPIDI_CH3_Init (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
|                                 ->01.34% (1,178B) 0xB69B394: MPID_Init (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
|                                   ->01.34% (1,178B) 0xB5031EE: MPIR_Init_thread (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
|                                     ->01.34% (1,178B) 0xB5036EE: PMPI_Init_thread (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
|                                       ->01.34% (1,178B) 0x4FAB272: PetscInitialize (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|                                         ->01.34% (1,178B) 0x41983C: main (main_petsc310.cc:2241)
|                                           
->02.18% (1,923B) 0xC2BF7CA: __tzfile_read (in /lib64/libc-2.12.so)
| ->02.18% (1,923B) 0xC2BE962: tzset_internal (in /lib64/libc-2.12.so)
|   ->02.18% (1,923B) 0xC2BEAC7: __tz_convert (in /lib64/libc-2.12.so)
|     ->02.18% (1,923B) 0x4F6EBAB: PetscGetDate (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|       ->02.18% (1,923B) 0x511FB42: PetscErrorPrintfInitialize (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|         ->02.18% (1,923B) 0x4FAB5F8: PetscInitialize (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|           ->02.18% (1,923B) 0x41983C: main (main_petsc310.cc:2241)
|             
->01.79% (1,582B) in 129 places, all below massif's threshold (01.00%)
| 
->01.29% (1,136B) 0xC287D39: __fopen_internal (in /lib64/libc-2.12.so)
| ->01.29% (1,136B) 0x50CFF95: PetscViewerFileSetName_ASCII (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
| | ->01.29% (1,136B) 0x50CED56: PetscViewerFileSetName (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
| |   ->01.29% (1,136B) 0x50D4EEF: PetscViewerASCIIOpen (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
| |     ->01.29% (1,136B) in 2 places, all below massif's threshold (01.00%)
| |       
| ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| 
->01.17% (1,034B) 0xC31D739: nss_parse_service_list (in /lib64/libc-2.12.so)
| ->01.17% (1,034B) 0xC31E1FF: __nss_database_lookup (in /lib64/libc-2.12.so)
|   ->01.17% (1,034B) 0xC31F356: __nss_hosts_lookup2 (in /lib64/libc-2.12.so)
|     ->01.17% (1,034B) 0xC324DEE: gethostbyname_r@@GLIBC_2.2.5 (in /lib64/libc-2.12.so)
|       ->01.17% (1,034B) 0xC324491: gethostbyname (in /lib64/libc-2.12.so)
|         ->01.17% (1,034B) 0xB6BC049: MPIDU_CH3U_GetSockInterfaceAddr (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
|           ->01.17% (1,034B) 0xB6BA105: MPIDI_CH3U_Get_business_card_sock (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
|             ->01.17% (1,034B) 0xB6B92D0: MPIDI_CH3U_Init_sock (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
|               ->01.17% (1,034B) 0xB6C3B8C: MPIDI_CH3_Init (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
|                 ->01.17% (1,034B) 0xB69B394: MPID_Init (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
|                   ->01.17% (1,034B) 0xB5031EE: MPIR_Init_thread (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
|                     ->01.17% (1,034B) 0xB5036EE: PMPI_Init_thread (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
|                       ->01.17% (1,034B) 0x4FAB272: PetscInitialize (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
|                         ->01.17% (1,034B) 0x41983C: main (main_petsc310.cc:2241)
|                           
->01.16% (1,024B) 0xC324563: gethostbyname (in /lib64/libc-2.12.so)
  ->01.16% (1,024B) 0xB6BC049: MPIDU_CH3U_GetSockInterfaceAddr (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
    ->01.16% (1,024B) 0xB6BA105: MPIDI_CH3U_Get_business_card_sock (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
      ->01.16% (1,024B) 0xB6B92D0: MPIDI_CH3U_Init_sock (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
        ->01.16% (1,024B) 0xB6C3B8C: MPIDI_CH3_Init (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
          ->01.16% (1,024B) 0xB69B394: MPID_Init (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
            ->01.16% (1,024B) 0xB5031EE: MPIR_Init_thread (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
              ->01.16% (1,024B) 0xB5036EE: PMPI_Init_thread (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libmpi.so.0.0.0)
                ->01.16% (1,024B) 0x4FAB272: PetscInitialize (in /gpfs4l/smplocal/pub/PETSc/3.10.2/real-debug-mpich-mumps-hypre/lib/libpetsc.so.3.10.2)
                  ->01.16% (1,024B) 0x41983C: main (main_petsc310.cc:2241)
                    
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 2975 bytes
Desc: Signature cryptographique S/MIME
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190308/99d1b272/attachment-0001.p7s>


More information about the petsc-users mailing list