[petsc-users] Poor weak scaling when solving successive linearsystems
Michael Becker
Michael.Becker at physik.uni-giessen.de
Mon Jun 4 11:59:40 CDT 2018
Hello again,
this took me longer than I anticipated, but here we go.
I did reruns of the cases where only half the processes per node were
used (without -log_sync):
125 procs,1st 125 procs,2nd 1000 procs,1st
1000 procs,2nd
Max Ratio Max RatioMax
RatioMax Ratio
KSPSolve 1.203E+02 1.0 1.210E+02 1.0 1.399E+02
1.1 1.365E+02 1.0
VecTDot 6.376E+00 3.7 6.551E+00 4.0 7.885E+00
2.9 7.175E+00 3.4
VecNorm 4.579E+00 7.1 5.803E+00 10.2 8.534E+00
6.9 6.026E+00 4.9
VecScale 1.070E-01 2.1 1.129E-01 2.2 1.301E-01
2.5 1.270E-01 2.4
VecCopy 1.123E-01 1.3 1.149E-01 1.3 1.301E-01
1.6 1.359E-01 1.6
VecSet 7.063E-01 1.7 6.968E-01 1.7 7.432E-01
1.8 7.425E-01 1.8
VecAXPY 1.166E+00 1.4 1.167E+00 1.4 1.221E+00
1.5 1.279E+00 1.6
VecAYPX 1.317E+00 1.6 1.290E+00 1.6 1.536E+00
1.9 1.499E+00 2.0
VecScatterBegin 6.142E+00 3.2 5.974E+00 2.8 6.448E+00
3.0 6.472E+00 2.9
VecScatterEnd 3.606E+01 4.2 3.551E+01 4.0 5.244E+01
2.7 4.995E+01 2.7
MatMult 3.561E+01 1.6 3.403E+01 1.5 3.435E+01
1.4 3.332E+01 1.4
MatMultAdd 1.124E+01 2.0 1.130E+01 2.1 2.093E+01
2.9 1.995E+01 2.7
MatMultTranspose 1.372E+01 2.5 1.388E+01 2.6 1.477E+01
2.2 1.381E+01 2.1
MatSolve 1.949E-02 0.0 1.653E-02 0.0 4.789E-02
0.0 4.466E-02 0.0
MatSOR 6.610E+01 1.3 6.673E+01 1.3 7.111E+01
1.3 7.105E+01 1.3
MatResidual 2.647E+01 1.7 2.667E+01 1.7 2.446E+01
1.4 2.467E+01 1.5
PCSetUpOnBlocks 5.266E-03 1.4 5.295E-03 1.4 5.427E-03
1.5 5.289E-03 1.4
PCApply 1.031E+02 1.0 1.035E+02 1.0 1.180E+02
1.0 1.164E+02 1.0
I also slimmed down my code and basically wrote a simple weak scaling
test (source files attached) so you can profile it yourself. I
appreciate the offer Junchao, thank you.
You can adjust the system size per processor at runtime via
"-nodes_per_proc 30" and the number of repeated calls to the function
containing KSPsolve() via "-iterations 1000". The physical problem is
simply calculating the electric potential from a homogeneous charge
distribution, done multiple times to accumulate time in KSPsolve().
A job would be started using something like
mpirun -n 125 ~/petsc_ws/ws_test -nodes_per_proc 30 -mesh_size 1E-4
-iterations 1000 \\
-ksp_rtol 1E-6 \
-log_view -log_sync\
-pc_type gamg -pc_gamg_type classical\
-ksp_type cg \
-ksp_norm_type unpreconditioned \
-mg_levels_ksp_type richardson \
-mg_levels_ksp_norm_type none \
-mg_levels_pc_type sor \
-mg_levels_ksp_max_it 1 \
-mg_levels_pc_sor_its 1 \
-mg_levels_esteig_ksp_type cg \
-mg_levels_esteig_ksp_max_it 10 \
-gamg_est_ksp_type cg
, ideally started on a cube number of processes for a cubical process grid.
Using 125 processes and 10.000 iterations I get the output in
"log_view_125_new.txt", which shows the same imbalance for me.
Michael
Am 02.06.2018 um 13:40 schrieb Mark Adams:
>
>
> On Fri, Jun 1, 2018 at 11:20 PM, Junchao Zhang <jczhang at mcs.anl.gov
> <mailto:jczhang at mcs.anl.gov>> wrote:
>
> Hi,Michael,
> You can add -log_sync besides -log_view, which adds barriers to
> certain events but measures barrier time separately from the
> events. I find this option makes it easier to interpret log_view
> output.
>
>
> That is great (good to know).
>
> This should give us a better idea if your large VecScatter costs are
> from slow communication or if it catching some sort of load imbalance.
>
>
> --Junchao Zhang
>
> On Wed, May 30, 2018 at 3:27 AM, Michael Becker
> <Michael.Becker at physik.uni-giessen.de
> <mailto:Michael.Becker at physik.uni-giessen.de>> wrote:
>
> Barry: On its way. Could take a couple days again.
>
> Junchao: I unfortunately don't have access to a cluster with a
> faster network. This one has a mixed 4X QDR-FDR InfiniBand 2:1
> blocking fat-tree network, which I realize causes parallel
> slowdown if the nodes are not connected to the same switch.
> Each node has 24 processors (2x12/socket) and four NUMA
> domains (two for each socket).
> The ranks are usually not distributed perfectly even, i.e. for
> 125 processes, of the six required nodes, five would use 21
> cores and one 20.
> Would using another CPU type make a difference
> communication-wise? I could switch to faster ones (on the same
> network), but I always assumed this would only improve
> performance of the stuff that is unrelated to communication.
>
> Michael
>
>
>
>> The log files have something like "Average time for zero size
>> MPI_Send(): 1.84231e-05". It looks you ran on a cluster with
>> a very slow network. A typical machine should give less than
>> 1/10 of the latency you have. An easy way to try is just
>> running the code on a machine with a faster network and see
>> what happens.
>>
>> Also, how many cores & numa domains does a compute node have?
>> I could not figure out how you distributed the 125 MPI ranks
>> evenly.
>>
>> --Junchao Zhang
>>
>> On Tue, May 29, 2018 at 6:18 AM, Michael Becker
>> <Michael.Becker at physik.uni-giessen.de
>> <mailto:Michael.Becker at physik.uni-giessen.de>> wrote:
>>
>> Hello again,
>>
>> here are the updated log_view files for 125 and 1000
>> processors. I ran both problems twice, the first time
>> with all processors per node allocated ("-1.txt"), the
>> second with only half on twice the number of nodes
>> ("-2.txt").
>>
>>
>>>> On May 24, 2018, at 12:24 AM, Michael Becker<Michael.Becker at physik.uni-giessen.de>
>>>> <mailto:Michael.Becker at physik.uni-giessen.de> wrote:
>>>>
>>>> I noticed that for every individual KSP iteration, six vector objects are created and destroyed (with CG, more with e.g. GMRES).
>>> Hmm, it is certainly not intended at vectors be created and destroyed within each KSPSolve() could you please point us to the code that makes you think they are being created and destroyed? We create all the work vectors at KSPSetUp() and destroy them in KSPReset() not during the solve. Not that this would be a measurable distance.
>>
>> I mean this, right in the log_view output:
>>
>>> Memory usage is given in bytes:
>>>
>>> Object Type Creations Destructions Memory Descendants' Mem.
>>> Reports information only for process 0.
>>>
>>> --- Event Stage 0: Main Stage
>>>
>>> ...
>>>
>>> --- Event Stage 1: First Solve
>>>
>>> ...
>>>
>>> --- Event Stage 2: Remaining Solves
>>>
>>> Vector 23904 23904 1295501184 0.
>> I logged the exact number of KSP iterations over the 999
>> timesteps and its exactly 23904/6 = 3984.
>>
>> Michael
>>
>>
>>
>> Am 24.05.2018 um 19:50 schrieb Smith, Barry F.:
>>> Please send the log file for 1000 with cg as the solver.
>>>
>>> You should make a bar chart of each event for the two cases to see which ones are taking more time and which are taking less (we cannot tell with the two logs you sent us since they are for different solvers.)
>>>
>>>
>>>
>>>> On May 24, 2018, at 12:24 AM, Michael Becker<Michael.Becker at physik.uni-giessen.de>
>>>> <mailto:Michael.Becker at physik.uni-giessen.de> wrote:
>>>>
>>>> I noticed that for every individual KSP iteration, six vector objects are created and destroyed (with CG, more with e.g. GMRES).
>>> Hmm, it is certainly not intended at vectors be created and destroyed within each KSPSolve() could you please point us to the code that makes you think they are being created and destroyed? We create all the work vectors at KSPSetUp() and destroy them in KSPReset() not during the solve. Not that this would be a measurable distance.
>>>
>>>
>>>
>>>> This seems kind of wasteful, is this supposed to be like this? Is this even the reason for my problems? Apart from that, everything seems quite normal to me (but I'm not the expert here).
>>>>
>>>>
>>>> Thanks in advance.
>>>>
>>>> Michael
>>>>
>>>>
>>>>
>>>> <log_view_125procs.txt><log_view_1000procs.txt>
>>
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20180604/34557441/attachment-0001.html>
-------------- next part --------------
************************************************************************************************************************
*** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document ***
************************************************************************************************************************
---------------------------------------------- PETSc Performance Summary: ----------------------------------------------
/home/ritsat/beckerm/petsc_ws2/ws_test on a arch-linux-amd-opt named node1-051 with 125 processors, by beckerm Mon Jun 4 16:37:26 2018
Using Petsc Development GIT revision: v3.9.2-503-g9e88a8b GIT Date: 2018-05-24 08:01:24 -0500
Max Max/Min Avg Total
Time (sec): 2.249e+03 1.00000 2.249e+03
Objects: 4.205e+05 1.00000 4.205e+05
Flop: 3.694e+11 1.15838 3.497e+11 4.372e+13
Flop/sec: 1.643e+08 1.15838 1.555e+08 1.944e+10
MPI Messages: 1.812e+07 3.38507 1.233e+07 1.542e+09
MPI Message Lengths: 2.261e+10 2.20109 1.418e+03 2.186e+12
MPI Reductions: 3.706e+05 1.00000
Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
e.g., VecAXPY() for real vectors of length N --> 2N flop
and VecAXPY() for complex vectors of length N --> 8N flop
Summary of Stages: ----- Time ------ ----- Flop ----- --- Messages --- -- Message Lengths -- -- Reductions --
Avg %Total Avg %Total counts %Total Avg %Total counts %Total
0: Main Stage: 1.4622e-01 0.0% 0.0000e+00 0.0% 1.200e+03 0.0% 1.802e+03 0.0% 1.700e+01 0.0%
1: First Solve: 2.9678e+00 0.1% 5.6491e+09 0.0% 4.212e+05 0.0% 3.421e+03 0.1% 5.660e+02 0.2%
2: Remaining Solves: 2.2459e+03 99.9% 4.3710e+13 100.0% 1.541e+09 100.0% 1.417e+03 99.9% 3.700e+05 99.8%
------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
Count: number of times phase was executed
Time and Flop: Max - maximum over all processors
Ratio - ratio of maximum to minimum over all processors
Mess: number of messages sent
Avg. len: average message length (bytes)
Reduct: number of global reductions
Global: entire computation
Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
%T - percent time in this phase %F - percent flop in this phase
%M - percent messages in this phase %L - percent message lengths in this phase
%R - percent reductions in this phase
Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event Count Time (sec) Flop --- Global --- --- Stage --- Total
Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------
--- Event Stage 0: Main Stage
VecSet 2 1.0 1.3185e-04 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
--- Event Stage 1: First Solve
BuildTwoSided 12 1.0 9.4547e-03 1.6 0.00e+00 0.0 8.8e+03 4.0e+00 0.0e+00 0 0 0 0 0 0 0 2 0 0 0
BuildTwoSidedF 30 1.0 2.2989e-01 3.1 0.00e+00 0.0 7.1e+03 1.0e+04 0.0e+00 0 0 0 0 0 5 0 2 5 0 0
KSPSetUp 9 1.0 5.4758e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.8e+01 0 0 0 0 0 0 0 0 0 3 0
KSPSolve 1 1.0 2.9672e+00 1.0 4.82e+07 1.2 4.2e+05 3.4e+03 5.7e+02 0 0 0 0 0 100100100100100 1904
VecTDot 14 1.0 1.8083e-02 5.9 7.56e+05 1.0 0.0e+00 0.0e+00 1.4e+01 0 0 0 0 0 0 2 0 0 2 5226
VecNormBarrier 9 1.0 2.1014e-03 5.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecNorm 9 1.0 9.9111e-04 1.2 4.86e+05 1.0 0.0e+00 0.0e+00 9.0e+00 0 0 0 0 0 0 1 0 0 2 61295
VecScale 42 1.0 4.4274e-04 3.0 9.47e+04 2.2 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 20631
VecCopy 1 1.0 1.2612e-04 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecSet 178 1.0 1.3885e-03 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecAXPY 14 1.0 2.0151e-03 2.0 7.56e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 2 0 0 0 46896
VecAYPX 49 1.0 2.4726e-03 2.1 6.46e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 32264
VecAssemblyBegin 2 1.0 2.3842e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecAssemblyEnd 2 1.0 2.1458e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecScatterBarrie 178 1.0 9.6261e-02 3.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 2 0 0 0 0 0
VecScatterBegin 178 1.0 1.4301e-02 4.6 0.00e+00 0.0 1.5e+05 1.4e+03 0.0e+00 0 0 0 0 0 0 0 37 15 0 0
VecScatterEnd 178 1.0 3.4915e-02 4.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0
MatMult 50 1.0 8.9105e-02 1.5 1.05e+07 1.1 5.1e+04 2.1e+03 0.0e+00 0 0 0 0 0 2 22 12 7 0 13981
MatMultAdd 42 1.0 3.3666e-02 1.7 2.40e+06 1.3 2.8e+04 6.7e+02 0.0e+00 0 0 0 0 0 1 5 7 1 0 8183
MatMultTranspose 42 1.0 2.7552e-02 1.5 2.40e+06 1.3 2.8e+04 6.7e+02 0.0e+00 0 0 0 0 0 1 5 7 1 0 10000
MatSolve 7 0.0 3.0994e-05 0.0 8.40e+02 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 27
MatSOR 84 1.0 1.2700e-01 1.7 1.90e+07 1.2 4.7e+04 1.6e+03 1.4e+01 0 0 0 0 0 4 40 11 5 2 17715
MatLUFactorSym 1 1.0 9.7990e-05 2.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatLUFactorNum 1 1.0 1.5020e-05 5.2 3.14e+02 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 21
MatResidual 42 1.0 7.5414e-02 1.7 7.97e+06 1.2 4.7e+04 1.6e+03 0.0e+00 0 0 0 0 0 2 17 11 5 0 12369
MatAssemblyBegin 94 1.0 2.3261e-01 2.8 0.00e+00 0.0 7.1e+03 1.0e+04 0.0e+00 0 0 0 0 0 5 0 2 5 0 0
MatAssemblyEnd 94 1.0 2.2487e-01 1.1 0.00e+00 0.0 6.3e+04 2.1e+02 2.3e+02 0 0 0 0 0 7 0 15 1 41 0
MatGetRow 3100250 1.2 4.5413e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 14 0 0 0 0 0
MatGetRowIJ 1 0.0 1.0014e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatCreateSubMats 6 1.0 5.7464e-01 1.5 0.00e+00 0.0 5.5e+04 1.8e+04 1.2e+01 0 0 0 0 0 16 0 13 67 2 0
MatCreateSubMat 4 1.0 9.9725e-02 1.0 0.00e+00 0.0 2.8e+03 2.8e+02 6.4e+01 0 0 0 0 0 3 0 1 0 11 0
MatGetOrdering 1 0.0 1.5283e-04 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatIncreaseOvrlp 6 1.0 7.9965e-02 1.3 0.00e+00 0.0 2.7e+04 1.0e+03 1.2e+01 0 0 0 0 0 2 0 6 2 2 0
MatCoarsen 6 1.0 2.0824e-02 1.1 0.00e+00 0.0 5.4e+04 6.0e+02 3.4e+01 0 0 0 0 0 1 0 13 2 6 0
MatZeroEntries 6 1.0 3.5489e-03 5.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatPtAP 6 1.0 3.1160e-01 1.0 1.13e+07 1.3 6.4e+04 2.7e+03 9.2e+01 0 0 0 0 0 10 23 15 12 16 4085
MatPtAPSymbolic 6 1.0 1.5794e-01 1.0 0.00e+00 0.0 3.4e+04 2.7e+03 4.2e+01 0 0 0 0 0 5 0 8 6 7 0
MatPtAPNumeric 6 1.0 1.4963e-01 1.0 1.13e+07 1.3 2.9e+04 2.6e+03 4.8e+01 0 0 0 0 0 5 23 7 5 8 8507
MatGetLocalMat 6 1.0 4.6241e-03 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatGetBrAoCol 6 1.0 1.0492e-02 1.6 0.00e+00 0.0 2.0e+04 3.6e+03 0.0e+00 0 0 0 0 0 0 0 5 5 0 0
SFSetGraph 12 1.0 1.5020e-05 3.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
SFSetUp 12 1.0 1.6023e-02 1.1 0.00e+00 0.0 2.6e+04 6.3e+02 0.0e+00 0 0 0 0 0 1 0 6 1 0 0
SFBcastBegin 46 1.0 2.0034e-03 2.2 0.00e+00 0.0 5.5e+04 7.0e+02 0.0e+00 0 0 0 0 0 0 0 13 3 0 0
SFBcastEnd 46 1.0 8.9741e-03 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
GAMG: createProl 6 1.0 2.2056e+00 1.0 0.00e+00 0.0 2.0e+05 5.3e+03 2.9e+02 0 0 0 0 0 74 0 47 73 51 0
GAMG: partLevel 6 1.0 4.1617e-01 1.0 1.13e+07 1.3 6.6e+04 2.6e+03 1.9e+02 0 0 0 0 0 14 23 16 12 34 3059
repartition 2 1.0 1.3080e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01 0 0 0 0 0 0 0 0 0 2 0
Invert-Sort 2 1.0 1.3320e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 0 0 0 0 0 1 0
Move A 2 1.0 2.3206e-02 1.0 0.00e+00 0.0 1.4e+03 5.4e+02 3.4e+01 0 0 0 0 0 1 0 0 0 6 0
Move P 2 1.0 7.7642e-02 1.0 0.00e+00 0.0 1.4e+03 1.3e+01 3.4e+01 0 0 0 0 0 3 0 0 0 6 0
PCSetUp 2 1.0 2.6313e+00 1.0 1.13e+07 1.3 2.7e+05 4.6e+03 5.1e+02 0 0 0 0 0 89 23 63 85 91 484
PCSetUpOnBlocks 7 1.0 4.0460e-04 1.9 3.14e+02 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1
PCApply 7 1.0 2.2264e-01 1.1 3.18e+07 1.2 1.5e+05 1.2e+03 1.4e+01 0 0 0 0 0 7 66 35 13 2 16770
--- Event Stage 2: Remaining Solves
KSPSolve 10000 1.0 2.2427e+03 1.0 3.69e+11 1.2 1.5e+09 1.4e+03 3.7e+05100100100100100 100100100100100 19490
VecTDot 140000 1.0 1.8447e+02 6.4 7.56e+09 1.0 0.0e+00 0.0e+00 1.4e+05 3 2 0 0 38 3 2 0 0 38 5123
VecNormBarrier 90000 1.0 1.9840e+01 5.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecNorm 90000 1.0 8.2702e+00 1.2 4.86e+09 1.0 0.0e+00 0.0e+00 9.0e+04 0 1 0 0 24 0 1 0 0 24 73456
VecScale 420000 1.0 2.1495e+00 2.2 9.47e+08 2.2 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 42496
VecCopy 10000 1.0 1.2405e+00 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecSet 1470000 1.0 1.3027e+01 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecAXPY 140000 1.0 1.8036e+01 1.8 7.56e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 52397
VecAYPX 490000 1.0 2.4474e+01 2.2 6.46e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 2 0 0 0 1 2 0 0 0 32596
VecScatterBarrie 1760000 1.0 7.9999e+02 6.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 15 0 0 0 0 15 0 0 0 0 0
VecScatterBegin 1760000 1.0 1.1154e+02 3.9 0.00e+00 0.0 1.5e+09 1.4e+03 0.0e+00 3 0100100 0 3 0100100 0 0
VecScatterEnd 1760000 1.0 3.2220e+02 4.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 8 0 0 0 0 8 0 0 0 0 0
MatMult 500000 1.0 7.7645e+02 1.6 1.05e+11 1.1 5.1e+08 2.1e+03 0.0e+00 24 28 33 49 0 24 29 33 49 0 16044
MatMultAdd 420000 1.0 3.0354e+02 1.8 2.40e+10 1.3 2.8e+08 6.7e+02 0.0e+00 10 6 18 9 0 10 6 18 9 0 9077
MatMultTranspose 420000 1.0 2.5492e+02 1.7 2.40e+10 1.3 2.8e+08 6.7e+02 0.0e+00 8 6 18 9 0 8 6 18 9 0 10807
MatSolve 70000 0.0 2.8931e-01 0.0 8.40e+06 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 29
MatSOR 840000 1.0 1.2249e+03 1.7 1.90e+11 1.2 4.7e+08 1.6e+03 1.4e+05 49 51 30 33 38 49 51 30 33 38 18326
MatResidual 420000 1.0 6.4252e+02 1.8 7.97e+10 1.2 4.7e+08 1.6e+03 0.0e+00 19 21 30 33 0 19 21 30 33 0 14518
PCSetUpOnBlocks 70000 1.0 9.7365e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
PCApply 70000 1.0 2.0260e+03 1.1 3.18e+11 1.2 1.5e+09 1.2e+03 1.4e+05 89 85 97 84 38 89 85 97 84 38 18404
------------------------------------------------------------------------------------------------------------------------
Memory usage is given in bytes:
Object Type Creations Destructions Memory Descendants' Mem.
Reports information only for process 0.
--- Event Stage 0: Main Stage
Krylov Solver 1 9 11424 0.
DMKSP interface 1 1 656 0.
Vector 4 52 2371888 0.
Matrix 0 72 14160468 0.
Distributed Mesh 1 1 5248 0.
Index Set 2 12 133928 0.
IS L to G Mapping 1 1 131728 0.
Star Forest Graph 2 2 1728 0.
Discrete System 1 1 932 0.
Vec Scatter 1 14 233696 0.
Preconditioner 1 9 9676 0.
Viewer 1 0 0 0.
--- Event Stage 1: First Solve
Krylov Solver 8 0 0 0.
Vector 158 110 3181312 0.
Matrix 140 68 21757144 0.
Matrix Coarsen 6 6 3816 0.
Index Set 110 100 543716 0.
Star Forest Graph 12 12 10368 0.
Vec Scatter 31 18 22752 0.
Preconditioner 8 0 0 0.
--- Event Stage 2: Remaining Solves
Vector 420000 420000 22766800000 0.
========================================================================================================================
Average time to get PetscTime(): 9.53674e-08
Average time for MPI_Barrier(): 1.80244e-05
Average time for zero size MPI_Send(): 1.40877e-05
#PETSc Option Table entries:
-gamg_est_ksp_type cg
-iterations 10000
-ksp_norm_type unpreconditioned
-ksp_rtol 1E-6
-ksp_type cg
-log_sync
-log_view
-mesh_size 1E-4
-mg_levels_esteig_ksp_max_it 10
-mg_levels_esteig_ksp_type cg
-mg_levels_ksp_max_it 1
-mg_levels_ksp_norm_type none
-mg_levels_ksp_type richardson
-mg_levels_pc_sor_its 1
-mg_levels_pc_type sor
-nodes_per_proc 30
-pc_gamg_type classical
-pc_type gamg
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure options: --known-level1-dcache-size=65536 --known-level1-dcache-linesize=64 --known-level1-dcache-assoc=2 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-memcmp-ok=1 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-has-attribute-aligned=1 PETSC_ARCH=arch-linux-amd-opt --download-f2cblaslapack --with-mpi-dir=/cm/shared/apps/mvapich2/intel-17.0.1/2.0 --download-hypre --download-ml --with-fc=0 --with-debugging=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 --with-batch --with-x --known-mpi-shared-libraries=1 --known-64-bit-blas-indices=4
-----------------------------------------
Libraries compiled on 2018-05-25 07:05:14 on node1-001
Machine characteristics: Linux-2.6.32-696.18.7.el6.x86_64-x86_64-with-redhat-6.6-Carbon
Using PETSc directory: /home/ritsat/beckerm/petsc
Using PETSc arch: arch-linux-amd-opt
-----------------------------------------
Using C compiler: /cm/shared/apps/mvapich2/intel-17.0.1/2.0/bin/mpicc -fPIC -wd1572 -O3
-----------------------------------------
Using include paths: -I/home/ritsat/beckerm/petsc/include -I/home/ritsat/beckerm/petsc/arch-linux-amd-opt/include -I/cm/shared/apps/mvapich2/intel-17.0.1/2.0/include
-----------------------------------------
Using C linker: /cm/shared/apps/mvapich2/intel-17.0.1/2.0/bin/mpicc
Using libraries: -Wl,-rpath,/home/ritsat/beckerm/petsc/arch-linux-amd-opt/lib -L/home/ritsat/beckerm/petsc/arch-linux-amd-opt/lib -lpetsc -Wl,-rpath,/home/ritsat/beckerm/petsc/arch-linux-amd-opt/lib -L/home/ritsat/beckerm/petsc/arch-linux-amd-opt/lib -lHYPRE -lml -lf2clapack -lf2cblas -lX11 -ldl
-----------------------------------------
-------------- next part --------------
************************************************************************************************************************
*** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document ***
************************************************************************************************************************
---------------------------------------------- PETSc Performance Summary: ----------------------------------------------
/home/ritsat/beckerm/ppp_test/plasmapic on a arch-linux-amd-opt named node4-083 with 125 processors, by beckerm Wed May 30 16:24:18 2018
Using Petsc Development GIT revision: v3.9.2-503-g9e88a8b GIT Date: 2018-05-24 08:01:24 -0500
Max Max/Min Avg Total
Time (sec): 2.497e+02 1.00000 2.497e+02
Objects: 2.438e+04 1.00004 2.438e+04
Flop: 2.125e+10 1.27708 1.963e+10 2.454e+12
Flop/sec: 8.510e+07 1.27708 7.862e+07 9.828e+09
MPI Messages: 1.042e+06 3.36140 7.129e+05 8.911e+07
MPI Message Lengths: 1.344e+09 2.32209 1.439e+03 1.282e+11
MPI Reductions: 2.250e+04 1.00000
Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
e.g., VecAXPY() for real vectors of length N --> 2N flop
and VecAXPY() for complex vectors of length N --> 8N flop
Summary of Stages: ----- Time ------ ----- Flop ----- --- Messages --- -- Message Lengths -- -- Reductions --
Avg %Total Avg %Total counts %Total Avg %Total counts %Total
0: Main Stage: 6.4457e+00 2.6% 0.0000e+00 0.0% 3.000e+03 0.0% 3.178e+03 0.0% 1.700e+01 0.1%
1: First Solve: 2.4843e+00 1.0% 3.6885e+09 0.2% 3.549e+05 0.4% 3.736e+03 1.0% 5.500e+02 2.4%
2: Remaining Solves: 2.4077e+02 96.4% 2.4504e+12 99.8% 8.875e+07 99.6% 1.430e+03 99.0% 2.192e+04 97.4%
------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
Count: number of times phase was executed
Time and Flop: Max - maximum over all processors
Ratio - ratio of maximum to minimum over all processors
Mess: number of messages sent
Avg. len: average message length (bytes)
Reduct: number of global reductions
Global: entire computation
Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
%T - percent time in this phase %F - percent flop in this phase
%M - percent messages in this phase %L - percent message lengths in this phase
%R - percent reductions in this phase
Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event Count Time (sec) Flop --- Global --- --- Stage --- Total
Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------
--- Event Stage 0: Main Stage
VecSet 3 1.0 4.7421e-04 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
--- Event Stage 1: First Solve
BuildTwoSided 12 1.0 6.9022e-03 1.9 0.00e+00 0.0 8.8e+03 4.0e+00 0.0e+00 0 0 0 0 0 0 0 2 0 0 0
BuildTwoSidedF 30 1.0 2.8585e-01 3.6 0.00e+00 0.0 7.1e+03 1.0e+04 0.0e+00 0 0 0 0 0 6 0 2 5 0 0
KSPSetUp 9 1.0 3.2625e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.8e+01 0 0 0 0 0 0 0 0 0 3 0
KSPSolve 1 1.0 2.4871e+00 1.0 3.26e+07 1.4 3.5e+05 3.7e+03 5.5e+02 1 0 0 1 2 100100100100100 1483
VecTDot 8 1.0 6.3627e-03 3.7 4.32e+05 1.0 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 0 0 1 0 0 1 8487
VecNorm 6 1.0 1.3714e-03 2.8 3.24e+05 1.0 0.0e+00 0.0e+00 6.0e+00 0 0 0 0 0 0 1 0 0 1 29532
VecScale 24 1.0 1.4067e-04 2.4 5.43e+04 2.4 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 36596
VecCopy 1 1.0 1.1802e-04 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecSet 115 1.0 7.8726e-04 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecAXPY 8 1.0 1.1842e-03 1.4 4.32e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 45599
VecAYPX 28 1.0 1.3180e-03 1.6 3.58e+05 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 33275
VecAssemblyBegin 2 1.0 2.1458e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecAssemblyEnd 2 1.0 2.1458e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecScatterBegin 103 1.0 6.1374e-03 2.8 0.00e+00 0.0 8.9e+04 1.4e+03 0.0e+00 0 0 0 0 0 0 0 25 9 0 0
VecScatterEnd 103 1.0 3.9134e-02 3.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0
MatMult 29 1.0 3.4050e-02 1.4 6.14e+06 1.2 3.0e+04 2.1e+03 0.0e+00 0 0 0 0 0 1 19 8 5 0 21052
MatMultAdd 24 1.0 1.4260e-02 2.5 1.37e+06 1.6 1.6e+04 6.5e+02 0.0e+00 0 0 0 0 0 0 4 5 1 0 10595
MatMultTranspose 24 1.0 1.4215e-02 2.7 1.37e+06 1.6 1.6e+04 6.5e+02 0.0e+00 0 0 0 0 0 0 4 5 1 0 10629
MatSolve 4 0.0 1.9312e-05 0.0 2.64e+02 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 14
MatSOR 48 1.0 7.1831e-02 1.3 1.09e+07 1.3 2.7e+04 1.5e+03 8.0e+00 0 0 0 0 0 3 34 8 3 1 17455
MatLUFactorSym 1 1.0 5.3883e-05 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatLUFactorNum 1 1.0 1.4067e-05 7.4 1.29e+02 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 9
MatResidual 24 1.0 2.7424e-02 1.7 4.55e+06 1.3 2.7e+04 1.5e+03 0.0e+00 0 0 0 0 0 1 14 8 3 0 18926
MatAssemblyBegin 94 1.0 2.8823e-01 3.2 0.00e+00 0.0 7.1e+03 1.0e+04 0.0e+00 0 0 0 0 0 7 0 2 5 0 0
MatAssemblyEnd 94 1.0 8.1276e-02 1.1 0.00e+00 0.0 6.3e+04 2.1e+02 2.3e+02 0 0 0 0 1 3 0 18 1 42 0
MatGetRow 3102093 1.3 4.5239e-01 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 16 0 0 0 0 0
MatGetRowIJ 1 0.0 5.9605e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatCreateSubMats 6 1.0 4.4482e-01 2.4 0.00e+00 0.0 5.5e+04 1.7e+04 1.2e+01 0 0 0 1 0 13 0 15 71 2 0
MatCreateSubMat 4 1.0 6.8214e-03 1.0 0.00e+00 0.0 2.9e+03 2.7e+02 6.4e+01 0 0 0 0 0 0 0 1 0 12 0
MatGetOrdering 1 0.0 1.1611e-04 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatIncreaseOvrlp 6 1.0 5.6694e-02 1.2 0.00e+00 0.0 2.7e+04 1.0e+03 1.2e+01 0 0 0 0 0 2 0 8 2 2 0
MatCoarsen 6 1.0 1.6149e-02 1.0 0.00e+00 0.0 5.3e+04 5.8e+02 3.3e+01 0 0 0 0 0 1 0 15 2 6 0
MatZeroEntries 6 1.0 3.4871e-03 4.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatPtAP 6 1.0 2.5855e-01 1.0 1.13e+07 1.6 6.3e+04 2.6e+03 9.2e+01 0 0 0 0 0 10 33 18 13 17 4731
MatPtAPSymbolic 6 1.0 1.4759e-01 1.0 0.00e+00 0.0 3.4e+04 2.7e+03 4.2e+01 0 0 0 0 0 6 0 10 7 8 0
MatPtAPNumeric 6 1.0 1.1022e-01 1.0 1.13e+07 1.6 2.9e+04 2.6e+03 4.8e+01 0 0 0 0 0 4 33 8 6 9 11099
MatGetLocalMat 6 1.0 4.5311e-03 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatGetBrAoCol 6 1.0 8.2998e-03 1.7 0.00e+00 0.0 2.0e+04 3.5e+03 0.0e+00 0 0 0 0 0 0 0 6 5 0 0
SFSetGraph 12 1.0 1.3113e-05 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
SFSetUp 12 1.0 1.2064e-02 1.1 0.00e+00 0.0 2.6e+04 6.2e+02 0.0e+00 0 0 0 0 0 0 0 7 1 0 0
SFBcastBegin 45 1.0 2.0254e-03 2.3 0.00e+00 0.0 5.4e+04 6.9e+02 0.0e+00 0 0 0 0 0 0 0 15 3 0 0
SFBcastEnd 45 1.0 5.0657e-03 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
GAMG: createProl 6 1.0 2.0193e+00 1.0 0.00e+00 0.0 2.0e+05 5.2e+03 2.8e+02 1 0 0 1 1 81 0 56 78 52 0
GAMG: partLevel 6 1.0 2.6818e-01 1.0 1.13e+07 1.6 6.6e+04 2.5e+03 1.9e+02 0 0 0 0 1 11 33 19 13 35 4562
repartition 2 1.0 6.9189e-04 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01 0 0 0 0 0 0 0 0 0 2 0
Invert-Sort 2 1.0 6.0797e-04 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 0 0 0 0 0 1 0
Move A 2 1.0 4.2229e-03 1.1 0.00e+00 0.0 1.4e+03 5.3e+02 3.4e+01 0 0 0 0 0 0 0 0 0 6 0
Move P 2 1.0 3.6728e-03 1.1 0.00e+00 0.0 1.4e+03 1.3e+01 3.4e+01 0 0 0 0 0 0 0 0 0 6 0
PCSetUp 2 1.0 2.2978e+00 1.0 1.13e+07 1.6 2.7e+05 4.5e+03 5.1e+02 1 0 0 1 2 92 33 75 90 93 532
PCSetUpOnBlocks 4 1.0 2.5201e-04 1.4 1.29e+02 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1
PCApply 4 1.0 1.1062e-01 1.0 1.82e+07 1.3 8.6e+04 1.2e+03 8.0e+00 0 0 0 0 0 4 56 24 8 1 18759
--- Event Stage 2: Remaining Solves
KSPSolve 999 1.0 1.2099e+02 1.0 2.12e+10 1.3 8.8e+07 1.4e+03 2.2e+04 47100 99 97 97 49100 99 98100 20253
VecTDot 7968 1.0 6.5506e+00 4.0 4.30e+08 1.0 0.0e+00 0.0e+00 8.0e+03 1 2 0 0 35 1 2 0 0 36 8210
VecNorm 5982 1.0 5.8032e+0010.2 3.23e+08 1.0 0.0e+00 0.0e+00 6.0e+03 1 2 0 0 27 1 2 0 0 27 6958
VecScale 23904 1.0 1.1292e-01 2.2 5.40e+07 2.4 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 45407
VecCopy 999 1.0 1.1493e-01 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecSet 83664 1.0 6.9683e-01 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecAXPY 7968 1.0 1.1667e+00 1.4 4.30e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 46100
VecAYPX 27888 1.0 1.2901e+00 1.6 3.56e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 33851
VecScatterBegin 100599 1.0 5.9735e+00 2.8 0.00e+00 0.0 8.8e+07 1.4e+03 0.0e+00 2 0 99 97 0 2 0 99 98 0 0
VecScatterEnd 100599 1.0 3.5510e+01 4.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 7 0 0 0 0 8 0 0 0 0 0
MatMult 28887 1.0 3.4026e+01 1.5 6.12e+09 1.2 3.0e+07 2.1e+03 0.0e+00 11 29 33 49 0 11 29 33 49 0 20986
MatMultAdd 23904 1.0 1.1303e+01 2.1 1.37e+09 1.6 1.6e+07 6.5e+02 0.0e+00 3 6 18 8 0 3 6 18 8 0 13314
MatMultTranspose 23904 1.0 1.3880e+01 2.6 1.37e+09 1.6 1.6e+07 6.5e+02 0.0e+00 4 6 18 8 0 4 6 18 8 0 10842
MatSolve 3984 0.0 1.6525e-02 0.0 2.63e+05 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 16
MatSOR 47808 1.0 6.6732e+01 1.3 1.08e+10 1.3 2.7e+07 1.5e+03 8.0e+03 25 51 30 32 35 26 51 30 32 36 18638
MatResidual 23904 1.0 2.6673e+01 1.7 4.54e+09 1.3 2.7e+07 1.5e+03 0.0e+00 8 21 30 32 0 8 21 30 32 0 19381
PCSetUpOnBlocks 3984 1.0 5.2946e-03 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
PCApply 3984 1.0 1.0349e+02 1.0 1.81e+10 1.3 8.5e+07 1.2e+03 8.0e+03 41 84 96 80 35 42 84 96 81 36 19921
------------------------------------------------------------------------------------------------------------------------
Memory usage is given in bytes:
Object Type Creations Destructions Memory Descendants' Mem.
Reports information only for process 0.
--- Event Stage 0: Main Stage
Krylov Solver 1 9 11424 0.
DMKSP interface 1 0 0 0.
Vector 5 52 2371496 0.
Matrix 0 72 14138216 0.
Distributed Mesh 1 0 0 0.
Index Set 2 12 133768 0.
IS L to G Mapping 1 0 0 0.
Star Forest Graph 2 0 0 0.
Discrete System 1 0 0 0.
Vec Scatter 1 13 16432 0.
Preconditioner 1 9 9676 0.
Viewer 1 0 0 0.
--- Event Stage 1: First Solve
Krylov Solver 8 0 0 0.
Vector 140 92 2204792 0.
Matrix 140 68 21738552 0.
Matrix Coarsen 6 6 3816 0.
Index Set 110 100 543240 0.
Star Forest Graph 12 12 10368 0.
Vec Scatter 31 18 22752 0.
Preconditioner 8 0 0 0.
--- Event Stage 2: Remaining Solves
Vector 23904 23904 1295501184 0.
========================================================================================================================
Average time to get PetscTime(): 9.53674e-08
Average time for MPI_Barrier(): 1.81675e-05
Average time for zero size MPI_Send(): 1.45779e-05
#PETSc Option Table entries:
-gamg_est_ksp_type cg
-ksp_norm_type unpreconditioned
-ksp_type cg
-log_view
-mg_levels_esteig_ksp_max_it 10
-mg_levels_esteig_ksp_type cg
-mg_levels_ksp_max_it 1
-mg_levels_ksp_norm_type none
-mg_levels_ksp_type richardson
-mg_levels_pc_sor_its 1
-mg_levels_pc_type sor
-pc_gamg_type classical
-pc_type gamg
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure options: --known-level1-dcache-size=65536 --known-level1-dcache-linesize=64 --known-level1-dcache-assoc=2 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-memcmp-ok=1 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-has-attribute-aligned=1 PETSC_ARCH=arch-linux-amd-opt --download-f2cblaslapack --with-mpi-dir=/cm/shared/apps/mvapich2/intel-17.0.1/2.0 --download-hypre --download-ml --with-fc=0 --with-debugging=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 --with-batch --with-x --known-mpi-shared-libraries=1 --known-64-bit-blas-indices=4
-----------------------------------------
Libraries compiled on 2018-05-25 07:05:14 on node1-001
Machine characteristics: Linux-2.6.32-696.18.7.el6.x86_64-x86_64-with-redhat-6.6-Carbon
Using PETSc directory: /home/ritsat/beckerm/petsc
Using PETSc arch: arch-linux-amd-opt
-----------------------------------------
Using C compiler: /cm/shared/apps/mvapich2/intel-17.0.1/2.0/bin/mpicc -fPIC -wd1572 -O3
-----------------------------------------
Using include paths: -I/home/ritsat/beckerm/petsc/include -I/home/ritsat/beckerm/petsc/arch-linux-amd-opt/include -I/cm/shared/apps/mvapich2/intel-17.0.1/2.0/include
-----------------------------------------
Using C linker: /cm/shared/apps/mvapich2/intel-17.0.1/2.0/bin/mpicc
Using libraries: -Wl,-rpath,/home/ritsat/beckerm/petsc/arch-linux-amd-opt/lib -L/home/ritsat/beckerm/petsc/arch-linux-amd-opt/lib -lpetsc -Wl,-rpath,/home/ritsat/beckerm/petsc/arch-linux-amd-opt/lib -L/home/ritsat/beckerm/petsc/arch-linux-amd-opt/lib -lHYPRE -lml -lf2clapack -lf2cblas -lX11 -ldl
-----------------------------------------
-------------- next part --------------
************************************************************************************************************************
*** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document ***
************************************************************************************************************************
---------------------------------------------- PETSc Performance Summary: ----------------------------------------------
/home/ritsat/beckerm/ppp_test/plasmapic on a arch-linux-amd-opt named node1-013 with 1000 processors, by beckerm Wed May 30 19:06:34 2018
Using Petsc Development GIT revision: v3.9.2-503-g9e88a8b GIT Date: 2018-05-24 08:01:24 -0500
Max Max/Min Avg Total
Time (sec): 2.892e+02 1.00001 2.892e+02
Objects: 2.440e+04 1.00004 2.440e+04
Flop: 2.124e+10 1.27708 2.041e+10 2.041e+13
Flop/sec: 7.342e+07 1.27708 7.057e+07 7.057e+10
MPI Messages: 1.238e+06 3.99536 8.489e+05 8.489e+08
MPI Message Lengths: 1.343e+09 2.32238 1.393e+03 1.183e+12
MPI Reductions: 2.256e+04 1.00000
Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
e.g., VecAXPY() for real vectors of length N --> 2N flop
and VecAXPY() for complex vectors of length N --> 8N flop
Summary of Stages: ----- Time ------ ----- Flop ----- --- Messages --- -- Message Lengths -- -- Reductions --
Avg %Total Avg %Total counts %Total Avg %Total counts %Total
0: Main Stage: 2.7802e+01 9.6% 0.0000e+00 0.0% 2.700e+04 0.0% 3.178e+03 0.0% 1.700e+01 0.1%
1: First Solve: 3.3072e+00 1.1% 3.0885e+10 0.2% 3.675e+06 0.4% 3.508e+03 1.1% 6.220e+02 2.8%
2: Remaining Solves: 2.5814e+02 89.2% 2.0380e+13 99.8% 8.452e+08 99.6% 1.384e+03 98.9% 2.191e+04 97.1%
------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
Count: number of times phase was executed
Time and Flop: Max - maximum over all processors
Ratio - ratio of maximum to minimum over all processors
Mess: number of messages sent
Avg. len: average message length (bytes)
Reduct: number of global reductions
Global: entire computation
Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
%T - percent time in this phase %F - percent flop in this phase
%M - percent messages in this phase %L - percent message lengths in this phase
%R - percent reductions in this phase
Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event Count Time (sec) Flop --- Global --- --- Stage --- Total
Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------
--- Event Stage 0: Main Stage
VecSet 3 1.0 5.7411e-04 3.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
--- Event Stage 1: First Solve
BuildTwoSided 12 1.0 1.3082e-02 1.5 0.00e+00 0.0 8.9e+04 4.0e+00 0.0e+00 0 0 0 0 0 0 0 2 0 0 0
BuildTwoSidedF 30 1.0 4.3104e-01 2.7 0.00e+00 0.0 6.5e+04 1.0e+04 0.0e+00 0 0 0 0 0 8 0 2 5 0 0
KSPSetUp 9 1.0 5.3465e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 1.8e+01 0 0 0 0 0 0 0 0 0 3 0
KSPSolve 1 1.0 3.3113e+00 1.0 3.25e+07 1.4 3.7e+06 3.5e+03 6.2e+02 1 0 0 1 3 100100100100100 9327
VecTDot 8 1.0 6.8474e-03 4.2 4.32e+05 1.0 0.0e+00 0.0e+00 8.0e+00 0 0 0 0 0 0 1 0 0 1 63089
VecNorm 6 1.0 1.6654e-03 2.8 3.24e+05 1.0 0.0e+00 0.0e+00 6.0e+00 0 0 0 0 0 0 1 0 0 1 194553
VecScale 24 1.0 1.3828e-04 2.4 5.43e+04 2.4 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 344362
VecCopy 1 1.0 1.4305e-04 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecSet 124 1.0 8.4376e-04 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecAXPY 8 1.0 1.2994e-03 1.6 4.32e+05 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 332466
VecAYPX 28 1.0 1.5504e-03 2.0 3.58e+05 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 1 0 0 0 228452
VecAssemblyBegin 3 1.0 3.0994e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecAssemblyEnd 3 1.0 3.0994e-06 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecScatterBegin 108 1.0 6.7146e-03 2.8 0.00e+00 0.0 8.4e+05 1.4e+03 0.0e+00 0 0 0 0 0 0 0 23 9 0 0
VecScatterEnd 108 1.0 4.8888e-02 2.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 1 0 0 0 0 0
MatMult 29 1.0 3.6063e-02 1.5 6.14e+06 1.2 2.8e+05 2.0e+03 0.0e+00 0 0 0 0 0 1 19 8 4 0 164544
MatMultAdd 24 1.0 2.2712e-02 3.2 1.37e+06 1.6 1.5e+05 6.5e+02 0.0e+00 0 0 0 0 0 1 4 4 1 0 56747
MatMultTranspose 24 1.0 1.4198e-02 2.3 1.37e+06 1.6 1.5e+05 6.5e+02 0.0e+00 0 0 0 0 0 0 4 4 1 0 90779
MatSolve 4 0.0 4.7922e-05 0.0 1.10e+04 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 229
MatSOR 48 1.0 7.6652e-02 1.3 1.09e+07 1.3 2.6e+05 1.5e+03 8.0e+00 0 0 0 0 0 2 34 7 3 1 136439
MatLUFactorSym 1 1.0 9.4891e-05 3.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatLUFactorNum 1 1.0 6.6042e-0534.6 3.29e+04 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 498
MatResidual 24 1.0 2.7970e-02 1.7 4.55e+06 1.3 2.6e+05 1.5e+03 0.0e+00 0 0 0 0 0 1 14 7 3 0 155507
MatAssemblyBegin 102 1.0 4.3351e-01 2.5 0.00e+00 0.0 6.5e+04 1.0e+04 0.0e+00 0 0 0 0 0 8 0 2 5 0 0
MatAssemblyEnd 102 1.0 1.1318e-01 1.1 0.00e+00 0.0 6.2e+05 2.0e+02 2.5e+02 0 0 0 0 1 3 0 17 1 40 0
MatGetRow 3102093 1.3 5.0438e-01 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 12 0 0 0 0 0
MatGetRowIJ 1 0.0 1.5974e-05 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatCreateSubMats 6 1.0 4.6014e-01 2.3 0.00e+00 0.0 5.7e+05 1.6e+04 1.2e+01 0 0 0 1 0 10 0 15 72 2 0
MatCreateSubMat 6 1.0 1.9575e-02 1.0 0.00e+00 0.0 2.2e+04 3.3e+02 9.4e+01 0 0 0 0 0 1 0 1 0 15 0
MatGetOrdering 1 0.0 1.4710e-04 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatIncreaseOvrlp 6 1.0 1.1349e-01 1.1 0.00e+00 0.0 2.6e+05 9.9e+02 1.2e+01 0 0 0 0 0 3 0 7 2 2 0
MatCoarsen 6 1.0 3.4475e-02 1.1 0.00e+00 0.0 7.1e+05 4.4e+02 5.6e+01 0 0 0 0 0 1 0 19 2 9 0
MatZeroEntries 6 1.0 3.4661e-03 4.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatPtAP 6 1.0 3.8021e-01 1.1 1.11e+07 1.6 6.3e+05 2.5e+03 9.2e+01 0 0 0 0 0 11 34 17 12 15 27282
MatPtAPSymbolic 6 1.0 2.1572e-01 1.0 0.00e+00 0.0 3.2e+05 2.7e+03 4.2e+01 0 0 0 0 0 7 0 9 7 7 0
MatPtAPNumeric 6 1.0 1.4471e-01 1.0 1.11e+07 1.6 3.0e+05 2.3e+03 4.8e+01 0 0 0 0 0 4 34 8 6 8 71679
MatGetLocalMat 6 1.0 4.7863e-03 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatGetBrAoCol 6 1.0 1.9191e-02 3.1 0.00e+00 0.0 1.9e+05 3.4e+03 0.0e+00 0 0 0 0 0 0 0 5 5 0 0
SFSetGraph 12 1.0 2.3127e-05 6.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
SFSetUp 12 1.0 1.9232e-02 1.1 0.00e+00 0.0 2.7e+05 5.8e+02 0.0e+00 0 0 0 0 0 1 0 7 1 0 0
SFBcastBegin 68 1.0 2.7168e-03 2.7 0.00e+00 0.0 7.2e+05 5.1e+02 0.0e+00 0 0 0 0 0 0 0 20 3 0 0
SFBcastEnd 68 1.0 1.5905e-02 3.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
GAMG: createProl 6 1.0 2.4556e+00 1.0 0.00e+00 0.0 2.2e+06 4.7e+03 3.1e+02 1 0 0 1 1 74 0 59 79 50 0
GAMG: partLevel 6 1.0 4.0791e-01 1.1 1.11e+07 1.6 6.5e+05 2.4e+03 2.4e+02 0 0 0 0 1 12 34 18 12 39 25428
repartition 3 1.0 2.3220e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.8e+01 0 0 0 0 0 0 0 0 0 3 0
Invert-Sort 3 1.0 2.0869e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.2e+01 0 0 0 0 0 0 0 0 0 2 0
Move A 3 1.0 1.2217e-02 1.1 0.00e+00 0.0 9.5e+03 7.4e+02 5.0e+01 0 0 0 0 0 0 0 0 0 8 0
Move P 3 1.0 9.7649e-03 1.1 0.00e+00 0.0 1.3e+04 1.3e+01 5.0e+01 0 0 0 0 0 0 0 0 0 8 0
PCSetUp 2 1.0 3.0642e+00 1.0 1.11e+07 1.6 2.8e+06 4.2e+03 5.8e+02 1 0 0 1 3 93 34 77 91 94 3385
PCSetUpOnBlocks 4 1.0 3.9506e-04 2.9 3.29e+04 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 83
PCApply 4 1.0 1.2272e-01 1.0 1.82e+07 1.3 8.2e+05 1.2e+03 8.0e+00 0 0 0 0 0 4 56 22 7 1 141671
--- Event Stage 2: Remaining Solves
KSPSolve 999 1.0 1.3648e+02 1.0 2.12e+10 1.3 8.4e+08 1.4e+03 2.2e+04 46100 99 97 97 51100 99 98100 149327
VecTDot 7964 1.0 7.1750e+00 3.4 4.30e+08 1.0 0.0e+00 0.0e+00 8.0e+03 1 2 0 0 35 1 2 0 0 36 59937
VecNorm 5980 1.0 6.0258e+00 4.9 3.23e+08 1.0 0.0e+00 0.0e+00 6.0e+03 1 2 0 0 27 1 2 0 0 27 53589
VecScale 23892 1.0 1.2695e-01 2.4 5.40e+07 2.4 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 373403
VecCopy 999 1.0 1.3586e-01 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecSet 83622 1.0 7.4253e-01 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecAXPY 7964 1.0 1.2791e+00 1.6 4.30e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 336221
VecAYPX 27874 1.0 1.4993e+00 2.0 3.56e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 235111
VecScatterBegin 100549 1.0 6.4721e+00 2.9 0.00e+00 0.0 8.4e+08 1.4e+03 0.0e+00 2 0 99 97 0 2 0 99 98 0 0
VecScatterEnd 100549 1.0 4.9949e+01 2.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 8 0 0 0 0 9 0 0 0 0 0
MatMult 28873 1.0 3.3321e+01 1.4 6.11e+09 1.2 2.8e+08 2.0e+03 0.0e+00 9 29 33 48 0 10 29 34 49 0 177317
MatMultAdd 23892 1.0 1.9948e+01 2.7 1.37e+09 1.6 1.5e+08 6.5e+02 0.0e+00 6 6 18 8 0 7 6 18 8 0 64321
MatMultTranspose 23892 1.0 1.3810e+01 2.1 1.37e+09 1.6 1.5e+08 6.5e+02 0.0e+00 3 6 18 8 0 3 6 18 8 0 92909
MatSolve 3982 0.0 4.4663e-02 0.0 1.09e+07 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 245
MatSOR 47784 1.0 7.1050e+01 1.3 1.08e+10 1.3 2.6e+08 1.5e+03 8.0e+03 23 51 30 32 35 26 51 30 32 36 145956
MatResidual 23892 1.0 2.4668e+01 1.5 4.53e+09 1.3 2.6e+08 1.5e+03 0.0e+00 7 21 30 32 0 7 21 30 32 0 175528
PCSetUpOnBlocks 3982 1.0 5.2893e-03 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
PCApply 3982 1.0 1.1642e+02 1.0 1.81e+10 1.3 8.1e+08 1.2e+03 8.0e+03 40 85 96 81 35 45 85 96 82 36 148313
------------------------------------------------------------------------------------------------------------------------
Memory usage is given in bytes:
Object Type Creations Destructions Memory Descendants' Mem.
Reports information only for process 0.
--- Event Stage 0: Main Stage
Krylov Solver 1 9 11424 0.
DMKSP interface 1 0 0 0.
Vector 5 52 2382208 0.
Matrix 0 65 14780672 0.
Distributed Mesh 1 0 0 0.
Index Set 2 18 171852 0.
IS L to G Mapping 1 0 0 0.
Star Forest Graph 2 0 0 0.
Discrete System 1 0 0 0.
Vec Scatter 1 13 16432 0.
Preconditioner 1 9 9676 0.
Viewer 1 0 0 0.
--- Event Stage 1: First Solve
Krylov Solver 8 0 0 0.
Vector 152 104 2238504 0.
Matrix 148 83 22951356 0.
Matrix Coarsen 6 6 3816 0.
Index Set 128 112 590828 0.
Star Forest Graph 12 12 10368 0.
Vec Scatter 34 21 26544 0.
Preconditioner 8 0 0 0.
--- Event Stage 2: Remaining Solves
Vector 23892 23892 1302241424 0.
========================================================================================================================
Average time to get PetscTime(): 9.53674e-08
Average time for MPI_Barrier(): 3.45707e-05
Average time for zero size MPI_Send(): 1.60329e-05
#PETSc Option Table entries:
-gamg_est_ksp_type cg
-ksp_norm_type unpreconditioned
-ksp_type cg
-log_view
-mg_levels_esteig_ksp_max_it 10
-mg_levels_esteig_ksp_type cg
-mg_levels_ksp_max_it 1
-mg_levels_ksp_norm_type none
-mg_levels_ksp_type richardson
-mg_levels_pc_sor_its 1
-mg_levels_pc_type sor
-pc_gamg_type classical
-pc_type gamg
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure options: --known-level1-dcache-size=65536 --known-level1-dcache-linesize=64 --known-level1-dcache-assoc=2 --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2 --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8 --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8 --known-bits-per-byte=8 --known-memcmp-ok=1 --known-sizeof-MPI_Comm=4 --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1 --known-mpi-c-double-complex=1 --known-has-attribute-aligned=1 PETSC_ARCH=arch-linux-amd-opt --download-f2cblaslapack --with-mpi-dir=/cm/shared/apps/mvapich2/intel-17.0.1/2.0 --download-hypre --download-ml --with-fc=0 --with-debugging=0 COPTFLAGS=-O3 CXXOPTFLAGS=-O3 --with-batch --with-x --known-mpi-shared-libraries=1 --known-64-bit-blas-indices=4
-----------------------------------------
Libraries compiled on 2018-05-25 07:05:14 on node1-001
Machine characteristics: Linux-2.6.32-696.18.7.el6.x86_64-x86_64-with-redhat-6.6-Carbon
Using PETSc directory: /home/ritsat/beckerm/petsc
Using PETSc arch: arch-linux-amd-opt
-----------------------------------------
Using C compiler: /cm/shared/apps/mvapich2/intel-17.0.1/2.0/bin/mpicc -fPIC -wd1572 -O3
-----------------------------------------
Using include paths: -I/home/ritsat/beckerm/petsc/include -I/home/ritsat/beckerm/petsc/arch-linux-amd-opt/include -I/cm/shared/apps/mvapich2/intel-17.0.1/2.0/include
-----------------------------------------
Using C linker: /cm/shared/apps/mvapich2/intel-17.0.1/2.0/bin/mpicc
Using libraries: -Wl,-rpath,/home/ritsat/beckerm/petsc/arch-linux-amd-opt/lib -L/home/ritsat/beckerm/petsc/arch-linux-amd-opt/lib -lpetsc -Wl,-rpath,/home/ritsat/beckerm/petsc/arch-linux-amd-opt/lib -L/home/ritsat/beckerm/petsc/arch-linux-amd-opt/lib -lHYPRE -lml -lf2clapack -lf2cblas -lX11 -ldl
-----------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: wstest.zip
Type: application/zip
Size: 5833 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20180604/34557441/attachment-0001.zip>
More information about the petsc-users
mailing list