[petsc-users] PETSc (3.9.0) GAMG weak scaling test issue
"Alberto F. Martín"
amartin at cimne.upc.edu
Mon Nov 19 04:52:12 CST 2018
Dear Mark, Dear Matthew,
in order to discard load imbalance as the cause of the reported weak
scaling issue in the GAMG preconditioner
set-up stage (as you said, we were feeding GAMG with a suboptimal mesh
distribution, having empty processors,
among others), we simplified the weak scaling test by considering the
standard body-fitted trilinear (Q1) FE discretization
of the 3D Poisson problem on a unit cube discretized with a */uniform,
structured hexahedral mesh,/* /*partitioned*//*
*//*optimally (by hand) among processors*/, with a fixed load/core of
30**3 hexahedra/core. Thus, all processors have the same load
(up-to strong Dirichlet boundary conditions on the subdomains touching
the global boundary), and the edge-cut is minimum.
We used the following GAMG preconditioner options:
-pc_type gamg
-pc_gamg_type agg
-pc_gamg_est_ksp_type cg
-mg_levels_esteig_ksp_type cg
-mg_coarse_sub_pc_type cholesky
-mg_coarse_sub_pc_factor_mat_ordering_type nd
-pc_gamg_process_eq_limit 50
-pc_gamg_square_graph 10
-pc_gamg_agg_nsmooths 1
The results that we obtained for 48 (4x4x3 subdomains), 10,368
(24x24x18 subdomains), and 16,464
(28x28x21 subdomains) CPU cores are as follows:
**preconditioner set up**
[0.9844961860, *7.017674042*, *12.10154881*]
**PCG stage**
[0.5849160422, 1.515251888, 1.859617710]
**number of PCG iterations**
[9,14,15]
As you can observe, *there is still a significant time increase when
scaling the problem from 48 to 10K/16K MPI tasks**
**for the preconditioner setup stage. *This time increase is not as
significant for the PCG stage.**Please find attached the combined
output of -ksp_view and -log_view for these three points of the weak
scaling curve.
Given these results, I am starting to suspect that something within the
underlying software + hardware stack might be
responsible for this. I am using OpenMPI 1.10.7 + Intel compilers
version 18.0. The underlying supercomputer is MN-IV at
BSC (https://www.bsc.es/marenostrum/marenostrum/technical-information).
Have you ever conducted a weak scaling test
of GAMG with OpenMPI on a similar computer architecture? Can you share
your experience with us?
(versions tested, outcome, etc.)
We also tried an alternative MPI library, Intel(R) MPI Library for
Linux* OS, Version 2018 Update 4 Build 20180823 (id: 18555),
*without success. *For this MPI library, the preconditioner set-up stage
crashes (find attached stack frames, and internal MPI library
errors) for the largest two core counts (it did not crash for the 48 CPU
cores case), while it did not crash with OpenMPI 1.10.7.
Have you ever experienced errors like the ones
attached? Is there anyway to set up PETSc such that the subroutine that
crashes is replaced by an alternative implementation of
the same concept? (this would be just a workaround). It might be a BUG
in the Intel MPI library, although I cannot confirm it. We also got
these errors with the unfitted FEM+space-filling curves version of our code.
Thanks a lot for your help and valuable feedback!
Best regards,
Alberto.
On 08/11/18 17:29, Mark Adams wrote:
>
>
> I did not configured PETSc with ParMetis support. Should I?
>
> I figured it out when I tried to use "-pc_gamg_repartition". PETSc
> complained that it was not compiled with ParMetis support.
>
>
> You need ParMetis, or some parallel mesh partitioner, configured to
> use repartitioning. I would guess that "-pc_gamg_repartition" would
> not help and might hurt, because it just does the coarse grids, not
> the fine grid. But it is worth a try. Just configure with
> --download-parmetis
>
> The problem is that you are using space filling curves on the
> background grid and are getting empty processors. Right? The mesh
> setup phase is not super optimized, but your times
>
> And you said in your attachment that you added the near null space,
> but just the constant vector. I trust you mean the three translational
> rigid body modes. That is the default and so you should not see any
> difference. If you added one vector of all 1s then that would be bad.
> You also want the rotational rigid body modes. Now, you are converging
> pretty well and if your solution does not have much rotation in it the
> the rotational modes are not needed, but they are required for
> optimality in general.
>
--
Alberto F. Martín-Huertas
Senior Researcher, PhD. Computational Science
Centre Internacional de Mètodes Numèrics a l'Enginyeria (CIMNE)
Parc Mediterrani de la Tecnologia, UPC
Esteve Terradas 5, Building C3, Office 215,
08860 Castelldefels (Barcelona, Spain)
Tel.: (+34) 9341 34223
e-mail:amartin at cimne.upc.edu
FEMPAR project co-founder
web: http://www.fempar.org
________________
IMPORTANT NOTICE
All personal data contained on this mail will be processed confidentially and registered in a file property of CIMNE in
order to manage corporate communications. You may exercise the rights of access, rectification, erasure and object by
letter sent to Ed. C1 Campus Norte UPC. Gran Capitán s/n Barcelona.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181119/f900cf40/attachment-0001.html>
-------------- next part --------------
KSP Object: 48 MPI processes
type: cg
maximum iterations=500, initial guess is zero
tolerances: relative=1e-08, absolute=1e-50, divergence=10000.
left preconditioning
using UNPRECONDITIONED norm type for convergence test
PC Object: 48 MPI processes
type: gamg
type is MULTIPLICATIVE, levels=4 cycles=v
Cycles per PCApply=1
Using externally compute Galerkin coarse grid matrices
GAMG specific options
Threshold for dropping small values in graph on each level = 0. 0.
Threshold scaling factor for each level not specified = 1.
AGG specific options
Symmetric graph false
Number of levels to square graph 10
Number smoothing steps 1
Coarse grid solver -- level -------------------------------
KSP Object: (mg_coarse_) 48 MPI processes
type: preonly
maximum iterations=10000, initial guess is zero
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
left preconditioning
using NONE norm type for convergence test
PC Object: (mg_coarse_) 48 MPI processes
type: bjacobi
number of blocks = 48
Local solve is same for all blocks, in the following KSP and PC objects:
KSP Object: (mg_coarse_sub_) 1 MPI processes
type: preonly
maximum iterations=1, initial guess is zero
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
left preconditioning
using NONE norm type for convergence test
PC Object: (mg_coarse_sub_) 1 MPI processes
type: cholesky
out-of-place factorization
tolerance for zero pivot 2.22045e-14
matrix ordering: nd
factor fill ratio given 5., needed 1.
Factored matrix follows:
Mat Object: 1 MPI processes
type: seqsbaij
rows=6, cols=6
package used to perform factorization: petsc
total: nonzeros=21, allocated nonzeros=21
total number of mallocs used during MatSetValues calls =0
block size is 1
linear system matrix = precond matrix:
Mat Object: 1 MPI processes
type: seqaij
rows=6, cols=6
total: nonzeros=36, allocated nonzeros=36
total number of mallocs used during MatSetValues calls =0
using I-node routines: found 2 nodes, limit used is 5
linear system matrix = precond matrix:
Mat Object: 48 MPI processes
type: mpiaij
rows=6, cols=6
total: nonzeros=36, allocated nonzeros=36
total number of mallocs used during MatSetValues calls =0
using I-node (on process 0) routines: found 2 nodes, limit used is 5
Down solver (pre-smoother) on level 1 -------------------------------
KSP Object: (mg_levels_1_) 48 MPI processes
type: chebyshev
eigenvalue estimates used: min = 0.135917, max = 1.49508
eigenvalues estimate via cg min 0.269407, max 1.35917
eigenvalues estimated using cg with translations [0. 0.1; 0. 1.1]
KSP Object: (mg_levels_1_esteig_) 48 MPI processes
type: cg
maximum iterations=10, initial guess is zero
tolerances: relative=1e-12, absolute=1e-50, divergence=10000.
left preconditioning
using PRECONDITIONED norm type for convergence test
estimating eigenvalues using noisy right hand side
maximum iterations=2, nonzero initial guess
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
left preconditioning
using NONE norm type for convergence test
PC Object: (mg_levels_1_) 48 MPI processes
type: sor
type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
linear system matrix = precond matrix:
Mat Object: 48 MPI processes
type: mpiaij
rows=359, cols=359
total: nonzeros=17751, allocated nonzeros=17751
total number of mallocs used during MatSetValues calls =0
not using I-node (on process 0) routines
Up solver (post-smoother) same as down solver (pre-smoother)
Down solver (pre-smoother) on level 2 -------------------------------
KSP Object: (mg_levels_2_) 48 MPI processes
type: chebyshev
eigenvalue estimates used: min = 0.137555, max = 1.5131
eigenvalues estimate via cg min 0.0576414, max 1.37555
eigenvalues estimated using cg with translations [0. 0.1; 0. 1.1]
KSP Object: (mg_levels_2_esteig_) 48 MPI processes
type: cg
maximum iterations=10, initial guess is zero
tolerances: relative=1e-12, absolute=1e-50, divergence=10000.
left preconditioning
using PRECONDITIONED norm type for convergence test
estimating eigenvalues using noisy right hand side
maximum iterations=2, nonzero initial guess
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
left preconditioning
using NONE norm type for convergence test
PC Object: (mg_levels_2_) 48 MPI processes
type: sor
type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
linear system matrix = precond matrix:
Mat Object: 48 MPI processes
type: mpiaij
rows=27123, cols=27123
total: nonzeros=1144833, allocated nonzeros=1144833
total number of mallocs used during MatSetValues calls =0
using nonscalable MatPtAP() implementation
not using I-node (on process 0) routines
Up solver (post-smoother) same as down solver (pre-smoother)
Down solver (pre-smoother) on level 3 -------------------------------
KSP Object: (mg_levels_3_) 48 MPI processes
type: chebyshev
eigenvalue estimates used: min = 0.136454, max = 1.50099
eigenvalues estimate via cg min 0.0376067, max 1.36454
eigenvalues estimated using cg with translations [0. 0.1; 0. 1.1]
KSP Object: (mg_levels_3_esteig_) 48 MPI processes
type: cg
maximum iterations=10, initial guess is zero
tolerances: relative=1e-12, absolute=1e-50, divergence=10000.
left preconditioning
using PRECONDITIONED norm type for convergence test
estimating eigenvalues using noisy right hand side
maximum iterations=2, nonzero initial guess
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
left preconditioning
using NONE norm type for convergence test
PC Object: (mg_levels_3_) 48 MPI processes
type: sor
type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
linear system matrix = precond matrix:
Mat Object: 48 MPI processes
type: mpiaij
rows=1260329, cols=1260329
total: nonzeros=33396625, allocated nonzeros=252065800
total number of mallocs used during MatSetValues calls =0
not using I-node (on process 0) routines
Up solver (post-smoother) same as down solver (pre-smoother)
linear system matrix = precond matrix:
Mat Object: 48 MPI processes
type: mpiaij
rows=1260329, cols=1260329
total: nonzeros=33396625, allocated nonzeros=252065800
total number of mallocs used during MatSetValues calls =0
not using I-node (on process 0) routines
************************************************************************************************************************
*** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document ***
************************************************************************************************************************
---------------------------------------------- PETSc Performance Summary: ----------------------------------------------
/gpfs/scratch/upc26/upc26229/build_rel_fempar_cell_agg_ompi/FEMPAR/bin/par_test_poisson_unfitted on a arch-linux2-c-opt named s05r2b62 with 48 processors, by upc26229 Fri Nov 16 09:37:39 2018
Using Petsc Release Version 3.9.0, Apr, 07, 2018
Max Max/Min Avg Total
Time (sec): 2.318e+01 1.00000 2.318e+01
Objects: 1.250e+03 1.00241 1.247e+03
Flop: 8.936e+08 1.13741 8.557e+08 4.107e+10
Flop/sec: 3.855e+07 1.13741 3.692e+07 1.772e+09
MPI Messages: 2.001e+04 3.63905 1.109e+04 5.322e+05
MPI Message Lengths: 7.575e+07 2.25574 4.717e+03 2.510e+09
MPI Reductions: 1.773e+03 1.00000
Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
e.g., VecAXPY() for real vectors of length N --> 2N flop
and VecAXPY() for complex vectors of length N --> 8N flop
Summary of Stages: ----- Time ------ ----- Flop ----- --- Messages --- -- Message Lengths -- -- Reductions --
Avg %Total Avg %Total counts %Total Avg %Total counts %Total
0: Main Stage: 2.3173e+01 100.0% 4.1074e+10 100.0% 5.322e+05 100.0% 4.717e+03 100.0% 1.759e+03 99.2%
------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
Count: number of times phase was executed
Time and Flop: Max - maximum over all processors
Ratio - ratio of maximum to minimum over all processors
Mess: number of messages sent
Avg. len: average message length (bytes)
Reduct: number of global reductions
Global: entire computation
Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
%T - percent time in this phase %F - percent flop in this phase
%M - percent messages in this phase %L - percent message lengths in this phase
%R - percent reductions in this phase
Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event Count Time (sec) Flop --- Global --- --- Stage --- Total
Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------
--- Event Stage 0: Main Stage
BuildTwoSided 9 1.0 8.8800e-03 4.2 0.00e+00 0.0 2.0e+03 8.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
BuildTwoSidedF 126 1.0 4.4196e-01 5.4 0.00e+00 0.0 1.6e+04 8.5e+04 0.0e+00 1 0 3 55 0 1 0 3 55 0 0
VecMDot 90 1.0 2.2851e-02 3.4 9.11e+06 1.1 0.0e+00 0.0e+00 9.0e+01 0 1 0 0 5 0 1 0 0 5 18597
VecTDot 243 1.0 9.9924e-02 9.2 6.39e+06 1.1 0.0e+00 0.0e+00 2.4e+02 0 1 0 0 14 0 1 0 0 14 2986
VecNorm 228 1.0 3.5723e-02 1.8 5.26e+06 1.1 0.0e+00 0.0e+00 2.3e+02 0 1 0 0 13 0 1 0 0 13 6875
VecScale 99 1.0 7.8892e-0313.7 9.11e+05 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 5387
VecCopy 114 1.0 2.5488e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecSet 483 1.0 1.8528e-03 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecAXPY 243 1.0 6.3856e-03 1.2 6.39e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 46727
VecAYPX 753 1.0 2.2839e-02 1.3 1.02e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 20918
VecAXPBYCZ 324 1.0 1.2422e-02 1.1 1.49e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 55982
VecMAXPY 99 1.0 9.3288e-03 2.0 1.08e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 53838
VecAssemblyBegin 63 1.0 1.4764e-02 2.5 0.00e+00 0.0 1.5e+03 3.1e+03 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecAssemblyEnd 63 1.0 3.3262e-04 3.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecPointwiseMult 99 1.0 3.4623e-03 1.5 9.11e+05 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 12275
VecScatterBegin 933 1.0 3.9603e-02 2.8 0.00e+00 0.0 3.9e+05 1.3e+03 0.0e+00 0 0 74 21 0 0 0 74 21 0 0
VecScatterEnd 933 1.0 1.1539e-01 4.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecSetRandom 9 1.0 2.5657e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecNormalize 99 1.0 2.0392e-02 2.0 2.73e+06 1.1 0.0e+00 0.0e+00 9.9e+01 0 0 0 0 6 0 0 0 0 6 6252
MatMult 693 1.0 7.2101e-01 1.2 3.68e+08 1.1 3.2e+05 1.4e+03 0.0e+00 3 41 60 17 0 3 41 60 17 0 23339
MatMultAdd 81 1.0 3.0372e-02 1.6 6.32e+06 1.2 2.1e+04 1.5e+02 0.0e+00 0 1 4 0 0 0 1 4 0 0 9453
MatMultTranspose 81 1.0 6.0862e-02 3.8 6.32e+06 1.2 2.1e+04 1.5e+02 0.0e+00 0 1 4 0 0 0 1 4 0 0 4717
MatSolve 27 0.0 7.1588e-05 0.0 1.78e+03 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 25
MatSOR 585 1.0 9.6473e-01 1.1 2.73e+08 1.1 0.0e+00 0.0e+00 0.0e+00 4 31 0 0 0 4 31 0 0 0 13218
MatCholFctrSym 3 1.0 8.9991e-03497.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatCholFctrNum 3 1.0 6.3463e-031399.7 1.80e+01 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatConvert 9 1.0 3.5641e-02 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatScale 27 1.0 2.1565e-02 1.2 5.23e+06 1.1 4.1e+03 1.3e+03 0.0e+00 0 1 1 0 0 0 1 1 0 0 11095
MatResidual 81 1.0 8.1534e-02 1.4 4.08e+07 1.1 3.7e+04 1.3e+03 0.0e+00 0 5 7 2 0 0 5 7 2 0 22889
MatAssemblyBegin 252 1.0 5.7321e-01 1.5 0.00e+00 0.0 1.5e+04 9.3e+04 0.0e+00 2 0 3 55 0 2 0 3 55 0 0
MatAssemblyEnd 252 1.0 5.3497e-01 1.1 0.00e+00 0.0 4.1e+04 5.6e+02 4.8e+02 2 0 8 1 27 2 0 8 1 27 0
MatGetRow 248445 1.1 2.9776e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatGetRowIJ 3 0.0 1.2791e-02 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatCreateSubMat 12 1.0 2.6623e-02 1.0 0.00e+00 0.0 2.5e+03 3.8e+02 1.9e+02 0 0 0 0 11 0 0 0 0 11 0
MatGetOrdering 3 0.0 1.5566e-02 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatCoarsen 9 1.0 4.3990e-02 1.5 0.00e+00 0.0 2.9e+04 2.5e+03 3.6e+01 0 0 5 3 2 0 0 5 3 2 0
MatZeroEntries 18 1.0 5.3911e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatView 21 1.4 6.7606e-02 2.5 0.00e+00 0.0 0.0e+00 0.0e+00 1.5e+01 0 0 0 0 1 0 0 0 0 1 0
MatAXPY 9 1.0 2.6150e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatMatMult 9 1.0 1.2417e-01 1.2 4.53e+06 1.1 2.3e+04 9.4e+02 1.1e+02 1 1 4 1 6 1 1 4 1 6 1670
MatMatMultSym 9 1.0 1.0047e-01 1.2 0.00e+00 0.0 1.9e+04 8.7e+02 1.1e+02 0 0 4 1 6 0 0 4 1 6 0
MatMatMultNum 9 1.0 2.3698e-02 1.0 4.53e+06 1.1 4.1e+03 1.3e+03 0.0e+00 0 1 1 0 0 0 1 1 0 0 8750
MatPtAP 9 1.0 2.8118e-01 1.0 2.87e+07 1.2 3.6e+04 2.7e+03 1.4e+02 1 3 7 4 8 1 3 7 4 8 4543
MatPtAPSymbolic 9 1.0 2.0049e-01 1.0 0.00e+00 0.0 2.2e+04 3.0e+03 6.3e+01 1 0 4 3 4 1 0 4 3 4 0
MatPtAPNumeric 9 1.0 8.0681e-02 1.0 2.87e+07 1.2 1.4e+04 2.2e+03 7.2e+01 0 3 3 1 4 0 3 3 1 4 15832
MatTrnMatMult 9 1.0 1.9582e+00 1.0 1.37e+08 1.2 2.6e+04 6.3e+04 1.4e+02 8 15 5 66 8 8 15 5 66 8 3150
MatTrnMatMultSym 9 1.0 9.0854e-01 1.0 0.00e+00 0.0 1.2e+04 3.5e+04 6.6e+01 4 0 2 17 4 4 0 2 17 4 0
MatTrnMatMultNum 9 1.0 1.0519e+00 1.0 1.37e+08 1.2 1.4e+04 8.7e+04 7.2e+01 5 15 3 49 4 5 15 3 49 4 5864
MatGetLocalMat 36 1.0 5.1365e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatGetBrAoCol 27 1.0 4.9700e-02 2.0 0.00e+00 0.0 2.8e+04 2.7e+03 0.0e+00 0 0 5 3 0 0 0 5 3 0 0
KSPGMRESOrthog 90 1.0 3.1736e-02 2.3 1.82e+07 1.1 0.0e+00 0.0e+00 9.0e+01 0 2 0 0 5 0 2 0 0 5 26782
KSPSetUp 36 1.0 1.1924e-02 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 2.4e+01 0 0 0 0 1 0 0 0 0 1 0
KSPSolve 3 1.0 1.7241e+00 1.0 6.50e+08 1.1 3.2e+05 1.2e+03 3.7e+02 7 73 60 15 21 7 73 60 15 21 17425
PCGAMGGraph_AGG 9 1.0 3.1316e-01 1.0 4.53e+06 1.1 1.2e+04 8.7e+02 1.1e+02 1 1 2 0 6 1 1 2 0 6 662
PCGAMGCoarse_AGG 9 1.0 2.0449e+00 1.0 1.37e+08 1.2 7.6e+04 2.4e+04 2.1e+02 9 15 14 71 12 9 15 14 71 12 3017
PCGAMGProl_AGG 9 1.0 1.2541e-01 1.1 0.00e+00 0.0 1.5e+04 1.9e+03 1.4e+02 1 0 3 1 8 1 0 3 1 8 0
PCGAMGPOpt_AGG 9 1.0 3.4309e-01 1.0 7.33e+07 1.1 6.4e+04 1.2e+03 3.7e+02 1 8 12 3 21 1 8 12 3 21 9849
GAMG: createProl 9 1.0 2.8587e+00 1.0 2.15e+08 1.2 1.7e+05 1.1e+04 8.3e+02 12 24 31 76 47 12 24 31 76 47 3412
Graph 18 1.0 3.0684e-01 1.0 4.53e+06 1.1 1.2e+04 8.7e+02 1.1e+02 1 1 2 0 6 1 1 2 0 6 676
MIS/Agg 9 1.0 4.4147e-02 1.4 0.00e+00 0.0 2.9e+04 2.5e+03 3.6e+01 0 0 5 3 2 0 0 5 3 2 0
SA: col data 9 1.0 1.2357e-02 1.1 0.00e+00 0.0 8.2e+03 2.8e+03 3.6e+01 0 0 2 1 2 0 0 2 1 2 0
SA: frmProl0 9 1.0 1.0247e-01 1.0 0.00e+00 0.0 6.7e+03 6.5e+02 7.2e+01 0 0 1 0 4 0 0 1 0 4 0
SA: smooth 9 1.0 1.6341e-01 1.1 5.23e+06 1.1 2.3e+04 9.4e+02 1.3e+02 1 1 4 1 7 1 1 4 1 7 1464
GAMG: partLevel 9 1.0 3.2724e-01 1.0 2.87e+07 1.2 3.9e+04 2.5e+03 4.4e+02 1 3 7 4 25 1 3 7 4 25 3903
repartition 6 1.0 2.7110e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 3.6e+01 0 0 0 0 2 0 0 0 0 2 0
Invert-Sort 6 1.0 7.3382e-03 5.8 0.00e+00 0.0 0.0e+00 0.0e+00 2.4e+01 0 0 0 0 1 0 0 0 0 1 0
Move A 6 1.0 2.5487e-02 1.3 0.00e+00 0.0 1.1e+03 8.5e+02 1.0e+02 0 0 0 0 6 0 0 0 0 6 0
Move P 6 1.0 7.7925e-03 1.0 0.00e+00 0.0 1.4e+03 3.2e+01 1.0e+02 0 0 0 0 6 0 0 0 0 6 0
PCSetUp 6 1.0 3.2475e+00 1.0 2.44e+08 1.2 2.0e+05 9.7e+03 1.3e+03 14 27 38 79 75 14 27 38 79 75 3397
PCSetUpOnBlocks 27 1.0 1.6300e-02 1.6 1.80e+01 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
PCApply 27 1.0 1.6327e+00 1.0 6.03e+08 1.1 3.0e+05 1.1e+03 2.9e+02 7 68 57 14 16 7 68 57 14 16 17066
SFSetGraph 9 1.0 7.1678e-032989.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
SFSetUp 9 1.0 1.6463e-02 2.2 0.00e+00 0.0 6.1e+03 1.9e+03 0.0e+00 0 0 1 0 0 0 0 1 0 0 0
SFBcastBegin 54 1.0 3.5743e-03 3.4 0.00e+00 0.0 2.3e+04 2.6e+03 0.0e+00 0 0 4 2 0 0 0 4 2 0 0
SFBcastEnd 54 1.0 3.3646e-03 2.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
------------------------------------------------------------------------------------------------------------------------
Memory usage is given in bytes:
Object Type Creations Destructions Memory Descendants' Mem.
Reports information only for process 0.
--- Event Stage 0: Main Stage
Vector 510 510 27602496 0.
Matrix 348 348 622432056 0.
Matrix Coarsen 9 9 6156 0.
Index Set 213 213 558288 0.
Vec Scatter 75 75 105384 0.
Krylov Solver 36 36 314928 0.
Preconditioner 27 27 29544 0.
Viewer 5 4 3584 0.
PetscRandom 18 18 12492 0.
Star Forest Graph 9 9 8496 0.
========================================================================================================================
Average time to get PetscTime(): 4.56115e-08
Average time for MPI_Barrier(): 5.97197e-06
Average time for zero size MPI_Send(): 3.71554e-06
#PETSc Option Table entries:
--prefix popcorn3d_full_l3_s1
-beta 7.0
-betaest .true.
-check .false.
-datadt data_distribution_fully_assembled
-dm 3
-in_space .true.
-ksp_converged_reason
-ksp_max_it 500
-ksp_monitor
-ksp_norm_type unpreconditioned
-ksp_rtol 1.0e-8
-ksp_type cg
-ksp_view
-l 1
-levelset popcorn
-levelsettol 0.0
-log_view
-lsdom -0.1
-mg_coarse_sub_pc_factor_mat_ordering_type nd
-mg_coarse_sub_pc_type cholesky
-mg_levels_esteig_ksp_type cg
-n 120
-no_signal_handler
-nruns 3
-pc_gamg_agg_nsmooths 1
-pc_gamg_est_ksp_type cg
-pc_gamg_process_eq_limit 50
-pc_gamg_square_graph 10
-pc_gamg_type agg
-pc_type gamg
-petscrc /gpfs/scratch/upc26/upc26229/NEW_STUFF/time_par_cell_agg_ompi.paper/petscrc-0
-tt 1
-uagg .false.
-wsolution .false.
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 8
Configure options: --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 -with-blaslapack-dir=/apps/INTEL/2017.4/mkl --with-debugging=0 --with-x=0 --with-shared-libraries=1 --with-mpi=1 --with-64-bit-indices --download-hypre=../v2.14.0.tar.gz
-----------------------------------------
Libraries compiled on 2018-11-07 17:23:07 on login1
Machine characteristics: Linux-4.4.120-92.70-default-x86_64-with-SuSE-12-x86_64
Using PETSc directory: /gpfs/scratch/upc26/upc26229/petsc_cell_agg_openmpi/release/petsc-3.9.0
Using PETSc arch: arch-linux2-c-opt
-----------------------------------------
Using C compiler: mpicc -fPIC -wd1572 -g -O3
Using Fortran compiler: mpif90 -fPIC -g -O3
-----------------------------------------
Using include paths: -I/gpfs/scratch/upc26/upc26229/petsc_cell_agg_openmpi/release/petsc-3.9.0/include -I/gpfs/scratch/upc26/upc26229/petsc_cell_agg_openmpi/release/petsc-3.9.0/arch-linux2-c-opt/include
-----------------------------------------
Using C linker: mpicc
Using Fortran linker: mpif90
Using libraries: -Wl,-rpath,/gpfs/scratch/upc26/upc26229/petsc_cell_agg_openmpi/release/petsc-3.9.0/arch-linux2-c-opt/lib -L/gpfs/scratch/upc26/upc26229/petsc_cell_agg_openmpi/release/petsc-3.9.0/arch-linux2-c-opt/lib -lpetsc -Wl,-rpath,/gpfs/scratch/upc26/upc26229/petsc_cell_agg_openmpi/release/petsc-3.9.0/arch-linux2-c-opt/lib -L/gpfs/scratch/upc26/upc26229/petsc_cell_agg_openmpi/release/petsc-3.9.0/arch-linux2-c-opt/lib -Wl,-rpath,/apps/INTEL/2017.4/mkl/lib/intel64 -L/apps/INTEL/2017.4/mkl/lib/intel64 -Wl,-rpath,/usr/mpi/intel/openmpi-1.10.4-hfi/lib64 -L/usr/mpi/intel/openmpi-1.10.4-hfi/lib64 -Wl,-rpath,/gpfs/apps/MN4/INTEL/2018.0.128/compilers_and_libraries_2018.0.128/linux/compiler/lib/intel64_lin -L/gpfs/apps/MN4/INTEL/2018.0.128/compilers_and_libraries_2018.0.128/linux/compiler/lib/intel64_lin -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/4.8 -L/usr/lib64/gcc/x86_64-suse-linux/4.8 -Wl,-rpath,/usr/x86_64-suse-linux/lib -L/usr/x86_64-suse-linux/lib -lHYPRE -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lstdc++ -ldl -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lifport -lifcoremt_pic -limf -lsvml -lm -lipgo -lirc -lpthread -lgcc_s -lirc_s -lstdc++ -ldl
-----------------------------------------
Ending run at vie nov 16 09:37:40 CET 2018
-------------- next part --------------
Linear solve converged due to CONVERGED_RTOL iterations 14
KSP Object: 10368 MPI processes
type: cg
maximum iterations=500, initial guess is zero
tolerances: relative=1e-08, absolute=1e-50, divergence=10000.
left preconditioning
using UNPRECONDITIONED norm type for convergence test
PC Object: 10368 MPI processes
type: gamg
type is MULTIPLICATIVE, levels=5 cycles=v
Cycles per PCApply=1
Using externally compute Galerkin coarse grid matrices
GAMG specific options
Threshold for dropping small values in graph on each level = 0. 0. 0.
Threshold scaling factor for each level not specified = 1.
AGG specific options
Symmetric graph false
Number of levels to square graph 10
Number smoothing steps 1
Coarse grid solver -- level -------------------------------
KSP Object: (mg_coarse_) 10368 MPI processes
type: preonly
maximum iterations=10000, initial guess is zero
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
left preconditioning
using NONE norm type for convergence test
PC Object: (mg_coarse_) 10368 MPI processes
type: bjacobi
number of blocks = 10368
Local solve is same for all blocks, in the following KSP and PC objects:
KSP Object: (mg_coarse_sub_) 1 MPI processes
type: preonly
maximum iterations=1, initial guess is zero
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
left preconditioning
using NONE norm type for convergence test
PC Object: (mg_coarse_sub_) 1 MPI processes
type: cholesky
out-of-place factorization
tolerance for zero pivot 2.22045e-14
matrix ordering: nd
factor fill ratio given 5., needed 1.
Factored matrix follows:
Mat Object: 1 MPI processes
type: seqsbaij
rows=7, cols=7
package used to perform factorization: petsc
total: nonzeros=28, allocated nonzeros=28
total number of mallocs used during MatSetValues calls =0
block size is 1
linear system matrix = precond matrix:
Mat Object: 1 MPI processes
type: seqaij
rows=7, cols=7
total: nonzeros=49, allocated nonzeros=49
total number of mallocs used during MatSetValues calls =0
using I-node routines: found 2 nodes, limit used is 5
linear system matrix = precond matrix:
Mat Object: 10368 MPI processes
type: mpiaij
rows=7, cols=7
total: nonzeros=49, allocated nonzeros=49
total number of mallocs used during MatSetValues calls =0
using I-node (on process 0) routines: found 2 nodes, limit used is 5
Down solver (pre-smoother) on level 1 -------------------------------
KSP Object: (mg_levels_1_) 10368 MPI processes
type: chebyshev
eigenvalue estimates used: min = 0.132992, max = 1.46291
eigenvalues estimate via cg min 0.252003, max 1.32992
eigenvalues estimated using cg with translations [0. 0.1; 0. 1.1]
KSP Object: (mg_levels_1_esteig_) 10368 MPI processes
type: cg
maximum iterations=10, initial guess is zero
tolerances: relative=1e-12, absolute=1e-50, divergence=10000.
left preconditioning
using PRECONDITIONED norm type for convergence test
estimating eigenvalues using noisy right hand side
maximum iterations=2, nonzero initial guess
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
left preconditioning
using NONE norm type for convergence test
PC Object: (mg_levels_1_) 10368 MPI processes
type: sor
type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
linear system matrix = precond matrix:
Mat Object: 10368 MPI processes
type: mpiaij
rows=598, cols=598
total: nonzeros=35260, allocated nonzeros=35260
total number of mallocs used during MatSetValues calls =0
not using I-node (on process 0) routines
Up solver (post-smoother) same as down solver (pre-smoother)
Down solver (pre-smoother) on level 2 -------------------------------
KSP Object: (mg_levels_2_) 10368 MPI processes
type: chebyshev
eigenvalue estimates used: min = 0.147257, max = 1.61982
eigenvalues estimate via cg min 0.0457702, max 1.47257
eigenvalues estimated using cg with translations [0. 0.1; 0. 1.1]
KSP Object: (mg_levels_2_esteig_) 10368 MPI processes
type: cg
maximum iterations=10, initial guess is zero
tolerances: relative=1e-12, absolute=1e-50, divergence=10000.
left preconditioning
using PRECONDITIONED norm type for convergence test
estimating eigenvalues using noisy right hand side
maximum iterations=2, nonzero initial guess
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
left preconditioning
using NONE norm type for convergence test
PC Object: (mg_levels_2_) 10368 MPI processes
type: sor
type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
linear system matrix = precond matrix:
Mat Object: 10368 MPI processes
type: mpiaij
rows=72415, cols=72415
total: nonzeros=5807437, allocated nonzeros=5807437
total number of mallocs used during MatSetValues calls =0
not using I-node (on process 0) routines
Up solver (post-smoother) same as down solver (pre-smoother)
Down solver (pre-smoother) on level 3 -------------------------------
KSP Object: (mg_levels_3_) 10368 MPI processes
type: chebyshev
eigenvalue estimates used: min = 0.138335, max = 1.52169
eigenvalues estimate via cg min 0.0338709, max 1.38335
eigenvalues estimated using cg with translations [0. 0.1; 0. 1.1]
KSP Object: (mg_levels_3_esteig_) 10368 MPI processes
type: cg
maximum iterations=10, initial guess is zero
tolerances: relative=1e-12, absolute=1e-50, divergence=10000.
left preconditioning
using PRECONDITIONED norm type for convergence test
estimating eigenvalues using noisy right hand side
maximum iterations=2, nonzero initial guess
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
left preconditioning
using NONE norm type for convergence test
PC Object: (mg_levels_3_) 10368 MPI processes
type: sor
type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
linear system matrix = precond matrix:
Mat Object: 10368 MPI processes
type: mpiaij
rows=5775893, cols=5775893
total: nonzeros=263455933, allocated nonzeros=263455933
total number of mallocs used during MatSetValues calls =0
using nonscalable MatPtAP() implementation
not using I-node (on process 0) routines
Up solver (post-smoother) same as down solver (pre-smoother)
Down solver (pre-smoother) on level 4 -------------------------------
KSP Object: (mg_levels_4_) 10368 MPI processes
type: chebyshev
eigenvalue estimates used: min = 0.136582, max = 1.5024
eigenvalues estimate via cg min 0.0328449, max 1.36582
eigenvalues estimated using cg with translations [0. 0.1; 0. 1.1]
KSP Object: (mg_levels_4_esteig_) 10368 MPI processes
type: cg
maximum iterations=10, initial guess is zero
tolerances: relative=1e-12, absolute=1e-50, divergence=10000.
left preconditioning
using PRECONDITIONED norm type for convergence test
estimating eigenvalues using noisy right hand side
maximum iterations=2, nonzero initial guess
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
left preconditioning
using NONE norm type for convergence test
PC Object: (mg_levels_4_) 10368 MPI processes
type: sor
type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
linear system matrix = precond matrix:
Mat Object: 10368 MPI processes
type: mpiaij
rows=278641979, cols=278641979
total: nonzeros=7500100375, allocated nonzeros=55728395800
total number of mallocs used during MatSetValues calls =0
not using I-node (on process 0) routines
Up solver (post-smoother) same as down solver (pre-smoother)
linear system matrix = precond matrix:
Mat Object: 10368 MPI processes
type: mpiaij
rows=278641979, cols=278641979
total: nonzeros=7500100375, allocated nonzeros=55728395800
total number of mallocs used during MatSetValues calls =0
not using I-node (on process 0) routines
************************************************************************************************************************
*** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document ***
************************************************************************************************************************
---------------------------------------------- PETSc Performance Summary: ----------------------------------------------
/gpfs/scratch/upc26/upc26229/build_rel_fempar_cell_agg_ompi/FEMPAR/bin/par_test_poisson_unfitted on a arch-linux2-c-opt named s08r2b08 with 10368 processors, by upc26229 Fri Nov 16 15:26:59 2018
Using Petsc Release Version 3.9.0, Apr, 07, 2018
Max Max/Min Avg Total
Time (sec): 4.915e+01 1.00001 4.915e+01
Objects: 1.658e+03 1.00181 1.655e+03
Flop: 1.198e+09 1.14246 1.183e+09 1.226e+13
Flop/sec: 2.436e+07 1.14246 2.406e+07 2.494e+11
MPI Messages: 1.789e+05 25.18126 2.640e+04 2.738e+08
MPI Message Lengths: 7.153e+07 2.83776 2.085e+03 5.708e+11
MPI Reductions: 2.568e+03 1.00000
Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
e.g., VecAXPY() for real vectors of length N --> 2N flop
and VecAXPY() for complex vectors of length N --> 8N flop
Summary of Stages: ----- Time ------ ----- Flop ----- --- Messages --- -- Message Lengths -- -- Reductions --
Avg %Total Avg %Total counts %Total Avg %Total counts %Total
0: Main Stage: 4.9150e+01 100.0% 1.2260e+13 100.0% 2.738e+08 100.0% 2.085e+03 100.0% 2.554e+03 99.5%
------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
Count: number of times phase was executed
Time and Flop: Max - maximum over all processors
Ratio - ratio of maximum to minimum over all processors
Mess: number of messages sent
Avg. len: average message length (bytes)
Reduct: number of global reductions
Global: entire computation
Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
%T - percent time in this phase %F - percent flop in this phase
%M - percent messages in this phase %L - percent message lengths in this phase
%R - percent reductions in this phase
Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event Count Time (sec) Flop --- Global --- --- Stage --- Total
Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------
--- Event Stage 0: Main Stage
BuildTwoSided 12 1.0 8.2886e-02 3.2 0.00e+00 0.0 8.4e+05 8.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
BuildTwoSidedF 174 1.0 2.7580e+00 1.9 0.00e+00 0.0 3.7e+06 1.4e+04 0.0e+00 4 0 1 9 0 4 0 1 9 0 0
VecMDot 120 1.0 2.6712e-01 1.3 9.24e+06 1.1 0.0e+00 0.0e+00 1.2e+02 0 1 0 0 5 0 1 0 0 5 351442
VecTDot 336 1.0 9.2127e-01 1.3 8.06e+06 1.1 0.0e+00 0.0e+00 3.4e+02 2 1 0 0 13 2 1 0 0 13 89719
VecNorm 309 1.0 7.2406e-01 1.2 6.12e+06 1.1 0.0e+00 0.0e+00 3.1e+02 1 1 0 0 12 1 1 0 0 12 86499
VecScale 132 1.0 1.2661e-0242.6 9.24e+05 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 741513
VecCopy 210 1.0 9.9707e-03 3.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecSet 843 1.0 6.0338e-03 4.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecAXPY 336 1.0 8.0444e-0211.5 8.06e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 1027513
VecAYPX 1491 1.0 3.7490e-02 2.3 1.54e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 4176676
VecAXPBYCZ 672 1.0 2.7469e-02 1.8 2.35e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 8699658
VecMAXPY 132 1.0 1.9974e-02 4.4 1.09e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 5554863
VecAssemblyBegin 84 1.0 4.3977e-01 2.0 0.00e+00 0.0 5.0e+05 2.6e+03 0.0e+00 1 0 0 0 0 1 0 0 0 0 0
VecAssemblyEnd 84 1.0 4.2681e-04 4.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecPointwiseMult 132 1.0 4.6056e-03 2.7 9.24e+05 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 2038447
VecScatterBegin 1731 1.0 6.6298e-02 3.3 0.00e+00 0.0 2.1e+08 1.0e+03 0.0e+00 0 0 76 37 0 0 0 76 37 0 0
VecScatterEnd 1731 1.0 1.1476e+00 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0
VecSetRandom 12 1.0 8.2220e-03 4.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecNormalize 132 1.0 3.1955e-01 1.4 2.77e+06 1.1 0.0e+00 0.0e+00 1.3e+02 1 0 0 0 5 1 0 0 0 5 88139
MatMult 1290 1.0 1.5137e+00 1.4 5.26e+08 1.2 1.7e+08 1.1e+03 0.0e+00 3 44 63 32 0 3 44 63 32 0 3552751
MatMultAdd 168 1.0 4.5490e-01 5.9 9.86e+06 1.2 1.2e+07 1.2e+02 0.0e+00 1 1 4 0 0 1 1 4 0 0 221941
MatMultTranspose 168 1.0 2.4255e-01 7.0 9.86e+06 1.2 1.2e+07 1.2e+02 0.0e+00 0 1 4 0 0 0 1 4 0 0 416246
MatSolve 42 0.0 9.4218e-05 0.0 3.82e+03 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 41
MatSOR 1140 1.0 1.4214e+00 1.2 4.11e+08 1.1 0.0e+00 0.0e+00 0.0e+00 3 34 0 0 0 3 34 0 0 0 2897344
MatCholFctrSym 3 1.0 2.4699e-021549.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatCholFctrNum 3 1.0 2.2717e-026755.8 2.10e+01 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatConvert 12 1.0 4.4884e-02 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatScale 36 1.0 3.6266e-0119.9 5.26e+06 1.2 1.5e+06 1.0e+03 0.0e+00 0 0 1 0 0 0 0 1 0 0 148427
MatResidual 168 1.0 2.3626e-01 1.9 6.38e+07 1.2 2.2e+07 1.0e+03 0.0e+00 0 5 8 4 0 0 5 8 4 0 2762345
MatAssemblyBegin 321 1.0 2.4651e+00 2.1 0.00e+00 0.0 3.2e+06 1.6e+04 0.0e+00 4 0 1 9 0 4 0 1 9 0 0
MatAssemblyEnd 321 1.0 5.1716e+00 1.0 0.00e+00 0.0 1.6e+07 4.4e+02 6.5e+02 10 0 6 1 25 10 0 6 1 25 0
MatGetRow 251901 1.1 5.1874e-02 2.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatGetRowIJ 3 0.0 5.2588e-03 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatCreateSubMat 18 1.0 2.6091e+00 1.0 0.00e+00 0.0 8.9e+05 3.5e+02 2.9e+02 5 0 0 0 11 5 0 0 0 11 0
MatGetOrdering 3 0.0 1.3637e-02 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatCoarsen 12 1.0 3.1793e-01 1.2 0.00e+00 0.0 2.4e+07 1.0e+03 2.0e+02 1 0 9 4 8 1 0 9 4 8 0
MatZeroEntries 18 1.0 1.3054e-03 6.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatView 24 1.3 2.5545e-01 4.5 0.00e+00 0.0 0.0e+00 0.0e+00 1.8e+01 0 0 0 0 1 0 0 0 0 1 0
MatAXPY 12 1.0 1.8095e-01 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatMatMult 12 1.0 2.0949e+00 1.1 4.56e+06 1.2 8.7e+06 7.4e+02 1.5e+02 4 0 3 1 6 4 0 3 1 6 22252
MatMatMultSym 12 1.0 1.8619e+00 1.0 0.00e+00 0.0 7.1e+06 6.8e+02 1.4e+02 4 0 3 1 6 4 0 3 1 6 0
MatMatMultNum 12 1.0 9.4086e-02 1.4 4.56e+06 1.2 1.5e+06 1.0e+03 0.0e+00 0 0 1 0 0 0 0 1 0 0 495466
MatPtAP 12 1.0 2.7205e+00 1.0 2.90e+07 1.2 1.3e+07 2.2e+03 1.8e+02 6 2 5 5 7 6 2 5 5 7 108153
MatPtAPSymbolic 12 1.0 1.6082e+00 1.1 0.00e+00 0.0 8.4e+06 2.5e+03 8.4e+01 3 0 3 4 3 3 0 3 4 3 0
MatPtAPNumeric 12 1.0 1.1666e+00 1.1 2.90e+07 1.2 5.1e+06 1.8e+03 9.6e+01 2 2 2 2 4 2 2 2 2 4 252212
MatTrnMatMult 12 1.0 5.1786e+00 1.0 1.32e+08 1.2 9.6e+06 2.7e+04 1.9e+02 11 11 4 45 7 11 11 4 45 7 255034
MatTrnMatMultSym 12 1.0 3.7081e+00 1.0 0.00e+00 0.0 7.6e+06 1.7e+04 1.4e+02 8 0 3 23 5 8 0 3 23 5 0
MatTrnMatMultNum 12 1.0 1.4973e+00 1.0 1.32e+08 1.2 2.0e+06 6.3e+04 4.8e+01 3 11 1 23 2 3 11 1 23 2 882036
MatGetLocalMat 54 1.0 7.5682e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatGetBrAoCol 36 1.0 7.0139e-02 4.1 0.00e+00 0.0 1.1e+07 2.2e+03 0.0e+00 0 0 4 4 0 0 0 4 4 0 0
KSPGMRESOrthog 120 1.0 2.7368e-01 1.3 1.85e+07 1.1 0.0e+00 0.0e+00 1.2e+02 0 2 0 0 5 0 2 0 0 5 686067
KSPSetUp 45 1.0 4.9518e-01 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+01 1 0 0 0 1 1 0 0 0 1 0
KSPSolve 3 1.0 4.3820e+00 1.0 9.63e+08 1.1 1.8e+08 9.4e+02 5.1e+02 9 80 66 30 20 9 80 66 30 20 2246294
PCGAMGGraph_AGG 12 1.0 2.1963e+00 1.0 4.56e+06 1.2 4.6e+06 6.8e+02 1.4e+02 4 0 2 1 6 4 0 2 1 6 21225
PCGAMGCoarse_AGG 12 1.0 6.2537e+00 1.0 1.32e+08 1.2 4.2e+07 7.1e+03 4.4e+02 13 11 15 53 17 13 11 15 53 17 211189
PCGAMGProl_AGG 12 1.0 2.3437e+00 1.0 0.00e+00 0.0 5.6e+06 1.5e+03 1.9e+02 5 0 2 1 7 5 0 2 1 8 0
PCGAMGPOpt_AGG 12 1.0 3.9482e+00 1.0 7.37e+07 1.1 2.4e+07 9.2e+02 5.0e+02 8 6 9 4 19 8 6 9 4 19 191367
GAMG: createProl 12 1.0 1.4742e+01 1.0 2.10e+08 1.2 7.7e+07 4.4e+03 1.3e+03 30 17 28 58 49 30 17 28 58 50 144003
Graph 24 1.0 2.1900e+00 1.0 4.56e+06 1.2 4.6e+06 6.8e+02 1.4e+02 4 0 2 1 6 4 0 2 1 6 21286
MIS/Agg 12 1.0 3.1835e-01 1.2 0.00e+00 0.0 2.4e+07 1.0e+03 2.0e+02 1 0 9 4 8 1 0 9 4 8 0
SA: col data 12 1.0 6.9797e-01 1.1 0.00e+00 0.0 3.4e+06 2.1e+03 4.8e+01 1 0 1 1 2 1 0 1 1 2 0
SA: frmProl0 12 1.0 1.1243e+00 1.0 0.00e+00 0.0 2.2e+06 5.5e+02 9.6e+01 2 0 1 0 4 2 0 1 0 4 0
SA: smooth 12 1.0 2.4933e+00 1.1 5.26e+06 1.2 8.7e+06 7.4e+02 1.7e+02 5 0 3 1 7 5 0 3 1 7 21589
GAMG: partLevel 12 1.0 7.1589e+00 1.0 2.90e+07 1.2 1.4e+07 2.1e+03 6.4e+02 15 2 5 5 25 15 2 5 5 25 41101
repartition 9 1.0 6.6796e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 5.4e+01 1 0 0 0 2 1 0 0 0 2 0
Invert-Sort 9 1.0 5.8634e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 3.6e+01 1 0 0 0 1 1 0 0 0 1 0
Move A 9 1.0 1.3901e+00 1.0 0.00e+00 0.0 3.3e+05 9.2e+02 1.5e+02 3 0 0 0 6 3 0 0 0 6 0
Move P 9 1.0 1.3170e+00 1.0 0.00e+00 0.0 5.6e+05 2.6e+01 1.5e+02 3 0 0 0 6 3 0 0 0 6 0
PCSetUp 6 1.0 2.2624e+01 1.0 2.39e+08 1.2 9.1e+07 4.0e+03 2.0e+03 46 20 33 64 77 46 20 33 64 77 106836
PCSetUpOnBlocks 42 1.0 2.9893e-0232.3 2.10e+01 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
PCApply 42 1.0 3.4927e+00 1.1 8.91e+08 1.1 1.7e+08 8.9e+02 3.8e+02 7 74 62 26 15 7 74 62 26 15 2600998
SFSetGraph 12 1.0 1.0152e-023939.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
SFSetUp 12 1.0 1.0030e-01 2.5 0.00e+00 0.0 2.5e+06 1.4e+03 0.0e+00 0 0 1 1 0 0 0 1 1 0 0
SFBcastBegin 225 1.0 2.3892e-0224.3 0.00e+00 0.0 2.2e+07 9.9e+02 0.0e+00 0 0 8 4 0 0 0 8 4 0 0
SFBcastEnd 225 1.0 4.7889e-0213.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
------------------------------------------------------------------------------------------------------------------------
Memory usage is given in bytes:
Object Type Creations Destructions Memory Descendants' Mem.
Reports information only for process 0.
--- Event Stage 0: Main Stage
Vector 675 675 28296816 0.
Matrix 456 456 597169344 0.
Matrix Coarsen 12 12 8208 0.
Index Set 294 294 7637280 0.
Vec Scatter 102 102 143424 0.
Krylov Solver 45 45 415944 0.
Preconditioner 33 33 35448 0.
Viewer 5 4 3584 0.
PetscRandom 24 24 16656 0.
Star Forest Graph 12 12 11328 0.
========================================================================================================================
Average time to get PetscTime(): 4.17233e-08
Average time for MPI_Barrier(): 0.00131778
Average time for zero size MPI_Send(): 2.1515e-06
#PETSc Option Table entries:
--prefix popcorn3d_full_l3_s6
-beta 7.0
-betaest .true.
-check .false.
-datadt data_distribution_fully_assembled
-dm 3
-in_space .true.
-ksp_converged_reason
-ksp_max_it 500
-ksp_monitor
-ksp_norm_type unpreconditioned
-ksp_rtol 1.0e-8
-ksp_type cg
-ksp_view
-l 1
-levelset popcorn
-levelsettol 0.0
-log_view
-lsdom -0.1
-mg_coarse_sub_pc_factor_mat_ordering_type nd
-mg_coarse_sub_pc_type cholesky
-mg_levels_esteig_ksp_type cg
-n 720
-no_signal_handler
-nruns 3
-pc_gamg_agg_nsmooths 1
-pc_gamg_est_ksp_type cg
-pc_gamg_process_eq_limit 50
-pc_gamg_square_graph 10
-pc_gamg_type agg
-pc_type gamg
-petscrc /gpfs/scratch/upc26/upc26229/NEW_STUFF/time_par_cell_agg_ompi.paper/petscrc-0
-tt 1
-uagg .false.
-wsolution .false.
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 8
Configure options: --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 -with-blaslapack-dir=/apps/INTEL/2017.4/mkl --with-debugging=0 --with-x=0 --with-shared-libraries=1 --with-mpi=1 --with-64-bit-indices --download-hypre=../v2.14.0.tar.gz
-----------------------------------------
Libraries compiled on 2018-11-07 17:23:07 on login1
Machine characteristics: Linux-4.4.120-92.70-default-x86_64-with-SuSE-12-x86_64
Using PETSc directory: /gpfs/scratch/upc26/upc26229/petsc_cell_agg_openmpi/release/petsc-3.9.0
Using PETSc arch: arch-linux2-c-opt
-----------------------------------------
Using C compiler: mpicc -fPIC -wd1572 -g -O3
Using Fortran compiler: mpif90 -fPIC -g -O3
-----------------------------------------
Using include paths: -I/gpfs/scratch/upc26/upc26229/petsc_cell_agg_openmpi/release/petsc-3.9.0/include -I/gpfs/scratch/upc26/upc26229/petsc_cell_agg_openmpi/release/petsc-3.9.0/arch-linux2-c-opt/include
-----------------------------------------
Using C linker: mpicc
Using Fortran linker: mpif90
Using libraries: -Wl,-rpath,/gpfs/scratch/upc26/upc26229/petsc_cell_agg_openmpi/release/petsc-3.9.0/arch-linux2-c-opt/lib -L/gpfs/scratch/upc26/upc26229/petsc_cell_agg_openmpi/release/petsc-3.9.0/arch-linux2-c-opt/lib -lpetsc -Wl,-rpath,/gpfs/scratch/upc26/upc26229/petsc_cell_agg_openmpi/release/petsc-3.9.0/arch-linux2-c-opt/lib -L/gpfs/scratch/upc26/upc26229/petsc_cell_agg_openmpi/release/petsc-3.9.0/arch-linux2-c-opt/lib -Wl,-rpath,/apps/INTEL/2017.4/mkl/lib/intel64 -L/apps/INTEL/2017.4/mkl/lib/intel64 -Wl,-rpath,/usr/mpi/intel/openmpi-1.10.4-hfi/lib64 -L/usr/mpi/intel/openmpi-1.10.4-hfi/lib64 -Wl,-rpath,/gpfs/apps/MN4/INTEL/2018.0.128/compilers_and_libraries_2018.0.128/linux/compiler/lib/intel64_lin -L/gpfs/apps/MN4/INTEL/2018.0.128/compilers_and_libraries_2018.0.128/linux/compiler/lib/intel64_lin -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/4.8 -L/usr/lib64/gcc/x86_64-suse-linux/4.8 -Wl,-rpath,/usr/x86_64-suse-linux/lib -L/usr/x86_64-suse-linux/lib -lHYPRE -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lstdc++ -ldl -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lifport -lifcoremt_pic -limf -lsvml -lm -lipgo -lirc -lpthread -lgcc_s -lirc_s -lstdc++ -ldl
-----------------------------------------
Ending run at vie nov 16 15:26:59 CET 2018
-------------- next part --------------
Linear solve converged due to CONVERGED_RTOL iterations 15
KSP Object: 16464 MPI processes
type: cg
maximum iterations=500, initial guess is zero
tolerances: relative=1e-08, absolute=1e-50, divergence=10000.
left preconditioning
using UNPRECONDITIONED norm type for convergence test
PC Object: 16464 MPI processes
type: gamg
type is MULTIPLICATIVE, levels=5 cycles=v
Cycles per PCApply=1
Using externally compute Galerkin coarse grid matrices
GAMG specific options
Threshold for dropping small values in graph on each level = 0. 0. 0.
Threshold scaling factor for each level not specified = 1.
AGG specific options
Symmetric graph false
Number of levels to square graph 10
Number smoothing steps 1
Coarse grid solver -- level -------------------------------
KSP Object: (mg_coarse_) 16464 MPI processes
type: preonly
maximum iterations=10000, initial guess is zero
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
left preconditioning
using NONE norm type for convergence test
PC Object: (mg_coarse_) 16464 MPI processes
type: bjacobi
number of blocks = 16464
Local solve is same for all blocks, in the following KSP and PC objects:
KSP Object: (mg_coarse_sub_) 1 MPI processes
type: preonly
maximum iterations=1, initial guess is zero
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
left preconditioning
using NONE norm type for convergence test
PC Object: (mg_coarse_sub_) 1 MPI processes
type: cholesky
out-of-place factorization
tolerance for zero pivot 2.22045e-14
matrix ordering: nd
factor fill ratio given 5., needed 1.
Factored matrix follows:
Mat Object: 1 MPI processes
type: seqsbaij
rows=11, cols=11
package used to perform factorization: petsc
total: nonzeros=66, allocated nonzeros=66
total number of mallocs used during MatSetValues calls =0
block size is 1
linear system matrix = precond matrix:
Mat Object: 1 MPI processes
type: seqaij
rows=11, cols=11
total: nonzeros=121, allocated nonzeros=121
total number of mallocs used during MatSetValues calls =0
using I-node routines: found 3 nodes, limit used is 5
linear system matrix = precond matrix:
Mat Object: 16464 MPI processes
type: mpiaij
rows=11, cols=11
total: nonzeros=121, allocated nonzeros=121
total number of mallocs used during MatSetValues calls =0
using I-node (on process 0) routines: found 3 nodes, limit used is 5
Down solver (pre-smoother) on level 1 -------------------------------
KSP Object: (mg_levels_1_) 16464 MPI processes
type: chebyshev
eigenvalue estimates used: min = 0.135989, max = 1.49588
eigenvalues estimate via cg min 0.195543, max 1.35989
eigenvalues estimated using cg with translations [0. 0.1; 0. 1.1]
KSP Object: (mg_levels_1_esteig_) 16464 MPI processes
type: cg
maximum iterations=10, initial guess is zero
tolerances: relative=1e-12, absolute=1e-50, divergence=10000.
left preconditioning
using PRECONDITIONED norm type for convergence test
estimating eigenvalues using noisy right hand side
maximum iterations=2, nonzero initial guess
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
left preconditioning
using NONE norm type for convergence test
PC Object: (mg_levels_1_) 16464 MPI processes
type: sor
type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
linear system matrix = precond matrix:
Mat Object: 16464 MPI processes
type: mpiaij
rows=893, cols=893
total: nonzeros=54833, allocated nonzeros=54833
total number of mallocs used during MatSetValues calls =0
not using I-node (on process 0) routines
Up solver (post-smoother) same as down solver (pre-smoother)
Down solver (pre-smoother) on level 2 -------------------------------
KSP Object: (mg_levels_2_) 16464 MPI processes
type: chebyshev
eigenvalue estimates used: min = 0.146943, max = 1.61637
eigenvalues estimate via cg min 0.037861, max 1.46943
eigenvalues estimated using cg with translations [0. 0.1; 0. 1.1]
KSP Object: (mg_levels_2_esteig_) 16464 MPI processes
type: cg
maximum iterations=10, initial guess is zero
tolerances: relative=1e-12, absolute=1e-50, divergence=10000.
left preconditioning
using PRECONDITIONED norm type for convergence test
estimating eigenvalues using noisy right hand side
maximum iterations=2, nonzero initial guess
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
left preconditioning
using NONE norm type for convergence test
PC Object: (mg_levels_2_) 16464 MPI processes
type: sor
type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
linear system matrix = precond matrix:
Mat Object: 16464 MPI processes
type: mpiaij
rows=114979, cols=114979
total: nonzeros=9348035, allocated nonzeros=9348035
total number of mallocs used during MatSetValues calls =0
not using I-node (on process 0) routines
Up solver (post-smoother) same as down solver (pre-smoother)
Down solver (pre-smoother) on level 3 -------------------------------
KSP Object: (mg_levels_3_) 16464 MPI processes
type: chebyshev
eigenvalue estimates used: min = 0.138364, max = 1.52201
eigenvalues estimate via cg min 0.0350501, max 1.38364
eigenvalues estimated using cg with translations [0. 0.1; 0. 1.1]
KSP Object: (mg_levels_3_esteig_) 16464 MPI processes
type: cg
maximum iterations=10, initial guess is zero
tolerances: relative=1e-12, absolute=1e-50, divergence=10000.
left preconditioning
using PRECONDITIONED norm type for convergence test
estimating eigenvalues using noisy right hand side
maximum iterations=2, nonzero initial guess
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
left preconditioning
using NONE norm type for convergence test
PC Object: (mg_levels_3_) 16464 MPI processes
type: sor
type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
linear system matrix = precond matrix:
Mat Object: 16464 MPI processes
type: mpiaij
rows=9167199, cols=9167199
total: nonzeros=419078313, allocated nonzeros=419078313
total number of mallocs used during MatSetValues calls =0
using nonscalable MatPtAP() implementation
not using I-node (on process 0) routines
Up solver (post-smoother) same as down solver (pre-smoother)
Down solver (pre-smoother) on level 4 -------------------------------
KSP Object: (mg_levels_4_) 16464 MPI processes
type: chebyshev
eigenvalue estimates used: min = 0.136589, max = 1.50248
eigenvalues estimate via cg min 0.0332852, max 1.36589
eigenvalues estimated using cg with translations [0. 0.1; 0. 1.1]
KSP Object: (mg_levels_4_esteig_) 16464 MPI processes
type: cg
maximum iterations=10, initial guess is zero
tolerances: relative=1e-12, absolute=1e-50, divergence=10000.
left preconditioning
using PRECONDITIONED norm type for convergence test
estimating eigenvalues using noisy right hand side
maximum iterations=2, nonzero initial guess
tolerances: relative=1e-05, absolute=1e-50, divergence=10000.
left preconditioning
using NONE norm type for convergence test
PC Object: (mg_levels_4_) 16464 MPI processes
type: sor
type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
linear system matrix = precond matrix:
Mat Object: 16464 MPI processes
type: mpiaij
rows=442766309, cols=442766309
total: nonzeros=11923049125, allocated nonzeros=88553261800
total number of mallocs used during MatSetValues calls =0
not using I-node (on process 0) routines
Up solver (post-smoother) same as down solver (pre-smoother)
linear system matrix = precond matrix:
Mat Object: 16464 MPI processes
type: mpiaij
rows=442766309, cols=442766309
total: nonzeros=11923049125, allocated nonzeros=88553261800
total number of mallocs used during MatSetValues calls =0
not using I-node (on process 0) routines
************************************************************************************************************************
*** WIDEN YOUR WINDOW TO 120 CHARACTERS. Use 'enscript -r -fCourier9' to print this document ***
************************************************************************************************************************
---------------------------------------------- PETSc Performance Summary: ----------------------------------------------
/gpfs/scratch/upc26/upc26229/build_rel_fempar_cell_agg_ompi/FEMPAR/bin/par_test_poisson_unfitted on a arch-linux2-c-opt named s01r2b26 with 16464 processors, by upc26229 Fri Nov 16 18:41:14 2018
Using Petsc Release Version 3.9.0, Apr, 07, 2018
Max Max/Min Avg Total
Time (sec): 6.644e+01 1.00006 6.644e+01
Objects: 1.652e+03 1.00182 1.649e+03
Flop: 1.273e+09 1.15475 1.244e+09 2.049e+13
Flop/sec: 1.916e+07 1.15475 1.873e+07 3.084e+11
MPI Messages: 2.616e+05 35.23821 2.868e+04 4.721e+08
MPI Message Lengths: 6.746e+07 2.62396 1.960e+03 9.251e+11
MPI Reductions: 2.616e+03 1.00000
Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
e.g., VecAXPY() for real vectors of length N --> 2N flop
and VecAXPY() for complex vectors of length N --> 8N flop
Summary of Stages: ----- Time ------ ----- Flop ----- --- Messages --- -- Message Lengths -- -- Reductions --
Avg %Total Avg %Total counts %Total Avg %Total counts %Total
0: Main Stage: 6.6434e+01 100.0% 2.0489e+13 100.0% 4.721e+08 100.0% 1.960e+03 100.0% 2.602e+03 99.5%
------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
Count: number of times phase was executed
Time and Flop: Max - maximum over all processors
Ratio - ratio of maximum to minimum over all processors
Mess: number of messages sent
Avg. len: average message length (bytes)
Reduct: number of global reductions
Global: entire computation
Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
%T - percent time in this phase %F - percent flop in this phase
%M - percent messages in this phase %L - percent message lengths in this phase
%R - percent reductions in this phase
Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over all processors)
------------------------------------------------------------------------------------------------------------------------
Event Count Time (sec) Flop --- Global --- --- Stage --- Total
Max Ratio Max Ratio Max Ratio Mess Avg len Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------
--- Event Stage 0: Main Stage
BuildTwoSided 12 1.0 1.4701e-01 2.7 0.00e+00 0.0 1.4e+06 8.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
BuildTwoSidedF 177 1.0 3.0236e+00 1.6 0.00e+00 0.0 5.8e+06 1.3e+04 0.0e+00 4 0 1 8 0 4 0 1 8 0 0
VecMDot 120 1.0 3.9575e-01 1.6 9.23e+06 1.1 0.0e+00 0.0e+00 1.2e+02 0 1 0 0 5 0 1 0 0 5 376930
VecTDot 342 1.0 1.5654e+00 1.4 8.39e+06 1.1 0.0e+00 0.0e+00 3.4e+02 2 1 0 0 13 2 1 0 0 13 87294
VecNorm 312 1.0 9.9940e-01 1.3 6.29e+06 1.1 0.0e+00 0.0e+00 3.1e+02 1 0 0 0 12 1 0 0 0 12 102238
VecScale 132 1.0 1.4698e-0249.3 9.24e+05 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1014961
VecCopy 222 1.0 5.5664e-03 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecSet 888 1.0 3.7285e-03 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecAXPY 342 1.0 4.5476e-02 6.2 8.39e+06 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 3005001
VecAYPX 1590 1.0 4.0597e-02 2.2 1.64e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 6528161
VecAXPBYCZ 720 1.0 2.9899e-02 1.8 2.52e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 2 0 0 0 0 2 0 0 0 13607494
VecMAXPY 132 1.0 1.0936e-02 2.4 1.09e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 16120658
VecAssemblyBegin 84 1.0 5.9557e-01 1.4 0.00e+00 0.0 8.1e+05 2.6e+03 0.0e+00 1 0 0 0 0 1 0 0 0 0 0
VecAssemblyEnd 84 1.0 4.3126e-04 4.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecPointwiseMult 132 1.0 8.7574e-03 5.9 9.24e+05 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 1703431
VecScatterBegin 1830 1.0 8.9270e-02 4.4 0.00e+00 0.0 3.6e+08 9.9e+02 0.0e+00 0 0 76 38 0 0 0 76 38 0 0
VecScatterEnd 1830 1.0 1.3435e+00 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 2 0 0 0 0 2 0 0 0 0 0
VecSetRandom 12 1.0 4.2692e-03 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecNormalize 132 1.0 3.9408e-01 1.4 2.77e+06 1.1 0.0e+00 0.0e+00 1.3e+02 1 0 0 0 5 1 0 0 0 5 113564
MatMult 1365 1.0 1.8317e+00 1.5 5.58e+08 1.2 2.9e+08 1.1e+03 0.0e+00 2 44 62 33 0 2 44 62 33 0 4944033
MatMultAdd 180 1.0 7.1299e-0111.3 1.06e+07 1.2 2.1e+07 1.2e+02 0.0e+00 1 1 5 0 0 1 1 5 0 0 241236
MatMultTranspose 180 1.0 4.6874e-0112.9 1.06e+07 1.2 2.1e+07 1.2e+02 0.0e+00 0 1 5 0 0 0 1 5 0 0 366940
MatSolve 45 0.0 1.2516e-04 0.0 1.04e+04 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 83
MatSOR 1212 1.0 1.5348e+00 1.3 4.36e+08 1.1 0.0e+00 0.0e+00 0.0e+00 2 34 0 0 0 2 34 0 0 0 4532804
MatCholFctrSym 3 1.0 1.2814e-02711.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatCholFctrNum 3 1.0 9.1130e-032313.2 3.30e+01 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatConvert 12 1.0 4.6102e-02 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatScale 36 1.0 6.0832e-02 3.4 5.28e+06 1.2 2.5e+06 1.0e+03 0.0e+00 0 0 1 0 0 0 0 1 0 0 1406766
MatResidual 180 1.0 2.6789e-01 1.9 6.87e+07 1.2 3.8e+07 1.0e+03 0.0e+00 0 5 8 4 0 0 5 8 4 0 4149588
MatAssemblyBegin 312 1.0 2.5479e+00 1.8 0.00e+00 0.0 5.0e+06 1.4e+04 0.0e+00 3 0 1 8 0 3 0 1 8 0 0
MatAssemblyEnd 312 1.0 9.6727e+00 1.0 0.00e+00 0.0 2.5e+07 4.4e+02 6.5e+02 14 0 5 1 25 14 0 5 1 25 0
MatGetRow 251874 1.1 3.8604e-02 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatGetRowIJ 3 0.0 2.7359e-03 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatCreateSubMat 18 1.0 5.7689e+00 1.0 0.00e+00 0.0 1.5e+06 3.4e+02 2.9e+02 9 0 0 0 11 9 0 0 0 11 0
MatGetOrdering 3 0.0 7.3662e-03 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatCoarsen 12 1.0 5.6386e-01 1.2 0.00e+00 0.0 4.7e+07 8.7e+02 2.3e+02 1 0 10 4 9 1 0 10 4 9 0
MatZeroEntries 15 1.0 1.4088e-03 6.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatView 24 1.3 4.0379e-0110.0 0.00e+00 0.0 0.0e+00 0.0e+00 1.8e+01 0 0 0 0 1 0 0 0 0 1 0
MatAXPY 12 1.0 1.9423e-01 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatMatMult 12 1.0 3.5538e+00 1.0 4.58e+06 1.2 1.4e+07 7.3e+02 1.5e+02 5 0 3 1 6 5 0 3 1 6 20853
MatMatMultSym 12 1.0 3.3264e+00 1.0 0.00e+00 0.0 1.2e+07 6.7e+02 1.4e+02 5 0 2 1 6 5 0 2 1 6 0
MatMatMultNum 12 1.0 9.7088e-02 1.1 4.58e+06 1.2 2.5e+06 1.0e+03 0.0e+00 0 0 1 0 0 0 0 1 0 0 763319
MatPtAP 12 1.0 4.0859e+00 1.0 2.90e+07 1.2 2.2e+07 2.2e+03 1.9e+02 6 2 5 5 7 6 2 5 5 7 114537
MatPtAPSymbolic 12 1.0 2.4298e+00 1.1 0.00e+00 0.0 1.4e+07 2.4e+03 8.4e+01 4 0 3 4 3 4 0 3 4 3 0
MatPtAPNumeric 12 1.0 1.7467e+00 1.1 2.90e+07 1.2 8.2e+06 1.8e+03 9.6e+01 3 2 2 2 4 3 2 2 2 4 267927
MatTrnMatMult 12 1.0 7.1406e+00 1.0 1.35e+08 1.2 1.6e+07 2.6e+04 1.9e+02 11 10 3 44 7 11 10 3 44 7 294270
MatTrnMatMultSym 12 1.0 5.6756e+00 1.0 0.00e+00 0.0 1.3e+07 1.6e+04 1.6e+02 9 0 3 23 6 9 0 3 23 6 0
MatTrnMatMultNum 12 1.0 1.5121e+00 1.0 1.35e+08 1.2 2.5e+06 7.9e+04 2.4e+01 2 10 1 21 1 2 10 1 21 1 1389611
MatGetLocalMat 57 1.0 7.5516e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatGetBrAoCol 36 1.0 6.7780e-02 4.0 0.00e+00 0.0 1.8e+07 2.1e+03 0.0e+00 0 0 4 4 0 0 0 4 4 0 0
KSPGMRESOrthog 120 1.0 4.0225e-01 1.6 1.85e+07 1.1 0.0e+00 0.0e+00 1.2e+02 0 1 0 0 5 0 1 0 0 5 741700
KSPSetUp 45 1.0 8.8595e-01 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+01 1 0 0 0 1 1 0 0 0 1 0
KSPSolve 3 1.0 5.6979e+00 1.1 1.03e+09 1.1 3.1e+08 9.3e+02 5.2e+02 8 81 66 31 20 8 81 66 31 20 2921121
PCGAMGGraph_AGG 12 1.0 3.3467e+00 1.0 4.58e+06 1.2 7.6e+06 6.7e+02 1.4e+02 5 0 2 1 6 5 0 2 1 6 22144
PCGAMGCoarse_AGG 12 1.0 8.9895e+00 1.0 1.35e+08 1.2 7.7e+07 6.2e+03 4.7e+02 14 10 16 51 18 14 10 16 51 18 233748
PCGAMGProl_AGG 12 1.0 3.9328e+00 1.0 0.00e+00 0.0 9.1e+06 1.5e+03 1.9e+02 6 0 2 1 7 6 0 2 1 7 0
PCGAMGPOpt_AGG 12 1.0 6.3192e+00 1.0 7.43e+07 1.1 3.9e+07 9.0e+02 5.0e+02 9 6 8 4 19 9 6 8 4 19 190048
GAMG: createProl 12 1.0 2.2585e+01 1.0 2.13e+08 1.2 1.3e+08 4.0e+03 1.3e+03 34 16 28 57 50 34 16 28 57 50 149493
Graph 24 1.0 3.3388e+00 1.0 4.58e+06 1.2 7.6e+06 6.7e+02 1.4e+02 5 0 2 1 6 5 0 2 1 6 22196
MIS/Agg 12 1.0 5.6411e-01 1.2 0.00e+00 0.0 4.7e+07 8.7e+02 2.3e+02 1 0 10 4 9 1 0 10 4 9 0
SA: col data 12 1.0 1.2982e+00 1.1 0.00e+00 0.0 5.6e+06 2.0e+03 4.8e+01 2 0 1 1 2 2 0 1 1 2 0
SA: frmProl0 12 1.0 1.6284e+00 1.0 0.00e+00 0.0 3.6e+06 5.5e+02 9.6e+01 2 0 1 0 4 2 0 1 0 4 0
SA: smooth 12 1.0 4.1778e+00 1.0 5.28e+06 1.2 1.4e+07 7.3e+02 1.7e+02 6 0 3 1 7 6 0 3 1 7 20483
GAMG: partLevel 12 1.0 1.3577e+01 1.0 2.90e+07 1.2 2.3e+07 2.1e+03 6.4e+02 20 2 5 5 25 20 2 5 5 25 34470
repartition 9 1.0 1.5048e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 5.4e+01 2 0 0 0 2 2 0 0 0 2 0
Invert-Sort 9 1.0 1.2282e+00 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 3.6e+01 2 0 0 0 1 2 0 0 0 1 0
Move A 9 1.0 2.8930e+00 1.0 0.00e+00 0.0 5.7e+05 8.4e+02 1.5e+02 4 0 0 0 6 4 0 0 0 6 0
Move P 9 1.0 3.0317e+00 1.0 0.00e+00 0.0 9.4e+05 2.5e+01 1.5e+02 5 0 0 0 6 5 0 0 0 6 0
PCSetUp 6 1.0 3.7378e+01 1.0 2.41e+08 1.2 1.6e+08 3.7e+03 2.0e+03 56 19 33 62 77 56 19 33 62 77 102849
PCSetUpOnBlocks 45 1.0 3.1359e-02137.6 3.30e+01 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
PCApply 45 1.0 4.5105e+00 1.1 9.54e+08 1.1 2.9e+08 8.7e+02 3.8e+02 7 75 62 28 15 7 75 62 28 15 3403589
SFSetGraph 12 1.0 8.7668e-033502.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
SFSetUp 12 1.0 1.6941e-01 2.3 0.00e+00 0.0 4.2e+06 1.4e+03 0.0e+00 0 0 1 1 0 0 0 1 1 0 0
SFBcastBegin 255 1.0 2.5270e-0229.0 0.00e+00 0.0 4.3e+07 8.3e+02 0.0e+00 0 0 9 4 0 0 0 9 4 0 0
SFBcastEnd 255 1.0 1.0079e-0128.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
------------------------------------------------------------------------------------------------------------------------
Memory usage is given in bytes:
Object Type Creations Destructions Memory Descendants' Mem.
Reports information only for process 0.
--- Event Stage 0: Main Stage
Vector 675 675 28316856 0.
Matrix 450 450 596794296 0.
Matrix Coarsen 12 12 8208 0.
Index Set 294 294 11758608 0.
Vec Scatter 102 102 143424 0.
Krylov Solver 45 45 415944 0.
Preconditioner 33 33 35448 0.
Viewer 5 4 3584 0.
PetscRandom 24 24 16656 0.
Star Forest Graph 12 12 11328 0.
========================================================================================================================
Average time to get PetscTime(): 4.15836e-08
Average time for MPI_Barrier(): 3.40383e-05
Average time for zero size MPI_Send(): 1.88055e-06
#PETSc Option Table entries:
--prefix popcorn3d_full_l3_s7
-beta 7.0
-betaest .true.
-check .false.
-datadt data_distribution_fully_assembled
-dm 3
-in_space .true.
-ksp_converged_reason
-ksp_max_it 500
-ksp_monitor
-ksp_norm_type unpreconditioned
-ksp_rtol 1.0e-8
-ksp_type cg
-ksp_view
-l 1
-levelset popcorn
-levelsettol 0.0
-log_view
-lsdom -0.1
-mg_coarse_sub_pc_factor_mat_ordering_type nd
-mg_coarse_sub_pc_type cholesky
-mg_levels_esteig_ksp_type cg
-n 840
-no_signal_handler
-nruns 3
-pc_gamg_agg_nsmooths 1
-pc_gamg_est_ksp_type cg
-pc_gamg_process_eq_limit 50
-pc_gamg_square_graph 10
-pc_gamg_type agg
-pc_type gamg
-petscrc /gpfs/scratch/upc26/upc26229/NEW_STUFF/time_par_cell_agg_ompi.paper/petscrc-0
-tt 1
-uagg .false.
-wsolution .false.
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 8
Configure options: --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 -with-blaslapack-dir=/apps/INTEL/2017.4/mkl --with-debugging=0 --with-x=0 --with-shared-libraries=1 --with-mpi=1 --with-64-bit-indices --download-hypre=../v2.14.0.tar.gz
-----------------------------------------
Libraries compiled on 2018-11-07 17:23:07 on login1
Machine characteristics: Linux-4.4.120-92.70-default-x86_64-with-SuSE-12-x86_64
Using PETSc directory: /gpfs/scratch/upc26/upc26229/petsc_cell_agg_openmpi/release/petsc-3.9.0
Using PETSc arch: arch-linux2-c-opt
-----------------------------------------
Using C compiler: mpicc -fPIC -wd1572 -g -O3
Using Fortran compiler: mpif90 -fPIC -g -O3
-----------------------------------------
Using include paths: -I/gpfs/scratch/upc26/upc26229/petsc_cell_agg_openmpi/release/petsc-3.9.0/include -I/gpfs/scratch/upc26/upc26229/petsc_cell_agg_openmpi/release/petsc-3.9.0/arch-linux2-c-opt/include
-----------------------------------------
Using C linker: mpicc
Using Fortran linker: mpif90
Using libraries: -Wl,-rpath,/gpfs/scratch/upc26/upc26229/petsc_cell_agg_openmpi/release/petsc-3.9.0/arch-linux2-c-opt/lib -L/gpfs/scratch/upc26/upc26229/petsc_cell_agg_openmpi/release/petsc-3.9.0/arch-linux2-c-opt/lib -lpetsc -Wl,-rpath,/gpfs/scratch/upc26/upc26229/petsc_cell_agg_openmpi/release/petsc-3.9.0/arch-linux2-c-opt/lib -L/gpfs/scratch/upc26/upc26229/petsc_cell_agg_openmpi/release/petsc-3.9.0/arch-linux2-c-opt/lib -Wl,-rpath,/apps/INTEL/2017.4/mkl/lib/intel64 -L/apps/INTEL/2017.4/mkl/lib/intel64 -Wl,-rpath,/usr/mpi/intel/openmpi-1.10.4-hfi/lib64 -L/usr/mpi/intel/openmpi-1.10.4-hfi/lib64 -Wl,-rpath,/gpfs/apps/MN4/INTEL/2018.0.128/compilers_and_libraries_2018.0.128/linux/compiler/lib/intel64_lin -L/gpfs/apps/MN4/INTEL/2018.0.128/compilers_and_libraries_2018.0.128/linux/compiler/lib/intel64_lin -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/4.8 -L/usr/lib64/gcc/x86_64-suse-linux/4.8 -Wl,-rpath,/usr/x86_64-suse-linux/lib -L/usr/x86_64-suse-linux/lib -lHYPRE -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lpthread -lstdc++ -ldl -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lifport -lifcoremt_pic -limf -lsvml -lm -lipgo -lirc -lpthread -lgcc_s -lirc_s -lstdc++ -ldl
-----------------------------------------
Ending run at vie nov 16 18:41:15 CET 2018
-------------- next part --------------
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=127, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=580, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=211, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=97, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=508, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list:[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list:[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=220, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list:[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list:[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=211, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=1012, tag=0
(context=e8, rank=127, tag=7ffffd8d, sreq=0x2d580b0)[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=211, tag=7ffffd8d, sreq=0x2fda8e0) (context=e8, rank=97, tag=7ffffd8d, sreq=0x44b02f0) (context=e8, rank=508, tag=7ffffd8d, sreq=0x3f55830)[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list:[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=133, tag=7ffffd8d, sreq=0x2d40fb0) (context=e8, rank=580, tag=7ffffd8d, sreq=0x5b1d140) (context=e8, rank=355, tag=7ffffd8d, sreq=0x2fc7b60)[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=100, tag=7ffffd8d, sreq=0x44b1df0) (context=e8, rank=652, tag=7ffffd8d, sreq=0x3f47bb0) (context=e8, rank=211, tag=7ffffd8d, sreq=0x4689c60) (context=e8, rank=1012, tag=7ffffd8d, sreq=0x5adf1f0)
(context=e8, rank=724, tag=7ffffd8d, sreq=0x5b1ccc0)[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=211, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=211, tag=7ffffd8d, sreq=0x321db60) (context=e8, rank=355, tag=7ffffd8d, sreq=0x32122e0)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f9560ff2640, &ssend_ack_recv_buffer=0x7f9560ff2648, &pad2=0x7f9560ff2658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
(context=e8, rank=220, tag=7ffffd8d, sreq=0x3c6b1b0)
(context=e8, rank=1156, tag=7ffffd8d, sreq=0x5ae42f0)MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f62a6443640, &ssend_ack_recv_buffer=0x7f62a6443648, &pad2=0x7f62a6443658
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=796, tag=0
(context=e8, rank=364, tag=7ffffd8d, sreq=0x3c76eb0)MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f460eef5640, &ssend_ack_recv_buffer=0x7f460eef5648, &pad2=0x7f460eef5658
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f089b5ae640, &ssend_ack_recv_buffer=0x7f089b5ae648, &pad2=0x7f089b5ae658
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f7017aab640, &ssend_ack_recv_buffer=0x7f7017aab648, &pad2=0x7f7017aab658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=283, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=958, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=817, tag=7ffffd8d, sreq=0x5b7d7b0) (context=e8, rank=820, tag=7ffffd8d, sreq=0x5b7c130) (context=e8, rank=958, tag=7ffffd8d, sreq=0x5b8ab30) (context=e8, rank=961, tag=7ffffd8d, sreq=0x5b75e30) (context=e8, rank=964, tag=7ffffd8d, sreq=0x5b8afb0)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f3e00a0d640, &ssend_ack_recv_buffer=0x7f3e00a0d648, &pad2=0x7f3e00a0d658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f2f61f9e640, &ssend_ack_recv_buffer=0x7f2f61f9e648, &pad2=0x7f2f61f9e658
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=868, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=955, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list:MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f9c131ed640, &ssend_ack_recv_buffer=0x7f9c131ed648, &pad2=0x7f9c131ed658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=1051, tag=0
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=715, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=709, tag=7ffffd8d, sreq=0x539bf30) (context=e8, rank=715, tag=7ffffd8d, sreq=0x539bab0) (context=e8, rank=853, tag=7ffffd8d, sreq=0x53acd30) (context=e8, rank=931, tag=7ffffd8d, sreq=0x5386930)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f37276cf640, &ssend_ack_recv_buffer=0x7f37276cf648, &pad2=0x7f37276cf658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=283, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=364, tag=0
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7fa5e6fa0640, &ssend_ack_recv_buffer=0x7fa5e6fa0648, &pad2=0x7fa5e6fa0658
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=787, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=643, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=499, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=355, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=349, tag=7ffffd8d, sreq=0x41fa2e0) (context=e8, rank=355, tag=7ffffd8d, sreq=0x41f99e0) (context=e8, rank=493, tag=7ffffd8d, sreq=0x4212ee0) (context=e8, rank=571, tag=7ffffd8d, sreq=0x42113e0)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7fbb1cbb1640, &ssend_ack_recv_buffer=0x7fbb1cbb1648, &pad2=0x7fbb1cbb1658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=571, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=148, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=1066, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=283, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=277, tag=7ffffd8d, sreq=0x3d48d30) (context=e8, rank=283, tag=7ffffd8d, sreq=0x3d491b0) (context=e8, rank=421, tag=7ffffd8d, sreq=0x3d557b0) (context=e8, rank=427, tag=7ffffd8d, sreq=0x3d52630) (context=e8, rank=499, tag=7ffffd8d, sreq=0x3d46030)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f9dd7f8a640, &ssend_ack_recv_buffer=0x7f9dd7f8a648, &pad2=0x7f9dd7f8a658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=859, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=859, tag=7ffffd8d, sreq=0x6c1f9f0) (context=e8, rank=1003, tag=7ffffd8d, sreq=0x6c0de70)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7fb5fd466640, &ssend_ack_recv_buffer=0x7fb5fd466648, &pad2=0x7fb5fd466658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=958, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=958, tag=7ffffd8d, sreq=0x51f0f80) (context=e8, rank=961, tag=7ffffd8d, sreq=0x51d9100) (context=e8, rank=964, tag=7ffffd8d, sreq=0x51ef480) (context=e8, rank=1102, tag=7ffffd8d, sreq=0x51d6d00) (context=e8, rank=1105, tag=7ffffd8d, sreq=0x51db980) (context=e8, rank=1108, tag=7ffffd8d, sreq=0x51e6480)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f78d4ec9640, &ssend_ack_recv_buffer=0x7f78d4ec9648, &pad2=0x7f78d4ec9658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
(context=e8, rank=796, tag=7ffffd8d, sreq=0x5ee11b0)MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7fa8cb4c5640, &ssend_ack_recv_buffer=0x7fa8cb4c5648, &pad2=0x7fa8cb4c5658
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list:[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=517, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=511, tag=7ffffd8d, sreq=0x44405b0) (context=e8, rank=514, tag=7ffffd8d, sreq=0x442fc30) (context=e8, rank=517, tag=7ffffd8d, sreq=0x44324b0) (context=e8, rank=655, tag=7ffffd8d, sreq=0x443b030) (context=e8, rank=658, tag=7ffffd8d, sreq=0x44495b0) (context=e8, rank=661, tag=7ffffd8d, sreq=0x4432930)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7feceff86640, &ssend_ack_recv_buffer=0x7feceff86648, &pad2=0x7feceff86658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=715, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=205, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=205, tag=7ffffd8d, sreq=0x40290e0)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7ff6c54a7640, &ssend_ack_recv_buffer=0x7ff6c54a7648, &pad2=0x7ff6c54a7658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list:[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list:MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=931, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=787, tag=7ffffd8d, sreq=0x5ebb0e0) (context=e8, rank=931, tag=7ffffd8d, sreq=0x5ea12e0)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f0e5d848640, &ssend_ack_recv_buffer=0x7f0e5d848648, &pad2=0x7f0e5d848658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list:[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=499, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=499, tag=7ffffd8d, sreq=0x4de0470) (context=e8, rank=643, tag=7ffffd8d, sreq=0x4dd03f0)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7fa65c639640, &ssend_ack_recv_buffer=0x7fa65c639648, &pad2=0x7fa65c639658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=355, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list:[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list:[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=1039, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=1039, tag=7ffffd8d, sreq=0x5228150) (context=e8, rank=1042, tag=7ffffd8d, sreq=0x5228a50) (context=e8, rank=1045, tag=7ffffd8d, sreq=0x52165d0) (context=e8, rank=1183, tag=7ffffd8d, sreq=0x521f5d0) (context=e8, rank=1186, tag=7ffffd8d, sreq=0x5231150) (context=e8, rank=1189, tag=7ffffd8d, sreq=0x521fed0)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f48b9ca1640, &ssend_ack_recv_buffer=0x7f48b9ca1648, &pad2=0x7f48b9ca1658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list:[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=571, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list:[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list:
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=355, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=205, tag=7ffffd8d, sreq=0x42512e0) (context=e8, rank=349, tag=7ffffd8d, sreq=0x4273360) (context=e8, rank=355, tag=7ffffd8d, sreq=0x4270ae0)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7fdb1fecb640, &ssend_ack_recv_buffer=0x7fdb1fecb648, &pad2=0x7fdb1fecb658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
(context=e8, rank=907, tag=7ffffd8d, sreq=0x3bdf250)[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=526, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=523, tag=7ffffd8d, sreq=0x3746ab0) (context=e8, rank=526, tag=7ffffd8d, sreq=0x3748130)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7ff6486d1640, &ssend_ack_recv_buffer=0x7ff6486d1648, &pad2=0x7ff6486d1658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=715, tag=7ffffd8d, sreq=0x6db92f0)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7fd3a4a9e640, &ssend_ack_recv_buffer=0x7fd3a4a9e648, &pad2=0x7fd3a4a9e658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
(context=e8, rank=277, tag=7ffffd8d, sreq=0x2b56db0) (context=e8, rank=364, tag=7ffffd8d, sreq=0x38f54b0)[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=1054, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=787, tag=7ffffd8d, sreq=0x5a93f20)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f47122dc640, &ssend_ack_recv_buffer=0x7f47122dc648, &pad2=0x7f47122dc658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
(context=e8, rank=643, tag=7ffffd8d, sreq=0x5314c60)[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list:[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=355, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=355, tag=7ffffd8d, sreq=0x44e1560)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7ff774d71640, &ssend_ack_recv_buffer=0x7ff774d71648, &pad2=0x7ff774d71658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
(context=e8, rank=421, tag=7ffffd8d, sreq=0x3409030) (context=e8, rank=148, tag=7ffffd8d, sreq=0x38553e0)[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=283, tag=7ffffd8d, sreq=0x4513d30)[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=571, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=571, tag=7ffffd8d, sreq=0x62bee40)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f635ece7640, &ssend_ack_recv_buffer=0x7f635ece7648, &pad2=0x7f635ece7658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
(context=e8, rank=868, tag=7ffffd8d, sreq=0x584cf70)[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=985, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=979, tag=7ffffd8d, sreq=0x38b7e00) (context=e8, rank=982, tag=7ffffd8d, sreq=0x38c5f00) (context=e8, rank=985, tag=7ffffd8d, sreq=0x38d2080) (context=e8, rank=1123, tag=7ffffd8d, sreq=0x38d1c00) (context=e8, rank=1126, tag=7ffffd8d, sreq=0x38d3700) (context=e8, rank=1129, tag=7ffffd8d, sreq=0x38c7580)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f83e9947640, &ssend_ack_recv_buffer=0x7f83e9947648, &pad2=0x7f83e9947658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7fd7d2b2b640, &ssend_ack_recv_buffer=0x7fd7d2b2b648, &pad2=0x7fd7d2b2b658
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=343, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=196, tag=7ffffd8d, sreq=0x318d460) (context=e8, rank=199, tag=7ffffd8d, sreq=0x3177560) (context=e8, rank=202, tag=7ffffd8d, sreq=0x317afe0) (context=e8, rank=340, tag=7ffffd8d, sreq=0x318c6e0) (context=e8, rank=343, tag=7ffffd8d, sreq=0x318f860) (context=e8, rank=346, tag=7ffffd8d, sreq=0x3197660)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f48361f8640, &ssend_ack_recv_buffer=0x7f48361f8648, &pad2=0x7f48361f8658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=223, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=223, tag=7ffffd8d, sreq=0x3aaf730) (context=e8, rank=367, tag=7ffffd8d, sreq=0x3ac3b30)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7ff7ff3e6640, &ssend_ack_recv_buffer=0x7ff7ff3e6648, &pad2=0x7ff7ff3e6658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
(context=e8, rank=910, tag=7ffffd8d, sreq=0x3bdedd0)[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=685, tag=0
(context=e8, rank=283, tag=7ffffd8d, sreq=0x2b6bf30)
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=1003, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=1003, tag=7ffffd8d, sreq=0x6c5a3b0)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f3ce6ad5640, &ssend_ack_recv_buffer=0x7f3ce6ad5648, &pad2=0x7f3ce6ad5658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=781, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=781, tag=7ffffd8d, sreq=0x7083320) (context=e8, rank=925, tag=7ffffd8d, sreq=0x706f820)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7fc6fad20640, &ssend_ack_recv_buffer=0x7fc6fad20648, &pad2=0x7fc6fad20658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
(context=e8, rank=787, tag=7ffffd8d, sreq=0x5321fe0)[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=643, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=493, tag=7ffffd8d, sreq=0x33118f0) (context=e8, rank=637, tag=7ffffd8d, sreq=0x3329bf0) (context=e8, rank=643, tag=7ffffd8d, sreq=0x53c3500)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f4fccb25640, &ssend_ack_recv_buffer=0x7f4fccb25648, &pad2=0x7f4fccb25658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=355, tag=7ffffd8d, sreq=0x35250e0)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7fa20945f640, &ssend_ack_recv_buffer=0x7fa20945f648, &pad2=0x7fa20945f658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
(context=e8, rank=565, tag=7ffffd8d, sreq=0x3427ab0) (context=e8, rank=1066, tag=7ffffd8d, sreq=0x3c18980) (context=e8, rank=427, tag=7ffffd8d, sreq=0x451f5b0)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7faa3aebb640, &ssend_ack_recv_buffer=0x7faa3aebb648, &pad2=0x7faa3aebb658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=571, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=571, tag=7ffffd8d, sreq=0x48ef0d0)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f4476827640, &ssend_ack_recv_buffer=0x7f4476827648, &pad2=0x7f4476827658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
(context=e8, rank=955, tag=7ffffd8d, sreq=0x3b650d0)MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=379, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=235, tag=7ffffd8d, sreq=0x482c1c0) (context=e8, rank=238, tag=7ffffd8d, sreq=0x482e5c0) (context=e8, rank=241, tag=7ffffd8d, sreq=0x4823f40) (context=e8, rank=379, tag=7ffffd8d, sreq=0x4834d40) (context=e8, rank=382, tag=7ffffd8d, sreq=0x482ea40) (context=e8, rank=385, tag=7ffffd8d, sreq=0x6ad7440)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f2d5bb45640, &ssend_ack_recv_buffer=0x7f2d5bb45648, &pad2=0x7f2d5bb45658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
(context=e8, rank=913, tag=7ffffd8d, sreq=0x3bd30d0)[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=355, tag=7ffffd8d, sreq=0x2b66530)MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f2eefed4640, &ssend_ack_recv_buffer=0x7f2eefed4648, &pad2=0x7f2eefed4658
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=1003, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=1003, tag=7ffffd8d, sreq=0x643b5b0)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f31d5a6a640, &ssend_ack_recv_buffer=0x7f31d5a6a648, &pad2=0x7f31d5a6a658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
(context=e8, rank=499, tag=7ffffd8d, sreq=0x3f24d70) (context=e8, rank=643, tag=7ffffd8d, sreq=0x3f35ff0)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f98486d5640, &ssend_ack_recv_buffer=0x7f98486d5648, &pad2=0x7f98486d5658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
(context=e8, rank=571, tag=7ffffd8d, sreq=0x3428cb0)
(context=e8, rank=1204, tag=7ffffd8d, sreq=0x3c0b180)[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=418, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=277, tag=7ffffd8d, sreq=0x488a1b0) (context=e8, rank=418, tag=7ffffd8d, sreq=0x486b730) (context=e8, rank=421, tag=7ffffd8d, sreq=0x488c130)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7fa49b2c0640, &ssend_ack_recv_buffer=0x7fa49b2c0648, &pad2=0x7fa49b2c0658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=571, tag=7ffffd8d, sreq=0x6313130)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f7e703f8640, &ssend_ack_recv_buffer=0x7f7e703f8648, &pad2=0x7f7e703f8658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f62d2832640, &ssend_ack_recv_buffer=0x7f62d2832648, &pad2=0x7f62d2832658
(context=e8, rank=958, tag=7ffffd8d, sreq=0x3b71b50)[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=676, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=673, tag=7ffffd8d, sreq=0x5c07e30) (context=e8, rank=676, tag=7ffffd8d, sreq=0x5c03f30) (context=e8, rank=817, tag=7ffffd8d, sreq=0x5c16cb0) (context=e8, rank=820, tag=7ffffd8d, sreq=0x5bfceb0)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7fbafd7b9640, &ssend_ack_recv_buffer=0x7fbafd7b9648, &pad2=0x7fbafd7b9658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=382, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=238, tag=7ffffd8d, sreq=0x41b98b0) (context=e8, rank=241, tag=7ffffd8d, sreq=0x41d5ab0) (context=e8, rank=244, tag=7ffffd8d, sreq=0x41cafb0) (context=e8, rank=382, tag=7ffffd8d, sreq=0x41def30) (context=e8, rank=385, tag=7ffffd8d, sreq=0x41db4b0) (context=e8, rank=388, tag=7ffffd8d, sreq=0x41e0a30)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f1e6287c640, &ssend_ack_recv_buffer=0x7f1e6287c648, &pad2=0x7f1e6287c658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
(context=e8, rank=1051, tag=7ffffd8d, sreq=0x3be9450)MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list:MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f4630746640, &ssend_ack_recv_buffer=0x7f4630746648, &pad2=0x7f4630746658
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f61e4836640, &ssend_ack_recv_buffer=0x7f61e4836648, &pad2=0x7f61e4836658
(context=e8, rank=1207, tag=7ffffd8d, sreq=0x3c25d00)MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
(context=e8, rank=961, tag=7ffffd8d, sreq=0x3b66750)[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=370, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=232, tag=7ffffd8d, sreq=0x35ac730) (context=e8, rank=370, tag=7ffffd8d, sreq=0x35c1430) (context=e8, rank=373, tag=7ffffd8d, sreq=0x35c2630)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f1ee1fe7640, &ssend_ack_recv_buffer=0x7f1ee1fe7648, &pad2=0x7f1ee1fe7658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
(context=e8, rank=1054, tag=7ffffd8d, sreq=0x3be7dd0) (context=e8, rank=982, tag=7ffffd8d, sreq=0x64c6b70) (context=e8, rank=985, tag=7ffffd8d, sreq=0x64afef0) (context=e8, rank=988, tag=7ffffd8d, sreq=0x64c3e70) (context=e8, rank=1054, tag=7ffffd8d, sreq=0x64adaf0) (context=e8, rank=1126, tag=7ffffd8d, sreq=0x64c1170) (context=e8, rank=1129, tag=7ffffd8d, sreq=0x64d47f0) (context=e8, rank=1132, tag=7ffffd8d, sreq=0x64d7070)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f9951f45640, &ssend_ack_recv_buffer=0x7f9951f45648, &pad2=0x7f9951f45658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=415, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=409, tag=7ffffd8d, sreq=0x4028230) (context=e8, rank=415, tag=7ffffd8d, sreq=0x4026730)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7fb0a66d1640, &ssend_ack_recv_buffer=0x7fb0a66d1648, &pad2=0x7fb0a66d1658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
(context=e8, rank=1210, tag=7ffffd8d, sreq=0x3c1c880)[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=556, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=550, tag=7ffffd8d, sreq=0x6c95040) (context=e8, rank=553, tag=7ffffd8d, sreq=0x6c8f1c0) (context=e8, rank=556, tag=7ffffd8d, sreq=0x6c7c440) (context=e8, rank=622, tag=7ffffd8d, sreq=0x6c95dc0) (context=e8, rank=694, tag=7ffffd8d, sreq=0x6c7fa40) (context=e8, rank=697, tag=7ffffd8d, sreq=0x6c7dac0)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7ff4ec6dc640, &ssend_ack_recv_buffer=0x7ff4ec6dc648, &pad2=0x7ff4ec6dc658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=1030, tag=0
(context=e8, rank=1099, tag=7ffffd8d, sreq=0x3b7cf50)[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=394, tag=0
(context=e8, rank=541, tag=7ffffd8d, sreq=0x540b9c0)
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=991, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=991, tag=7ffffd8d, sreq=0x5eaec30) (context=e8, rank=994, tag=7ffffd8d, sreq=0x5eaf530) (context=e8, rank=997, tag=7ffffd8d, sreq=0x5ead5b0) (context=e8, rank=1135, tag=7ffffd8d, sreq=0x5e9b5b0) (context=e8, rank=1138, tag=7ffffd8d, sreq=0x5ebbfb0) (context=e8, rank=1141, tag=7ffffd8d, sreq=0x5eac830)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f5cd28bd640, &ssend_ack_recv_buffer=0x7f5cd28bd648, &pad2=0x7f5cd28bd658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=637, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=637, tag=7ffffd8d, sreq=0x6d4d260) (context=e8, rank=781, tag=7ffffd8d, sreq=0x6d60460)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f76d63dd640, &ssend_ack_recv_buffer=0x7f76d63dd648, &pad2=0x7f76d63dd658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7fa94e9c0640, &ssend_ack_recv_buffer=0x7fa94e9c0648, &pad2=0x7fa94e9c0658
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=697, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=556, tag=7ffffd8d, sreq=0x5b665a0) (context=e8, rank=697, tag=7ffffd8d, sreq=0x5b734a0) (context=e8, rank=703, tag=7ffffd8d, sreq=0x5b8e020)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f53f49e6640, &ssend_ack_recv_buffer=0x7f53f49e6648, &pad2=0x7f53f49e6658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=1033, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=883, tag=7ffffd8d, sreq=0x3a55a80) (context=e8, rank=886, tag=7ffffd8d, sreq=0x3a4e580) (context=e8, rank=889, tag=7ffffd8d, sreq=0x3a61c00) (context=e8, rank=961, tag=7ffffd8d, sreq=0x3a54d00) (context=e8, rank=1027, tag=7ffffd8d, sreq=0x3a6be00) (context=e8, rank=1030, tag=7ffffd8d, sreq=0x3a6e200) (context=e8, rank=1033, tag=7ffffd8d, sreq=0x3a6c700)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7fdac7cca640, &ssend_ack_recv_buffer=0x7fdac7cca648, &pad2=0x7fdac7cca658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
(context=e8, rank=1102, tag=7ffffd8d, sreq=0x3b6d7d0)[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=250, tag=7ffffd8d, sreq=0x35651e0) (context=e8, rank=253, tag=7ffffd8d, sreq=0x354e9e0) (context=e8, rank=256, tag=7ffffd8d, sreq=0x3558760) (context=e8, rank=394, tag=7ffffd8d, sreq=0x356e660) (context=e8, rank=397, tag=7ffffd8d, sreq=0x35705e0) (context=e8, rank=400, tag=7ffffd8d, sreq=0x3557e60)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7ff3039cd640, &ssend_ack_recv_buffer=0x7ff3039cd648, &pad2=0x7ff3039cd658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7fcd9734e640, &ssend_ack_recv_buffer=0x7fcd9734e648, &pad2=0x7fcd9734e658
(context=e8, rank=544, tag=7ffffd8d, sreq=0x53f5640)MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f30e3e7f640, &ssend_ack_recv_buffer=0x7f30e3e7f648, &pad2=0x7f30e3e7f658
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=988, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=988, tag=7ffffd8d, sreq=0x5287b70)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f18da01f640, &ssend_ack_recv_buffer=0x7f18da01f648, &pad2=0x7f18da01f658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f6e0e0d2640, &ssend_ack_recv_buffer=0x7f6e0e0d2648, &pad2=0x7f6e0e0d2658
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=880, tag=7ffffd8d, sreq=0x5e270a0) (context=e8, rank=883, tag=7ffffd8d, sreq=0x5e3aba0) (context=e8, rank=886, tag=7ffffd8d, sreq=0x5e21b20) (context=e8, rank=1024, tag=7ffffd8d, sreq=0x5e231a0) (context=e8, rank=1027, tag=7ffffd8d, sreq=0x5e483a0) (context=e8, rank=1030, tag=7ffffd8d, sreq=0x5e324a0)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f7956b65640, &ssend_ack_recv_buffer=0x7f7956b65648, &pad2=0x7f7956b65658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
(context=e8, rank=1105, tag=7ffffd8d, sreq=0x3b73ad0)MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
(context=e8, rank=547, tag=7ffffd8d, sreq=0x53f63c0)MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=418, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=412, tag=7ffffd8d, sreq=0x3f666b0) (context=e8, rank=418, tag=7ffffd8d, sreq=0x3f6e4b0)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7fe520611640, &ssend_ack_recv_buffer=0x7fe520611648, &pad2=0x7fe520611658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=1003, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=853, tag=7ffffd8d, sreq=0x5a32570) (context=e8, rank=997, tag=7ffffd8d, sreq=0x5a1cf70) (context=e8, rank=1003, tag=7ffffd8d, sreq=0x5a2e1f0)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f8db2022640, &ssend_ack_recv_buffer=0x7f8db2022648, &pad2=0x7f8db2022658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=970, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=970, tag=7ffffd8d, sreq=0x5e5eda0) (context=e8, rank=973, tag=7ffffd8d, sreq=0x5e59ca0) (context=e8, rank=976, tag=7ffffd8d, sreq=0x5e44fa0) (context=e8, rank=1114, tag=7ffffd8d, sreq=0x5e52c20) (context=e8, rank=1117, tag=7ffffd8d, sreq=0x5e5c0a0) (context=e8, rank=1120, tag=7ffffd8d, sreq=0x5e69420)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7fb7b57c3640, &ssend_ack_recv_buffer=0x7fb7b57c3648, &pad2=0x7fb7b57c3658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=919, tag=0
(context=e8, rank=685, tag=7ffffd8d, sreq=0x54032c0)[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=268, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=418, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=418, tag=7ffffd8d, sreq=0x4540ab0) (context=e8, rank=421, tag=7ffffd8d, sreq=0x453fd30) (context=e8, rank=565, tag=7ffffd8d, sreq=0x45491b0)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f1c32ffc640, &ssend_ack_recv_buffer=0x7f1c32ffc648, &pad2=0x7f1c32ffc658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=688, tag=7ffffd8d, sreq=0x53f1740)[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list:MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f494cd94640, &ssend_ack_recv_buffer=0x7f494cd94648, &pad2=0x7f494cd94658
(context=e8, rank=916, tag=7ffffd8d, sreq=0x5d697a0) (context=e8, rank=691, tag=7ffffd8d, sreq=0x53f2940)[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=271, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=271, tag=7ffffd8d, sreq=0x30225b0) (context=e8, rank=277, tag=7ffffd8d, sreq=0x302b130)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f6a3257c640, &ssend_ack_recv_buffer=0x7f6a3257c648, &pad2=0x7f6a3257c658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
(context=e8, rank=919, tag=7ffffd8d, sreq=0x5d68120)[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=532, tag=0
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=532, tag=7ffffd8d, sreq=0x6a31720)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f887ff24640, &ssend_ack_recv_buffer=0x7f887ff24648, &pad2=0x7f887ff24658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
(context=e8, rank=268, tag=7ffffd8d, sreq=0x2e88ab0) (context=e8, rank=271, tag=7ffffd8d, sreq=0x2e8aeb0) (context=e8, rank=274, tag=7ffffd8d, sreq=0x2e80cb0)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7fc286885640, &ssend_ack_recv_buffer=0x7fc286885648, &pad2=0x7fc286885658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: ERROR: can not find matching ssend. context=e8, rank=1117, tag=0
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7efeef587640, &ssend_ack_recv_buffer=0x7efeef587648, &pad2=0x7efeef587658
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f722c9cf640, &ssend_ack_recv_buffer=0x7f722c9cf648, &pad2=0x7f722c9cf658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
[MPID_nem_tmi_pending_ssend_dequeue]: pending ssend list: (context=e8, rank=976, tag=7ffffd8d, sreq=0x67744a0) (context=e8, rank=1117, tag=7ffffd8d, sreq=0x6775b20) (context=e8, rank=1123, tag=7ffffd8d, sreq=0x676eaa0)
MPID_nem_tmi_handle_ssend_ack: &pad1=0x7f8181246640, &ssend_ack_recv_buffer=0x7f8181246648, &pad2=0x7f8181246658
MPID_nem_tmi_handle_ssend_ack: pad1=0, pad2=0 (both should be 0)
===================================================================================
= BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
= PID 455075 RUNNING AT s08r1b31
= EXIT CODE: 9
= CLEANING UP REMAINING PROCESSES
= YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
Intel(R) MPI Library troubleshooting guide:
https://software.intel.com/node/561764
===================================================================================
-------------- next part --------------
[424]PETSC ERROR: ------------------------------------------------------------------------
[424]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
[424]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[424]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[424]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
[726]PETSC ERROR: [424]PETSC ERROR: likely location of problem given in stack below
[424]PETSC ERROR: --------------------- Stack Frames ------------------------------------
[424]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[424]PETSC ERROR: INSTEAD the line number of the start of the function
[424]PETSC ERROR: is given.
[424]PETSC ERROR: [424] PetscCommBuildTwoSidedFReq_Ibarrier line 367 /gpfs/scratch/upc26/upc26229/petsc_cell_agg/debug/petsc-3.9.0/src/sys/utils/mpits.c
[424]PETSC ERROR: [424] PetscCommBuildTwoSidedFReq line 548 /gpfs/scratch/upc26/upc26229/petsc_cell_agg/debug/petsc-3.9.0/src/sys/utils/mpits.c
[726]PETSC ERROR: [424]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[424]PETSC ERROR: Signal received
[424]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[424]PETSC ERROR: Petsc Release Version 3.9.0, Apr, 07, 2018
[424]PETSC ERROR: [1693]PETSC ERROR: ------------------------------------------------------------------------
[424]PETSC ERROR: Configure options --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpiifort -with-blaslapack-dir=/apps/INTEL/2017.4/mkl --with-debugging --with-x=0 --with-shared-libraries=1 --with-mpi=1 --with-64-bit-indices --CFLAGS=-traceback --FFLAGS=-traceback
[424]PETSC ERROR: #1 User provided function() line 0 in unknown file
[726]PETSC ERROR: application called MPI_Abort(MPI_COMM_WORLD, 59) - process 424
orrtl: severe (174): SIGSEGV, segmentation fault occurred
Image PC Routine Line Source
libpetsc.so.3.9.0 00007F4608388A9C for__signal_handl Unknown Unknown
libpthread-2.22.s 00007F460DC5DB10 Unknown Unknown Unknown
libmpi.so.12.0 00007F460E3D1CFF Unknown Unknown Unknown
libmpi.so.12.0 00007F460E736050 Unknown Unknown Unknown
libmpi.so.12.0 00007F460E5EABD0 Unknown Unknown Unknown
libmpi.so.12 00007F460E3AEBC8 PMPIDI_CH3I_Progr Unknown Unknown
libmpi.so.12 00007F460E72B90C MPI_Testall Unknown Unknown
libpetsc.so.3.9.0 00007F4607391FFE PetscCommBuildTwo Unknown Unknown
libpetsc.so.3.9.0 00007F460793A613 Unknown Unknown Unknown
libpetsc.so.3.9.0 00007F4607936EC4 Unknown Unknown Unknown
libpetsc.so.3.9.0 00007F46078AEC47 MatAssemblyBegin_ Unknown Unknown
libpetsc.so.3.9.0 00007F4607587E98 MatAssemblyBegin Unknown Unknown
libpetsc.so.3.9.0 00007F4607921A9D Unknown Unknown Unknown
libpetsc.so.3.9.0 00007F460791A557 Unknown Unknown Unknown
libpetsc.so.3.9.0 00007F4607598DA8 MatTransposeMatMu Unknown Unknown
libpetsc.so.3.9.0 00007F4607DC7AE0 Unknown Unknown Unknown
libpetsc.so.3.9.0 00007F4607DBFFC8 PCSetUp_GAMG Unknown Unknown
libpetsc.so.3.9.0 00007F4607E36A74 PCSetUp Unknown Unknown
libpetsc.so.3.9.0 00007F4607EEAA7E KSPSetUp Unknown Unknown
libpetsc.so.3.9.0 00007F4607EFF40D kspsetup_ Unknown Unknown
par_test_poisson_ 0000000000440FC6 Unknown Unknown Unknown
par_test_poisson_ 000000000045EDCE Unknown Unknown Unknown
par_test_poisson_ 000000000040E4F5 Unknown Unknown Unknown
par_test_poisson_ 00000000004055AE Unknown Unknown Unknown
libc-2.22.so 00007F46034FB6E5 __libc_start_main Unknown Unknown
par_test_poisson_ 00000000004054B9 Unknown Unknown Unknown
More information about the petsc-users
mailing list