A 3D example of KSPSolve?

[Satish Balay]

Just looking at 8 proc run [diffusion stage] we have:

MatMult        :  79 sec
MatMultAdd     :   2 sec
VecScatterBegin:  17 sec
VecScatterEnd  :  51 sec

So basically the communication in MatMult/Add is represented by
VecScatters. Here out of 81 sec total - 68 seconds are used for
communication [with a load imbalance of 11 for vecscaterend]

So - I think MPI performance is reducing scalability here..

Things to try:

* -vecstatter_rr etc options I sugested earlier

* install mpich with '--with-device=ch3:ssm' and see
if it makes a difference

Satish

--- Event Stage 4: Diffusion

[x]rhsLtP            297 1.0 1.1017e+02 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  7  0  0  0  0  39  0  0  0  0     0
[x]rhsGravity         99 1.0 4.2582e+0083.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0  0  0     0
VecDot              4657 1.0 2.5748e+01 3.2 7.60e+07 3.2 0.0e+00 0.0e+00 4.7e+03  1  1  0  0  6   5  3  0  0 65   191
VecNorm             2477 1.0 2.2109e+01 2.2 3.22e+07 2.2 0.0e+00 0.0e+00 2.5e+03  1  0  0  0  3   5  2  0  0 35   118
VecScale             594 1.0 2.9330e-02 1.5 2.61e+08 1.5 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  1361
VecCopy              594 1.0 2.7552e-01 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet              3665 1.0 6.0793e-01 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPY             5251 1.0 2.5892e+00 1.2 3.31e+08 1.2 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   1  4  0  0  0  2137
VecAYPX             1883 1.0 8.6419e-01 1.3 3.62e+08 1.3 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  1  0  0  0  2296
VecScatterBegin     2873 1.0 1.7569e+01 3.0 0.00e+00 0.0 3.8e+04 1.6e+05 0.0e+00  1  0 10 20  0   5  0100100  0     0
VecScatterEnd       2774 1.0 5.1519e+0110.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   7  0  0  0  0     0
MatMult             2477 1.0 7.9186e+01 2.4 2.34e+08 2.4 3.5e+04 1.7e+05 0.0e+00  3 11  9 20  0  20 48 91 98  0   850
MatMultAdd           297 1.0 2.8161e+00 5.4 4.46e+07 2.2 3.6e+03 3.4e+04 0.0e+00  0  0  1  0  0   0  0  9  2  0   125
MatSolve            2477 1.0 6.2245e+01 1.2 1.41e+08 1.2 0.0e+00 0.0e+00 0.0e+00  4 10  0  0  0  22 41  0  0  0   926
MatLUFactorNum         3 1.0 2.7686e-01 1.1 2.79e+08 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  2016
MatGetRow        19560420 1.0 5.5195e+01 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  3  0  0  0  0  20  0  0  0  0     0
KSPSetup               6 1.0 3.0756e-05 2.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSolve             297 1.0 1.3142e+02 1.0 1.31e+08 1.1 3.1e+04 1.7e+05 7.1e+03  8 22  8 18  9  50 93 80 86100  1001
PCSetUp                6 1.0 2.7700e-01 1.1 2.78e+08 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  2015
PCSetUpOnBlocks      297 1.0 2.7794e-01 1.1 2.78e+08 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  2008
PCApply             2477 1.0 6.2772e+01 1.2 1.39e+08 1.2 0.0e+00 0.0e+00 0.0e+00  4 10  0  0  0  23 41  0  0  0   918