A 3D example of KSPSolve?

[Shi Jin]

Hi, I am a bit confused at how to interpret the
log_summary results. In my previous log files, I
logged   everything in that solving staging, including
constructing the matrix and vector and the KSPSolve.
I then specifically change the code so that each
KSPSolve() function is tightly included within the 
PetscLogStagePush() and PetscLogStagePop() pair so
that we exclude the other timings and concentrate on
the linear solver.
In this way, I still get list of 16 functions in that
stage, although I only included one (KSPSolve). They
are
VecDot         
VecNorm        
VecCopy        
VecSet         
VecAXPY        
VecAYPX        
VecScatterBegin
VecScatterEnd  
MatMult        
MatSolve       
MatLUFactorNum 
KSPSetup       
KSPSolve       
PCSetUp        
PCSetUpOnBlocks
PCApply        
Are these functions called by the KSPSolve() (in this
case, I used -ksp_type cg).
I suppose the only network communications are done in
the function calls 
VecScatterBegin
VecScatterEnd  
If I am to compute the percentage of communication
specifically for KSPSolve(), shall I just use the
times of VecScatterBegin & VecScatterEnd  devided by
the time of KSPSolve? Or shall I use MatMult, like
Satish did in his previous emails? I am a bit
confused. Please advise.

Thank you very much.

Shi
--- Satish Balay <balay at mcs.anl.gov> wrote:

> 
> Just looking at 8 proc run [diffusion stage] we
> have:
> 
> MatMult        :  79 sec
> MatMultAdd     :   2 sec
> VecScatterBegin:  17 sec
> VecScatterEnd  :  51 sec
> 
> So basically the communication in MatMult/Add is
> represented by
> VecScatters. Here out of 81 sec total - 68 seconds
> are used for
> communication [with a load imbalance of 11 for
> vecscaterend]
> 
> So - I think MPI performance is reducing scalability
> here..
> 
> Things to try:
> 
> * -vecstatter_rr etc options I sugested earlier
> 
> * install mpich with '--with-device=ch3:ssm' and see
> if it makes a difference
> 
> Satish
> 
> --- Event Stage 4: Diffusion
> 
> [x]rhsLtP            297 1.0 1.1017e+02 1.5 0.00e+00
> 0.0 0.0e+00 0.0e+00 0.0e+00  7  0  0  0  0  39  0  0
>  0  0     0
> [x]rhsGravity         99 1.0 4.2582e+0083.5 0.00e+00
> 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0
>  0  0     0
> VecDot              4657 1.0 2.5748e+01 3.2 7.60e+07
> 3.2 0.0e+00 0.0e+00 4.7e+03  1  1  0  0  6   5  3  0
>  0 65   191
> VecNorm             2477 1.0 2.2109e+01 2.2 3.22e+07
> 2.2 0.0e+00 0.0e+00 2.5e+03  1  0  0  0  3   5  2  0
>  0 35   118
> VecScale             594 1.0 2.9330e-02 1.5 2.61e+08
> 1.5 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0
>  0  0  1361
> VecCopy              594 1.0 2.7552e-01 1.3 0.00e+00
> 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0
>  0  0     0
> VecSet              3665 1.0 6.0793e-01 1.4 0.00e+00
> 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0
>  0  0     0
> VecAXPY             5251 1.0 2.5892e+00 1.2 3.31e+08
> 1.2 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   1  4  0
>  0  0  2137
> VecAYPX             1883 1.0 8.6419e-01 1.3 3.62e+08
> 1.3 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  1  0
>  0  0  2296
> VecScatterBegin     2873 1.0 1.7569e+01 3.0 0.00e+00
> 0.0 3.8e+04 1.6e+05 0.0e+00  1  0 10 20  0   5 
> 0100100  0     0
> VecScatterEnd       2774 1.0 5.1519e+0110.9 0.00e+00
> 0.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   7  0  0
>  0  0     0
> MatMult             2477 1.0 7.9186e+01 2.4 2.34e+08
> 2.4 3.5e+04 1.7e+05 0.0e+00  3 11  9 20  0  20 48 91
> 98  0   850
> MatMultAdd           297 1.0 2.8161e+00 5.4 4.46e+07
> 2.2 3.6e+03 3.4e+04 0.0e+00  0  0  1  0  0   0  0  9
>  2  0   125
> MatSolve            2477 1.0 6.2245e+01 1.2 1.41e+08
> 1.2 0.0e+00 0.0e+00 0.0e+00  4 10  0  0  0  22 41  0
>  0  0   926
> MatLUFactorNum         3 1.0 2.7686e-01 1.1 2.79e+08
> 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0
>  0  0  2016
> MatGetRow        19560420 1.0 5.5195e+01 1.6
> 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  3  0  0  0  0 
> 20  0  0  0  0     0
> KSPSetup               6 1.0 3.0756e-05 2.8 0.00e+00
> 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0
>  0  0     0
> KSPSolve             297 1.0 1.3142e+02 1.0 1.31e+08
> 1.1 3.1e+04 1.7e+05 7.1e+03  8 22  8 18  9  50 93 80
> 86100  1001
> PCSetUp                6 1.0 2.7700e-01 1.1 2.78e+08
> 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0
>  0  0  2015
> PCSetUpOnBlocks      297 1.0 2.7794e-01 1.1 2.78e+08
> 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0
>  0  0  2008
> PCApply             2477 1.0 6.2772e+01 1.2 1.39e+08
> 1.2 0.0e+00 0.0e+00 0.0e+00  4 10  0  0  0  23 41  0
>  0  0   918
> 
> 



 
____________________________________________________________________________________
It's here! Your new message!  
Get new email alerts with the free Yahoo! Toolbar.
http://tools.search.yahoo.com/toolbar/features/mail/