A 3D example of KSPSolve?

Sat Feb 10 18:43:33 CST 2007

On Sat, 10 Feb 2007, Shi Jin wrote:

> Hi, I am a bit confused at how to interpret the
> log_summary results. In my previous log files, I
> logged   everything in that solving staging, including
> constructing the matrix and vector and the KSPSolve.
> I then specifically change the code so that each
> KSPSolve() function is tightly included within the 
> PetscLogStagePush() and PetscLogStagePop() pair so
> that we exclude the other timings and concentrate on
> the linear solver.
> In this way, I still get list of 16 functions in that
> stage, although I only included one (KSPSolve). They
> are
> VecDot         
> VecNorm        
> VecCopy        
> VecSet         
> VecAXPY        
> VecAYPX        
> VecScatterBegin
> VecScatterEnd  
> MatMult        
> MatSolve       
> MatLUFactorNum 
> KSPSetup       
> KSPSolve       
> PCSetUp        
> PCSetUpOnBlocks
> PCApply        
> Are these functions called by the KSPSolve() (in this
> case, I used -ksp_type cg).

  YES

> I suppose the only network communications are done in
> the function calls 
> VecScatterBegin
> VecScatterEnd  

  The message passing. VecDot, VecNorm have MPI_Allreduce()s
> If I am to compute the percentage of communication
> specifically for KSPSolve(), shall I just use the
> times of VecScatterBegin & VecScatterEnd  devided by
> the time of KSPSolve? Or shall I use MatMult, like
> Satish did in his previous emails? I am a bit
> confused. Please advise.

  You can do either; using Mult tells you how well the mult is doing
in terms of message passing communication. Using ksp tells how
in the entire solve.

  You can add the option -log_sync and it will try to seperate the 
amount of time in the dot, norm and scatters that is actually spent on
communication and how much is spent on synchronization (due to load inbalance).

  Barry

> 
> Thank you very much.
> 
> Shi
> --- Satish Balay <balay at mcs.anl.gov> wrote:
> 
> > 
> > Just looking at 8 proc run [diffusion stage] we
> > have:
> > 
> > MatMult        :  79 sec
> > MatMultAdd     :   2 sec
> > VecScatterBegin:  17 sec
> > VecScatterEnd  :  51 sec
> > 
> > So basically the communication in MatMult/Add is
> > represented by
> > VecScatters. Here out of 81 sec total - 68 seconds
> > are used for
> > communication [with a load imbalance of 11 for
> > vecscaterend]
> > 
> > So - I think MPI performance is reducing scalability
> > here..
> > 
> > Things to try:
> > 
> > * -vecstatter_rr etc options I sugested earlier
> > 
> > * install mpich with '--with-device=ch3:ssm' and see
> > if it makes a difference
> > 
> > Satish
> > 
> > --- Event Stage 4: Diffusion
> > 
> > [x]rhsLtP            297 1.0 1.1017e+02 1.5 0.00e+00
> > 0.0 0.0e+00 0.0e+00 0.0e+00  7  0  0  0  0  39  0  0
> >  0  0     0
> > [x]rhsGravity         99 1.0 4.2582e+0083.5 0.00e+00
> > 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   1  0  0
> >  0  0     0
> > VecDot              4657 1.0 2.5748e+01 3.2 7.60e+07
> > 3.2 0.0e+00 0.0e+00 4.7e+03  1  1  0  0  6   5  3  0
> >  0 65   191
> > VecNorm             2477 1.0 2.2109e+01 2.2 3.22e+07
> > 2.2 0.0e+00 0.0e+00 2.5e+03  1  0  0  0  3   5  2  0
> >  0 35   118
> > VecScale             594 1.0 2.9330e-02 1.5 2.61e+08
> > 1.5 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0
> >  0  0  1361
> > VecCopy              594 1.0 2.7552e-01 1.3 0.00e+00
> > 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0
> >  0  0     0
> > VecSet              3665 1.0 6.0793e-01 1.4 0.00e+00
> > 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0
> >  0  0     0
> > VecAXPY             5251 1.0 2.5892e+00 1.2 3.31e+08
> > 1.2 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   1  4  0
> >  0  0  2137
> > VecAYPX             1883 1.0 8.6419e-01 1.3 3.62e+08
> > 1.3 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  1  0
> >  0  0  2296
> > VecScatterBegin     2873 1.0 1.7569e+01 3.0 0.00e+00
> > 0.0 3.8e+04 1.6e+05 0.0e+00  1  0 10 20  0   5 
> > 0100100  0     0
> > VecScatterEnd       2774 1.0 5.1519e+0110.9 0.00e+00
> > 0.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   7  0  0
> >  0  0     0
> > MatMult             2477 1.0 7.9186e+01 2.4 2.34e+08
> > 2.4 3.5e+04 1.7e+05 0.0e+00  3 11  9 20  0  20 48 91
> > 98  0   850
> > MatMultAdd           297 1.0 2.8161e+00 5.4 4.46e+07
> > 2.2 3.6e+03 3.4e+04 0.0e+00  0  0  1  0  0   0  0  9
> >  2  0   125
> > MatSolve            2477 1.0 6.2245e+01 1.2 1.41e+08
> > 1.2 0.0e+00 0.0e+00 0.0e+00  4 10  0  0  0  22 41  0
> >  0  0   926
> > MatLUFactorNum         3 1.0 2.7686e-01 1.1 2.79e+08
> > 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0
> >  0  0  2016
> > MatGetRow        19560420 1.0 5.5195e+01 1.6
> > 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  3  0  0  0  0 
> > 20  0  0  0  0     0
> > KSPSetup               6 1.0 3.0756e-05 2.8 0.00e+00
> > 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0
> >  0  0     0
> > KSPSolve             297 1.0 1.3142e+02 1.0 1.31e+08
> > 1.1 3.1e+04 1.7e+05 7.1e+03  8 22  8 18  9  50 93 80
> > 86100  1001
> > PCSetUp                6 1.0 2.7700e-01 1.1 2.78e+08
> > 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0
> >  0  0  2015
> > PCSetUpOnBlocks      297 1.0 2.7794e-01 1.1 2.78e+08
> > 1.1 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0
> >  0  0  2008
> > PCApply             2477 1.0 6.2772e+01 1.2 1.39e+08
> > 1.2 0.0e+00 0.0e+00 0.0e+00  4 10  0  0  0  23 41  0
> >  0  0   918
> > 
> > 
> 
> 
> 
>  
> ____________________________________________________________________________________
> It's here! Your new message!  
> Get new email alerts with the free Yahoo! Toolbar.
> http://tools.search.yahoo.com/toolbar/features/mail/
> 
> 