profiling PETSc code

Matt Funk mafunk at nmsu.edu
Tue Aug 15 11:34:00 CDT 2006


Hi Matt,

sorry for the delay since the last email, but there were some other things i 
needed to do.

Anyway, I hope that maybe I can get some more help from you guys with respect 
to the loadimbalance problem i have. Here is the situtation:
I run my code on 2 procs. I profile my KSPSolve call and here is what i get:

...

--- Event Stage 2: Stage 2 of ChomboPetscInterface

VecDot             20000 1.0 1.9575e+01 2.0 2.39e+08 2.0 0.0e+00 0.0e+00 
2.0e+04  2  8  0  0 56   7  8  0  0 56   245
VecNorm            16000 1.0 4.0559e+01 3.2 1.53e+08 3.2 0.0e+00 0.0e+00 
1.6e+04  3  7  0  0 44  13  7  0  0 44    95
VecCopy             4000 1.0 1.5148e+00 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 
0.0e+00  0  0  0  0  0   1  0  0  0  0     0
VecSet             16000 1.0 3.1937e+00 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 
0.0e+00  0  0  0  0  0   1  0  0  0  0     0
VecAXPY            16000 1.0 8.2395e+00 1.4 3.22e+08 1.4 0.0e+00 0.0e+00 
0.0e+00  1  7  0  0  0   3  7  0  0  0   465
VecAYPX             8000 1.0 3.6370e+00 1.3 3.46e+08 1.3 0.0e+00 0.0e+00 
0.0e+00  0  3  0  0  0   2  3  0  0  0   527
VecScatterBegin    12000 1.0 5.7721e-01 1.1 0.00e+00 0.0 2.4e+04 2.1e+04 
0.0e+00  0  0100100  0   0  0100100  0     0
VecScatterEnd      12000 1.0 1.4059e+01 9.2 0.00e+00 0.0 0.0e+00 0.0e+00 
0.0e+00  1  0  0  0  0   4  0  0  0  0     0
MatMult            12000 1.0 5.0567e+01 1.0 1.82e+08 1.0 2.4e+04 2.1e+04 
0.0e+00  5 32100100  0  25 32100100  0   361
MatSolve           16000 1.0 1.1285e+02 1.4 1.50e+08 1.4 0.0e+00 0.0e+00 
0.0e+00 10 43  0  0  0  47 43  0  0  0   214
MatLUFactorNum         1 1.0 1.9693e-02 1.0 5.79e+07 1.1 0.0e+00 0.0e+00 
0.0e+00  0  0  0  0  0   0  0  0  0  0   110
MatILUFactorSym        1 1.0 7.8840e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 
1.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetOrdering         1 1.0 1.2250e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 
2.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSetup               1 1.0 1.1921e-06 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 
0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSolve            4000 1.0 2.0419e+02 1.0 1.39e+08 1.0 2.4e+04 2.1e+04 
3.6e+04 21100100100100 100100100100100   278
PCSetUp                1 1.0 2.8828e-02 1.1 3.99e+07 1.1 0.0e+00 0.0e+00 
3.0e+00  0  0  0  0  0   0  0  0  0  0    75
PCSetUpOnBlocks     4000 1.0 3.5605e-02 1.1 3.35e+07 1.1 0.0e+00 0.0e+00 
3.0e+00  0  0  0  0  0   0  0  0  0  0    61
PCApply            16000 1.0 1.1661e+02 1.4 1.46e+08 1.4 0.0e+00 0.0e+00 
0.0e+00 10 43  0  0  0  49 43  0  0  0   207
------------------------------------------------------------------------------------------------------------------------

...


Some things to note are the following: 
I allocate my vector as: 
 VecCreateMPI(PETSC_COMM_WORLD, //communicator
	       a_totallocal_numPoints[a_thisproc], //local points on this proc
	       a_totalglobal_numPoints, //total number of global points
	       &m_globalRHSVector); //the vector to be created

where the vector a_totallocal_numPoints is : 
a_totallocal_numPoints: 59904 59904 

The matrix is allocated as: 
  m_ierr = MatCreateMPIAIJ(PETSC_COMM_WORLD, //communicator
			   a_totallocal_numPoints[a_thisproc], //total number of local rows (that 
is rows residing on this proc
			   a_totallocal_numPoints[a_thisproc], //total number of columns 
corresponding to local part of parallel vector
			   a_totalglobal_numPoints, //number of global rows
			   a_totalglobal_numPoints, //number of global columns
			   PETSC_NULL,
			   a_NumberOfNZPointsInDiagonalMatrix,
			   PETSC_NULL,
			   a_NumberOfNZPointsInOffDiagonalMatrix,
			   &m_globalMatrix);

With the info option i checked and there is no extra mallocs at all.
My problems setup is symmetric so it seems that everything is set up so that 
it should be essentially perfectly balanced. However, the numbers given above 
certainly do not reflect that.

However, the in all other parts of my code (except the PETSc call), i get the 
expected, almost perfect loadbalance. 

Is there anything that i am overlooking? Any help is greatly appreciated.

thanks
mat



On Wednesday 02 August 2006 16:21, Matt Funk wrote:
> Hi Matt,
>
> It could be a bad load imbalance because i don't let PETSc decide. I need
> to fix that anyway, so i think i'll try that first and then let you know.
> Thanks though for the quick response and helping me to interpret those
> numbers ...
>
>
> mat
>
> On Wednesday 02 August 2006 15:50, Matthew Knepley wrote:
> > On 8/2/06, Matt Funk <mafunk at nmsu.edu> wrote:
> > > Hi Matt,
> > >
> > > thanks for all the help so far. The -info option is really very
> > > helpful. So i think i straightened the actual errors out. However, now
> > > i am back to the original question i had. That is why it takes so much
> > > longer on 4 procs than on 1 proc.
> >
> > So you have a 1.5 load imbalance for MatMult(), which probably cascades
> > to give the 133! load imbalance for VecDot(). You probably have either:
> >
> >   1) VERY bad laod imbalance
> >
> >   2) a screwed up network
> >
> >   3) bad contention on the network (loaded cluster)
> >
> > Can you help us narrow this down?
> >
> >
> >    Matt
> >
> > > I profiled the KSPSolve(...) as stage 2:
> > >
> > > For 1 proc i have:
> > > --- Event Stage 2: Stage 2 of ChomboPetscInterface
> > >
> > > VecDot              4000 1.0 4.9158e-02 1.0 4.74e+08 1.0 0.0e+00
> > > 0.0e+00 0.0e+00  0 18  0  0  0   2 18  0  0  0   474
> > > VecNorm             8000 1.0 2.1798e-01 1.0 2.14e+08 1.0 0.0e+00
> > > 0.0e+00 4.0e+03  1 36  0  0 28   7 36  0  0 33   214
> > > VecAYPX             4000 1.0 1.3449e-01 1.0 1.73e+08 1.0 0.0e+00
> > > 0.0e+00 0.0e+00  0 18  0  0  0   5 18  0  0  0   173
> > > MatMult             4000 1.0 3.6004e-01 1.0 3.24e+07 1.0 0.0e+00
> > > 0.0e+00 0.0e+00  1  9  0  0  0  12  9  0  0  0    32
> > > MatSolve            8000 1.0 1.0620e+00 1.0 2.19e+07 1.0 0.0e+00
> > > 0.0e+00 0.0e+00  3 18  0  0  0  36 18  0  0  0    22
> > > KSPSolve            4000 1.0 2.8338e+00 1.0 4.52e+07 1.0 0.0e+00
> > > 0.0e+00 1.2e+04  7100  0  0 84  97100  0  0100    45
> > > PCApply             8000 1.0 1.1133e+00 1.0 2.09e+07 1.0 0.0e+00
> > > 0.0e+00 0.0e+00  3 18  0  0  0  38 18  0  0  0    21
> > >
> > >
> > > for 4 procs i have :
> > > --- Event Stage 2: Stage 2 of ChomboPetscInterface
> > >
> > > VecDot              4000 1.0 3.5884e+01133.7 2.17e+07133.7 0.0e+00
> > > 0.0e+00 4.0e+03  8 18  0  0  5   9 18  0  0 14     1
> > > VecNorm             8000 1.0 3.4986e-01 1.3 4.43e+07 1.3 0.0e+00
> > > 0.0e+00 8.0e+03  0 36  0  0 10   0 36  0  0 29   133
> > > VecSet              8000 1.0 3.5024e-02 1.4 0.00e+00 0.0 0.0e+00
> > > 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> > > VecAYPX             4000 1.0 5.6790e-02 1.3 1.28e+08 1.3 0.0e+00
> > > 0.0e+00 0.0e+00  0 18  0  0  0   0 18  0  0  0   410
> > > VecScatterBegin     4000 1.0 6.0042e+01 1.4 0.00e+00 0.0 0.0e+00
> > > 0.0e+00 0.0e+00 38  0  0  0  0  45  0  0  0  0     0
> > > VecScatterEnd       4000 1.0 5.9364e+01 1.4 0.00e+00 0.0 0.0e+00
> > > 0.0e+00 0.0e+00 37  0  0  0  0  44  0  0  0  0     0
> > > MatMult             4000 1.0 1.1959e+02 1.4 3.46e+04 1.4 0.0e+00
> > > 0.0e+00 0.0e+00 75  9  0  0  0  89  9  0  0  0     0
> > > MatSolve            8000 1.0 2.8150e-01 1.0 2.16e+07 1.0 0.0e+00
> > > 0.0e+00 0.0e+00  0 18  0  0  0   0 18  0  0  0    83
> > > MatLUFactorNum         1 1.0 1.3685e-04 1.1 5.64e+06 1.1 0.0e+00
> > > 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0    21
> > > MatILUFactorSym        1 1.0 2.3389e-04 1.2 0.00e+00 0.0 0.0e+00
> > > 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
> > > MatGetOrdering         1 1.0 9.6083e-05 1.2 0.00e+00 0.0 0.0e+00
> > > 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
> > > KSPSetup               1 1.0 2.1458e-06 2.2 0.00e+00 0.0 0.0e+00
> > > 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> > > KSPSolve            4000 1.0 1.2200e+02 1.0 2.63e+05 1.0 0.0e+00
> > > 0.0e+00 2.8e+04 84100  0  0 34 100100  0  0100     1
> > > PCSetUp                1 1.0 5.0187e-04 1.2 1.68e+06 1.2 0.0e+00
> > > 0.0e+00 4.0e+00  0  0  0  0  0   0  0  0  0  0     6
> > > PCSetUpOnBlocks     4000 1.0 1.2104e-02 2.2 1.34e+05 2.2 0.0e+00
> > > 0.0e+00 4.0e+00  0  0  0  0  0   0  0  0  0  0     0
> > > PCApply             8000 1.0 8.4254e-01 1.2 8.27e+06 1.2 0.0e+00
> > > 0.0e+00 8.0e+03  1 18  0  0 10   1 18  0  0 29    28
> > > -----------------------------------------------------------------------
> > >-- -----------------------------------------------
> > >
> > > Now if i understand it right, all these calls summarize all calls
> > > between the pop and push commands. That would mean that the majority of
> > > the time is spend in the MatMult and in within that the VecScatterBegin
> > > and VecScatterEnd commands (if i understand it right).
> > >
> > > My problem size is really small. So i was wondering if the problem lies
> > > in that (namely that the major time is simply spend communicating
> > > between processors, or whether there is still something wrong with how
> > > i wrote the code?)
> > >
> > >
> > > thanks
> > > mat
> > >
> > > On Tuesday 01 August 2006 18:28, Matthew Knepley wrote:
> > > > On 8/1/06, Matt Funk <mafunk at nmsu.edu> wrote:
> > > > > Actually the errors occur on my calls to a PETSc functions after
> > > > > calling PETSCInitialize.
> > > >
> > > > Yes, it is the error I pointed out in the last message.
> > > >
> > > >    Matt
> > > >
> > > > > mat




More information about the petsc-users mailing list