[petsc-users] GAMG setup scalability

[Mark F. Adams]

On Aug 17, 2012, at 10:59 AM, John Fettig wrote:

> Mark,
> 
> Thanks for your response, comments inline:
> 
> On Fri, Aug 17, 2012 at 10:50 AM, Mark F. Adams <mark.adams at columbia.edu> wrote:
>> Yes John, the GAMG setup is not great.  So we know there are problems.  We do have a new staff person that will be working on matrix product methods soon.  Matrix products are hard to make fast and GAMG setup relies on them quite a bit.
>> 
>> A few things to note:
>> 
>> 1) This is in the mesh setup phase, so many applications can amortize this cost.  (MatPtAP is a "matrix" setup cost and so it is amortized for linear problems or non-full Newton solves, but not for full nonlinear solves).
> 
> I'd be willing to live with the MatPtAP time, it is the
> MatTransposeMatMult time that is really poor.  Any idea why the latter
> is so slow?

Its a complex problem and I did not work on the code.

> 
>> 3) This test is getting less than 2x speedup in KSPSolve for 8x processors.  So this problem looks hard: small or poorly partitioned, and not in the range of where we want people to run to get good performance.
> 
> It's about 725k unknowns, and it was run with ex10 from the ksp
> tutorials (i.e. loaded from a file and solved).  It is possible the
> partitioning isn't great.

This big enough and the partitioning is fine for our examples.  If this is a shared memory machine then you are probably seeing memory bandwidth effects (one processor gets the memory bandwidth of the whole machine and rocks). If this is distributed memory then it looks like you have a crapy network.

> 
>> 4) I have found that the setup times are about twice that of ML, which uses a similar algorithm, and about 5x slower than hypre, which uses a very different algorithm.  So if you can not amortize theses setup costs then ML or hypre would probably work better for you.
> 
> I think you actually have this backwards, or your experience is
> different from mine.  In my experience ML's setup is much faster than
> that of hypre.

2D and 3D can make a big difference.  Hypre has problems in 3D (that can be addressed to some degree with parameters).  But yes my numbers were from limited experiments a while ago so you probably have a better idea than me.

Mark

> 
> John
>