[petsc-users] possible performance issues with PETSc on Cray

[Samar Khatiwala]

Hi Jed,

This is useful to know. My matrices are all very sparse but just may not be ordered 
optimally (there's a problem-specific reason why I order them in a certain way). That 
said, this is the first time in many years of similar computations with similar matrices 
that I've encountered this problem. It may just be peculiar to the XC30's.

Thanks,

Samar

On May 9, 2014, at 8:30 AM, Jed Brown <jed at jedbrown.org> wrote:

> Samar Khatiwala <spk at ldeo.columbia.edu> writes:
>> CRAY has finally gotten back with a solution,
>> which is to use the run-time option -vecscatter_alltoall. Apparently
>> this is a known issue and according to the HLRN folks passing this
>> command line option to PETSc seems to work nicely.
> 
> This option is good when you have nearly-dense rows or columns (in terms
> of processors depended on).  For problems with actual dense rows or
> columns, it is good to formulate as a sparse matrix plus a low-rank
> correction.  The other cases are usually poor dof layout, and reordering
> will make the graph sparser.  Sparse problems with good layout usually
> run faster with the default VecScatter, though there are exceptions
> (mostly non-PDE problems).