[petsc-users] possible performance issues with PETSc on Cray
Samar Khatiwala
spk at ldeo.columbia.edu
Fri May 9 07:40:37 CDT 2014
Hi Jed,
This is useful to know. My matrices are all very sparse but just may not be ordered
optimally (there's a problem-specific reason why I order them in a certain way). That
said, this is the first time in many years of similar computations with similar matrices
that I've encountered this problem. It may just be peculiar to the XC30's.
Thanks,
Samar
On May 9, 2014, at 8:30 AM, Jed Brown <jed at jedbrown.org> wrote:
> Samar Khatiwala <spk at ldeo.columbia.edu> writes:
>> CRAY has finally gotten back with a solution,
>> which is to use the run-time option -vecscatter_alltoall. Apparently
>> this is a known issue and according to the HLRN folks passing this
>> command line option to PETSc seems to work nicely.
>
> This option is good when you have nearly-dense rows or columns (in terms
> of processors depended on). For problems with actual dense rows or
> columns, it is good to formulate as a sparse matrix plus a low-rank
> correction. The other cases are usually poor dof layout, and reordering
> will make the graph sparser. Sparse problems with good layout usually
> run faster with the default VecScatter, though there are exceptions
> (mostly non-PDE problems).
More information about the petsc-users
mailing list