[petsc-dev] Wrapper for WSMP

[Xiaoye S. Li]

I can clarify a couple of questions re. SuperLU_DIST.

1) SuperLU does support multiple right-hand sides.  That is, the B matrix on
the right can be a dense matrix of size n-by-nrhs.   Also, the B matrix is
distributed among all the processors, each processor takes one block row of
B.  There is no need to have the entire B on every processor.

2) We are preparing to upgrade to a newer version.  The parallel
factorization is improved with a better scheduling algorithm.  This is
particularly effective using larger core count, say in 100s.

3) Regarding memory usage, the factorization algorithm used in SuperLU
mostly performs "update in-place", and requires just a little bit extra
working storage.  So, if the load balance is not too bad, the memory per
core should go down consistently.   It will be helpful if you can give some
concreate numbers showing how much memory usage increases with increasing
core count.


Sherry Li



On Tue, Aug 16, 2011 at 8:34 PM, Rebecca Yuan <rebeccayxf at gmail.com> wrote:

>
>
>
> Begin forwarded message:
>
> *From:* Jack Poulson <jack.poulson at gmail.com>>
> *Date:* August 16, 2011 10:18:16 PM CDT
>
> *To:* For users of the development version of PETSc <petsc-dev at mcs.anl.gov
> >
> *Subject:* *Re: [petsc-dev] Wrapper for WSMP*
> *Reply-To:* For users of the development version of PETSc <
> petsc-dev at mcs.anl.gov>
>
> On Tue, Aug 16, 2011 at 9:35 PM, Barry Smith < <bsmith at mcs.anl.gov>
> bsmith at mcs.anl.gov> wrote:
>
>>
>> On Aug 16, 2011, at 5:14 PM, Jack Poulson wrote:
>>
>> > Hello all,
>> >
>> > I am working on a project that requires very fast sparse direct solves
>> and MUMPS and SuperLU_Dist haven't been cutting it. From what I've read,
>> when properly tuned, WSMP is significantly faster, particularly with
>> multiple right-hand sides on large machines. The obvious drawback is that
>> it's not open source, but the binaries seem to be readily available for most
>> platforms.
>> >
>> > Before I reinvent the wheel, I would like to check if anyone has already
>> done some work on adding it into PETSc. If not, its interface is quite
>> similar to MUMPS and I should be able to mirror most of that code. On the
>> other hand, there are a large number of platform-specific details that need
>> to be handled, so keeping things both portable and fast might be a
>> challenge. It seems that the CSC storage format should be used since it is
>> required for Hermitian matrices.
>> >
>> > Thanks,
>> > Jack
>>
>>   Jack,
>>
>>   By all means do it. That would be a nice thing to have. But be aware
>> that the WSMP folks have a reputation for exaggerating how much better their
>> software is so don't be surprised if after all that work it is not much
>> better.
>>
>>
> Good to know. I was somewhat worried about that, but perhaps it is a matter
> of getting all of the tuning parameters right. The manual does mention that
> performance is significantly degraded without tuning. I would sincerely hope
> no one would out right lie in their publications, e.g., this one:
>  <http://portal.acm.org/citation.cfm?id=1654061>
> http://portal.acm.org/citation.cfm?id=1654061
>
>
>>   BTW: are you solving with many right hand sides? Maybe before you muck
>> with WSMP we should figure out how to get you access to the multiple right
>> hand side support of MUMPS (I don't know if SuperLU_Dist has it) so you can
>> speed up your current computations a good amount? Currently PETSc's
>> MatMatSolve() calls a separate solve for each right hand side with MUMPS.
>>
>>   Barry
>>
>>
> I will eventually need to solve against many right-hand sides, but for now
> I am solving against one and it is still taking too long; in fact, not only
> does it take too long, but memory per core increased for fixed problem sizes
> as I increase the number of MPI processes (for both SuperLU_Dist and MUMPS).
> This was occurring for quasi2d Helmholtz problems over a couple hundred
> cores. My only logical explanation for this behavior is that the
> communication buffers grow proportional to the number of processes on each
> process, but I stress that this is just a guess. I tried reading through the
> MUMPS code and quickly gave up.
>
> Another problem with MUMPS is that requires the entire set of right-hand
> sides to reside on the root process...that will clearly not work for a
> billion degrees of freedom with several hundred RHSs. WSMP gets this part
> right and actually distributes those vectors.
>
> Jack
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20110818/0f417931/attachment.html>