I can clarify a couple of questions re. SuperLU_DIST.<br><br>1) SuperLU does support multiple right-hand sides. That is, the B matrix on the right can be a dense matrix of size n-by-nrhs. Also, the B matrix is distributed among all the processors, each processor takes one block row of B. There is no need to have the entire B on every processor.<br>
<br>2) We are preparing to upgrade to a newer version. The parallel factorization is improved with a better scheduling algorithm. This is particularly effective using larger core count, say in 100s.<br><br>3) Regarding memory usage, the factorization algorithm used in SuperLU mostly performs "update in-place", and requires just a little bit extra working storage. So, if the load balance is not too bad, the memory per core should go down consistently. It will be helpful if you can give some concreate numbers showing how much memory usage increases with increasing core count.<br>
<br><br>Sherry Li<br><br><br><br><div class="gmail_quote">On Tue, Aug 16, 2011 at 8:34 PM, Rebecca Yuan <span dir="ltr"><<a href="mailto:rebeccayxf@gmail.com">rebeccayxf@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div bgcolor="#FFFFFF"><div><br><br><br>Begin forwarded message:<br><br></div><blockquote type="cite"><div><b>From:</b> Jack Poulson<a href="mailto:jack.poulson@gmail.com" target="_blank"></a>><br><b>Date:</b> August 16, 2011 10:18:16 PM CDT<div class="im">
<br><b>To:</b> For users of the development version of PETSc <<a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a>><br><b>Subject:</b> <b>Re: [petsc-dev] Wrapper for WSMP</b><br><b>Reply-To:</b> For users of the development version of PETSc <<a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a>><br>
<br></div></div></blockquote><div></div><blockquote type="cite"><div><div class="im">On Tue, Aug 16, 2011 at 9:35 PM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank"></a><a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br>
</div><div class="gmail_quote"><div class="im"><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div><div></div><div><br>
On Aug 16, 2011, at 5:14 PM, Jack Poulson wrote:<br>
<br>
> Hello all,<br>
><br>
> I am working on a project that requires very fast sparse direct solves and MUMPS and SuperLU_Dist haven't been cutting it. From what I've read, when properly tuned, WSMP is significantly faster, particularly with multiple right-hand sides on large machines. The obvious drawback is that it's not open source, but the binaries seem to be readily available for most platforms.<br>
><br>
> Before I reinvent the wheel, I would like to check if anyone has already done some work on adding it into PETSc. If not, its interface is quite similar to MUMPS and I should be able to mirror most of that code. On the other hand, there are a large number of platform-specific details that need to be handled, so keeping things both portable and fast might be a challenge. It seems that the CSC storage format should be used since it is required for Hermitian matrices.<br>
><br>
> Thanks,<br>
> Jack<br>
<br>
</div></div> Jack,<br>
<br>
By all means do it. That would be a nice thing to have. But be aware that the WSMP folks have a reputation for exaggerating how much better their software is so don't be surprised if after all that work it is not much better.<br>
<br></blockquote><div><br></div></div><div>Good to know. I was somewhat worried about that, but perhaps it is a matter of getting all of the tuning parameters right. The manual does mention that performance is significantly degraded without tuning. I would sincerely hope no one would out right lie in their publications, e.g., this one: </div>
<div><a href="http://portal.acm.org/citation.cfm?id=1654061" target="_blank"></a><a href="http://portal.acm.org/citation.cfm?id=1654061" target="_blank">http://portal.acm.org/citation.cfm?id=1654061</a></div><div class="im">
<div> </div><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
BTW: are you solving with many right hand sides? Maybe before you muck with WSMP we should figure out how to get you access to the multiple right hand side support of MUMPS (I don't know if SuperLU_Dist has it) so you can speed up your current computations a good amount? Currently PETSc's MatMatSolve() calls a separate solve for each right hand side with MUMPS.<br>
<font color="#888888"><br>
Barry<br>
<br></font></blockquote><div><br></div></div><div>I will eventually need to solve against many right-hand sides, but for now I am solving against one and it is still taking too long; in fact, not only does it take too long, but memory per core increased for fixed problem sizes as I increase the number of MPI processes (for both SuperLU_Dist and MUMPS). This was occurring for quasi2d Helmholtz problems over a couple hundred cores. My only logical explanation for this behavior is that the communication buffers grow proportional to the number of processes on each process, but I stress that this is just a guess. I tried reading through the MUMPS code and quickly gave up.</div>
<div><br></div><div>Another problem with MUMPS is that requires the entire set of right-hand sides to reside on the root process...that will clearly not work for a billion degrees of freedom with several hundred RHSs. WSMP gets this part right and actually distributes those vectors.</div>
<div><br></div><div>Jack</div></div>
</div></blockquote></div></blockquote></div><br>