Hello,<br><br><div class="gmail_quote">On Thu, Aug 18, 2011 at 5:11 PM, Xiaoye S. Li <span dir="ltr"><<a href="mailto:xsli@lbl.gov">xsli@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
I can clarify a couple of questions re. SuperLU_DIST.<br><br>1) SuperLU does support multiple right-hand sides. That is, the B matrix on the right can be a dense matrix of size n-by-nrhs. Also, the B matrix is distributed among all the processors, each processor takes one block row of B. There is no need to have the entire B on every processor.<br>
<br></blockquote><div><br></div><div>I certainly agree that this is an improvement over MUMPS.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">2) We are preparing to upgrade to a newer version. The parallel factorization is improved with a better scheduling algorithm. This is particularly effective using larger core count, say in 100s.<br>
<br></blockquote><div><br></div><div>I will be looking forward to the new version. I took a several month hiatus from trying to use sparse-direct solvers for my subproblems since I wasn't seeing substantial speedups over the serial algorithm; it was taking a few hundred cores to solve 3d Helmholtz over 256^3 domains in an hour.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">3) Regarding memory usage, the factorization algorithm used in SuperLU mostly performs "update in-place", and requires just a little bit extra working storage. So, if the load balance is not too bad, the memory per core should go down consistently. It will be helpful if you can give some concreate numbers showing how much memory usage increases with increasing core count.<br>
<br></blockquote><div><br></div><div>As for specifics, factorizations of 256 x 256 x 10 grids with 7-point finite-difference stencils required more memory per process when I increased past ~200 processes. I was storing roughly 50 of these factorizations before running out of memory, no matter how many more processes I threw at SuperLU_Dist and MUMPS. I would think that the load balance would be decent since it is such a regular grid.</div>
<div><br></div><div>Also, if I'm recalling correctly, on 256 cores I was only seeing ~5 GFlops in each triangle solve against a subdomain, which isn't much of an improvement over my laptop. Does this sound pathological to you or is it to be expected from SuperLU_Dist and MUMPS? The linked WSMP paper showed some impressive triangle solve scalability. Will the new scheduling algorithm effect the solves as well?</div>
<div><br></div><div>Best Regards,</div><div>Jack</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><br><br><div class="gmail_quote">On Tue, Aug 16, 2011 at 8:34 PM, Rebecca Yuan <span dir="ltr"><<a href="mailto:rebeccayxf@gmail.com" target="_blank">rebeccayxf@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">
<div bgcolor="#FFFFFF"><div><br><br><br>Begin forwarded message:<br><br></div><blockquote type="cite"><div><b>From:</b> Jack Poulson<a href="mailto:jack.poulson@gmail.com" target="_blank"></a>><br><b>Date:</b> August 16, 2011 10:18:16 PM CDT<div>
<br><b>To:</b> For users of the development version of PETSc <<a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a>><br><b>Subject:</b> <b>Re: [petsc-dev] Wrapper for WSMP</b><br><b>Reply-To:</b> For users of the development version of PETSc <<a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a>><br>
<br></div></div></blockquote><div><div></div><div class="h5"><div></div><blockquote type="cite"><div><div>On Tue, Aug 16, 2011 at 9:35 PM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank"></a><a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br>
</div><div class="gmail_quote"><div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">
<div><div></div><div><br>
On Aug 16, 2011, at 5:14 PM, Jack Poulson wrote:<br>
<br>
> Hello all,<br>
><br>
> I am working on a project that requires very fast sparse direct solves and MUMPS and SuperLU_Dist haven't been cutting it. From what I've read, when properly tuned, WSMP is significantly faster, particularly with multiple right-hand sides on large machines. The obvious drawback is that it's not open source, but the binaries seem to be readily available for most platforms.<br>
><br>
> Before I reinvent the wheel, I would like to check if anyone has already done some work on adding it into PETSc. If not, its interface is quite similar to MUMPS and I should be able to mirror most of that code. On the other hand, there are a large number of platform-specific details that need to be handled, so keeping things both portable and fast might be a challenge. It seems that the CSC storage format should be used since it is required for Hermitian matrices.<br>
><br>
> Thanks,<br>
> Jack<br>
<br>
</div></div> Jack,<br>
<br>
By all means do it. That would be a nice thing to have. But be aware that the WSMP folks have a reputation for exaggerating how much better their software is so don't be surprised if after all that work it is not much better.<br>
<br></blockquote><div><br></div></div><div>Good to know. I was somewhat worried about that, but perhaps it is a matter of getting all of the tuning parameters right. The manual does mention that performance is significantly degraded without tuning. I would sincerely hope no one would out right lie in their publications, e.g., this one: </div>
<div><a href="http://portal.acm.org/citation.cfm?id=1654061" target="_blank"></a><a href="http://portal.acm.org/citation.cfm?id=1654061" target="_blank">http://portal.acm.org/citation.cfm?id=1654061</a></div><div>
<div> </div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">
BTW: are you solving with many right hand sides? Maybe before you muck with WSMP we should figure out how to get you access to the multiple right hand side support of MUMPS (I don't know if SuperLU_Dist has it) so you can speed up your current computations a good amount? Currently PETSc's MatMatSolve() calls a separate solve for each right hand side with MUMPS.<br>
<font color="#888888"><br>
Barry<br>
<br></font></blockquote><div><br></div></div><div>I will eventually need to solve against many right-hand sides, but for now I am solving against one and it is still taking too long; in fact, not only does it take too long, but memory per core increased for fixed problem sizes as I increase the number of MPI processes (for both SuperLU_Dist and MUMPS). This was occurring for quasi2d Helmholtz problems over a couple hundred cores. My only logical explanation for this behavior is that the communication buffers grow proportional to the number of processes on each process, but I stress that this is just a guess. I tried reading through the MUMPS code and quickly gave up.</div>
<div><br></div><div>Another problem with MUMPS is that requires the entire set of right-hand sides to reside on the root process...that will clearly not work for a billion degrees of freedom with several hundred RHSs. WSMP gets this part right and actually distributes those vectors.</div>
<div><br></div><div>Jack</div></div>
</div></blockquote></div></div></div></blockquote></div><br>
</blockquote></div><br>