<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi Sherry,<div><br></div><div>Thanks for the offer to help!</div><div><br></div><div>I tried superlu_dist again and it crashes even more quickly than MUMPS with just the following error:</div><div><br></div><div>ERROR: 0031-250 task 128: Killed</div><div><br></div><div>Absolutely nothing else is written out to either stderr or stdout. This is with -mat_superlu_dist_statprint. </div><div>The program works fine on a smaller matrix.</div><div><br></div><div>This is the sequence of calls:</div><div><br></div><div>KSPSetType(ksp,KSPPREONLY);</div><div><div>PCSetType(pc,PCLU);</div><div>PCFactorSetMatSolverPackage(pc,MATSOLVERSUPERLU_DIST);</div><div>KSPSetFromOptions(ksp);</div><div>PCSetFromOptions(pc);</div><div><div>KSPSolve(ksp,b,x);</div></div></div><div><br></div><div>All of these successfully return *except* the very last one to KSPSolve.</div><div><br></div><div>Any help would be appreciated. Thanks!</div><div><br></div><div>Samar</div><div><br></div><div><div><div>On Feb 24, 2014, at 3:58 PM, Xiaoye S. Li <<a href="mailto:xsli@lbl.gov">xsli@lbl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Samar:<br>If you include the error message while crashing using superlu_dist, I probably know the reason. (better yet, include the printout before the crash. )<br>
<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Sherry<br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Feb 24, 2014 at 9:56 AM, Hong Zhang <span dir="ltr"><<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Samar :<div>There are limitations for direct solvers. </div><div>Do not expect any solver can be used on arbitrarily large problems.</div>
<div>Since superlu_dist also crashes, direct solvers may not be able to work on your application. </div>
<div>This is why I suggest to increase size incrementally. </div><div>You may have to experiment other type of solvers.</div><div><br></div><div>Hong</div><div><br><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">
<div>Hi Hong and Jed,</div>
<div><br>
</div>
<div>Many thanks for replying. It would indeed be nice if the error messages from MUMPS were less cryptic!</div>
<div><br>
</div>
<div>1) I have tried smaller matrices although given how my problem is set up a jump is difficult to avoid. But a good idea </div>
<div>that I will try.</div>
<div><br>
</div>
<div>2) I did try various ordering but not the one you suggested.</div>
<div><br>
</div>
<div>3) Tracing the error through the MUMPS code suggest a rather abrupt termination of the program (there should be more </div>
<div>error messages if, for example, memory was a problem). I therefore thought it might be an interface problem rather than </div>
<div>one with mumps and turned to the petsc-users group first. </div>
<div><br>
</div>
<div>4) I've tried superlu_dist but it also crashes (also unclear as to why) at which point I decided to try mumps. The fact that both </div>
<div>crash would again indicate a common (memory?) problem.</div>
<div><br>
</div>
<div>I'll try a few more things before asking the MUMPS developers.</div>
<div><br>
</div>
<div>Thanks again for your help!</div><span><font color="#888888">
<div><br>
</div>
<div>Samar</div></font></span><div>
<div><br>
</div>
<div>
<div>On Feb 24, 2014, at 11:47 AM, Hong Zhang <<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>> wrote:</div>
<br>
<blockquote type="cite">
<div dir="ltr">Samar:
<div>The crash occurs in
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
...<br>
[161]PETSC ERROR: Error in external library!<br>
[161]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFO(1)=-1, INFO(2)=48</blockquote>
<div> </div>
<div>for very large matrix, likely memory problem as you suspected. </div>
<div>I would suggest </div>
<div>1. run problems with increased sizes (not jump from a small one to a very large one) and observe memory usage using</div>
<div>'-ksp_view'.</div>
<div> I see you use '-mat_mumps_icntl_14 1000', i.e., percentage of estimated workspace increase. Is it too large?</div>
<div> Anyway, this input should not cause the crash, I guess.</div>
<div>2. experimenting with different matrix ordering <span style="font-family:arial,sans-serif;font-size:13px">-mat_mumps_icntl_7 <> (I usually use sequential ordering 2) </span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px"> I see you use parallel ordering -</span>mat_mumps_icntl_29 2.</div>
<div>3. send bug report to mumps developers for their suggestion.<br>
</div>
<div><br>
</div>
<div>4. try other direct solvers, e.g., superlu_dist.</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
…<br>
<br>
etc etc. The above error I can tell has something to do with processor 48 (INFO(2)) and so forth but not the previous one.<br>
<br>
The full output enabled with -mat_mumps_icntl_4 3 looks as in the attached file. Any hints as to what could be giving this<br>
error would be very much appreciated.<br>
</blockquote>
<div> </div>
<div>I do not know how to interpret this output file. mumps developer would give you better suggestion on it.</div>
<div>I would appreciate to learn as well :-)</div>
<div><br>
</div>
<div>Hong</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div>
</blockquote></div><br></div></body></html>