<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">The default SuperLU_DIST setting is to serial symbolic factorization. Therefore, what matters is how much memory do you have per MPI task? <br><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">The code failed to malloc memory during redistribution of matrix A to {L\U} data struction (using result of serial symbolic factorization.)<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">You can use parallel symbolic factorization, by runtime option: '-mat_superlu_dist_parsymbfact'<br><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Sherry Li<br><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jul 20, 2015 at 8:59 AM, <a href="mailto:Mahir.Ulker-Kaustell@tyrens.se">Mahir.Ulker-Kaustell@tyrens.se</a> <span dir="ltr"><<a href="mailto:Mahir.Ulker-Kaustell@tyrens.se" target="_blank">Mahir.Ulker-Kaustell@tyrens.se</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div link="blue" vlink="purple" lang="SV">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Hong:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US">Previous experiences with this equation have shown that it is very difficult to solve it iteratively. Hence the use of a direct solver.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US">The large test problem I am trying to solve has slightly less than 10^6 degrees of freedom. The matrices are derived from finite elements so they
are sparse.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US">The machine I am working on has 128GB ram. I have estimated the memory needed to less than 20GB, so if the solver needs twice or even three times
as much, it should still work well. Or have I completely misunderstood something here?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US">Mahir<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" lang="EN-US"><u></u> <u></u></span></b></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" lang="EN-US">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" lang="EN-US"> Hong [mailto:<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>]
<br>
<b>Sent:</b> den 20 juli 2015 17:39<span class=""><br>
<b>To:</b> Ülker-Kaustell, Mahir<br>
<b>Cc:</b> petsc-users<br>
<b>Subject:</b> Re: [petsc-users] SuperLU MPI-problem<u></u><u></u></span></span></p><span class="HOEnZb"><font color="#888888">
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
</font></span><div><span class="HOEnZb"><font color="#888888">
</font></span><div><span class="HOEnZb"><font color="#888888">
<div>
<p class="MsoNormal">Mahir:<u></u><u></u></p>
</div></font></span><div><div class="h5">
<div>
<p class="MsoNormal">Direct solvers consume large amount of memory. Suggest to try followings:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">1. A sparse iterative solver if <span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> [-omega^2M + K] is not too ill-conditioned. You may test it using the small matrix.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">2. Incrementally increase your matrix sizes. Try different matrix orderings.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Do you get memory crash in the 1st symbolic factorization? </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">In your case, matrix data structure stays same when omega changes, so you only need to do one matrix symbolic factorization and reuse it.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">3. Use a machine that gives larger memory.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Hong</span><u></u><u></u></p>
</div>
<div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">Dear Petsc-Users,<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">I am trying to use PETSc to solve a set of linear equations arising from Naviers equation (elastodynamics) in the frequency domain.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">The frequency dependency of the problem requires that the system</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> [-omega^2M + K]u = F</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">where M and K are constant, square, positive definite matrices (mass and stiffness respectively) is solved for each frequency omega of interest.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">K is a complex matrix, including material damping.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">I have written a PETSc program which solves this problem for a small (1000 degrees of freedom) test problem on one or several processors, but it keeps crashing
when I try it on my full scale (in the order of 10^6 degrees of freedom) problem.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">The program crashes at KSPSetUp() and from what I can see in the error messages, it appears as if it consumes too much memory.
</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">I would guess that similar problems have occurred in this mail-list, so I am hoping that someone can push me in the right direction…</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:#888888" lang="EN-US"> </span><span style="color:#888888"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#888888" lang="EN-US">Mahir</span><span style="color:#888888"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#888888" lang="EN-US"> </span><span style="color:#888888"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#888888" lang="EN-US"> </span><span style="color:#888888"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#888888" lang="EN-US"> </span><span style="color:#888888"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:#888888" lang="EN-US"> </span><span style="color:#888888"><u></u><u></u></span></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div></div></div>
</div>
</div>
</div>
</blockquote></div><br></div>