<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Mon, Jul 20, 2015 at 11:14 AM, <a href="mailto:Mahir.Ulker-Kaustell@tyrens.se">Mahir.Ulker-Kaustell@tyrens.se</a> <span dir="ltr"><<a href="mailto:Mahir.Ulker-Kaustell@tyrens.se" target="_blank">Mahir.Ulker-Kaustell@tyrens.se</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="SV" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I am roughly guessing that my sparse matrix will have 1000 non-zeros on each row. As the matrix I symmetric I divide by half:<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">1000*1000000(rows)*8(bytes)*2(complex numbers)/2 = 8GB</span></p></div></div></blockquote><div><br></div><div>1) You have missed the memory to represent the columns, so multiply by 1.5</div><div><br></div><div>2) Are you using a symmetric format?</div><div><br></div><div>3) This is only the space to represent the matrix, not its factors.</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="SV" link="blue" vlink="purple"><div><p class="MsoNormal"> </p></div></div></blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="SV" link="blue" vlink="purple"><div><p class="MsoNormal"></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">However, it turns out to be much less: I create the matrices in another program and the binary file holding my matrix K is only 800MB. M is a lumped
mass matrix…<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">By the way, I tried using MUMPS and have the same problem there.
<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Mahir</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Matthew Knepley [mailto:<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>]
<br>
<b>Sent:</b> den 20 juli 2015 18:05<br>
<b>To:</b> Ülker-Kaustell, Mahir<br>
<b>Cc:</b> Hong; petsc-users<span class=""><br>
<b>Subject:</b> Re: [petsc-users] SuperLU MPI-problem<u></u><u></u></span></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<div>
<div>
<div><span class="">
<p class="MsoNormal">On Mon, Jul 20, 2015 at 10:59 AM, <a href="mailto:Mahir.Ulker-Kaustell@tyrens.se" target="_blank">
Mahir.Ulker-Kaustell@tyrens.se</a> <<a href="mailto:Mahir.Ulker-Kaustell@tyrens.se" target="_blank">Mahir.Ulker-Kaustell@tyrens.se</a>> wrote:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Hong:</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Previous experiences with this equation have shown that it is very difficult to solve
it iteratively. Hence the use of a direct solver.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">The large test problem I am trying to solve has slightly less than 10^6 degrees of freedom.
The matrices are derived from finite elements so they are sparse.</span><u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</span><div>
<p class="MsoNormal">Estimated how? It is very difficult to estimate flll-in.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> Matt<u></u><u></u></p>
</div><div><div class="h5">
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">The machine I am working on has 128GB ram. I have estimated the memory needed to less
than 20GB, so if the solver needs twice or even three times as much, it should still work well. Or have I completely misunderstood something here?</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Mahir</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> </span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Hong
[mailto:<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>]
<br>
<b>Sent:</b> den 20 juli 2015 17:39<br>
<b>To:</b> Ülker-Kaustell, Mahir<br>
<b>Cc:</b> petsc-users<br>
<b>Subject:</b> Re: [petsc-users] SuperLU MPI-problem</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal">Mahir:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Direct solvers consume large amount of memory. Suggest to try followings:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">1. A sparse iterative solver if <span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> [-omega^2M + K] is not too ill-conditioned. You may test it using the small matrix.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">2. Incrementally increase your matrix sizes. Try different matrix orderings.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Do you get memory crash in the 1st symbolic factorization? </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">In your case, matrix data structure stays same when omega changes, so you only need to do one matrix symbolic factorization
and reuse it.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">3. Use a machine that gives larger memory.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Hong</span><u></u><u></u></p>
</div>
<div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">Dear Petsc-Users,<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">I am trying to use PETSc to solve a set of linear equations arising from Naviers equation (elastodynamics) in the frequency domain.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">The frequency dependency of the problem requires that the system</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> [-omega^2M + K]u = F</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">where M and K are constant, square, positive definite matrices (mass and stiffness respectively) is solved for each frequency omega of interest.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">K is a complex matrix, including material damping.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">I have written a PETSc program which solves this problem for a small (1000 degrees of freedom) test problem on one or several processors, but it keeps crashing
when I try it on my full scale (in the order of 10^6 degrees of freedom) problem.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">The program crashes at KSPSetUp() and from what I can see in the error messages, it appears as if it consumes too much memory.
</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">I would guess that similar problems have occurred in this mail-list, so I am hoping that someone can push me in the right direction…</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#888888"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#888888">Mahir</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#888888"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#888888"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#888888"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#888888"> </span><u></u><u></u></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</div>
</div>
</blockquote>
</div></div></div>
<p class="MsoNormal"><br>
<br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<p class="MsoNormal">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</div></div>