Jinquan:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<div>
<p class="MsoNormal"><br></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I have a question on how to use mumps properly in PETSc. It appears that I didn’t set up mumps right. I followed the example in
<u></u><u></u></span></p>
<p class="MsoNormal" style="text-indent:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><a href="http://www.mcs.anl.gov/petsc/petsc-dev/src/mat/examples/tests/ex125.c.html" target="_blank">http://www.mcs.anl.gov/petsc/petsc-dev/src/mat/examples/tests/ex125.c.html</a></span></p>
</div></div></div></blockquote><div> </div><div>This example is for our internal testing, not intended for production runs.</div><div>Suggest using PETSc high level KSP solver which provides more flexibility.</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple"><div><div><p class="MsoNormal" style="text-indent:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">to set up my program. Here is my situation on using the default setting
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<pre> <a href="http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Sys/PetscInt.html#PetscInt" target="_blank">PetscInt</a> icntl_7 = 5;<u></u><u></u></pre>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Courier New"">
<a href="http://www.mcs.anl.gov/petsc/petsc-dev/docs/manualpages/Mat/MatMumpsSetIcntl.html#MatMumpsSetIcntl" target="_blank">
MatMumpsSetIcntl</a>(F,7,icntl_7);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">in the example ex125.c:</span></p></div></div></div></blockquote><div>Using KSP sover, the mumps options can be chosen at runtime,</div>
<div>e.g. </div><div>~petsc/src/ksp/ksp/examples/tutorials/ex2.c:</div><div><div>mpiexec -n 2 ./ex2 -pc_type lu -pc_factor_mat_solver_package mumps -mat_mumps_icntl_7 5</div><div>Norm of error 1.49777e-15 iterations 1</div>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div><div>
<p><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><span>1.<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">The program work fine on all
<b>small</b> models (sparse matrices at the order of m= 894, 1097, 31k with a dense matrix included in the sparse matrix). The residuals are at the magnitude of 10^-3.</span></p></div></div></div></blockquote><div> ^^^^</div>
<div>With a direct solver, residual = <span style="color:rgb(31,73,125);font-family:Calibri,sans-serif;font-size:15px">10^-3 indicates your matrix might be very ill-conditioned or close to singular. What do you get for |R|/|rhs| = ?</span></div>
<div><span style="color:rgb(31,73,125);font-family:Calibri,sans-serif;font-size:15px">Is this the reason you want to use a direct solver instead of iterative one?</span></div><div><font color="#1f497d" face="Calibri, sans-serif"><span style="font-size:15px">What do you mean "</span></font><span style="color:rgb(31,73,125);font-family:Calibri,sans-serif;font-size:15px">31k with a dense matrix included in the sparse matrix"?</span></div>
<div><span style="color:rgb(31,73,125);font-family:Calibri,sans-serif;font-size:15px">How sparse is your matrix, e.g., nnz(A)/(m*m)=?</span></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple"><div><div>
<p><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><span>2.<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">The program has some issues on
<b>medium</b> size problem (m=460k with a dense matrix at the order of n=30k included in the sparse matrix). The full sparse matrix is sized at 17GB.<u></u><u></u></span></p>
<p style="margin-left:1.0in">
<u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><span>a.<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">We used another software to generate sparse matrix by using 144 cores:
<u></u><u></u></span></p>
<p style="margin-left:1.5in">
<u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><span><span style="font:7.0pt "Times New Roman"">
</span>i.<span style="font:7.0pt "Times New Roman""> </span></span></span><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">When I used the resource from 144 cores (12 nodes with 48GB/node), it could not provide
the solution. There was a complain on the memory violation.<u></u><u></u></span></p>
<p style="margin-left:1.5in">
<u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><span><span style="font:7.0pt "Times New Roman"">
</span>ii.<span style="font:7.0pt "Times New Roman""> </span></span></span><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">When I used the resource from 432 cores (36 nodes with 48GB/node), it provided the solution. </span></p>
</div></div></div></blockquote><div>Direct solvers are notoriously memory consuming. It seems your matrix is quite dense, requiring more memory than 144 cores could provide. </div><div>What is "<span style="color:rgb(31,73,125);font-family:Calibri,sans-serif;font-size:15px">another software</span><span style="color:rgb(31,73,125);font-family:Calibri,sans-serif;font-size:15px"> "?</span></div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div><div><p style="margin-left:1.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">
<u></u><u></u></span></p>
<p style="margin-left:1.0in">
<u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><span>b.<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">We used another software to generate the same sparse matrix by using 576 cores:
<u></u><u></u></span></p>
<p style="margin-left:1.5in">
<u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><span><span style="font:7.0pt "Times New Roman"">
</span>i.<span style="font:7.0pt "Times New Roman""> </span></span></span><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">When I used the resource from 576 cores (48 nodes with 48GB/node), it could not provide
the solution. There was a complain on the memory violation.<u></u><u></u></span></p>
<p style="margin-left:1.5in">
<u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><span><span style="font:7.0pt "Times New Roman"">
</span>ii.<span style="font:7.0pt "Times New Roman""> </span></span></span><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">When I used the resource from 1152 cores (96 nodes with 48GB/node), it provided the solution. </span></p>
</div></div></div></blockquote><div>Both a and b seem indicate that, you can use small num of cores to generate original matrix A, but need more cores (resource) to solve A x =b.</div><div>This is because A = LU, the factored matrices L and U require far more memory than original A. Run your code using KSP with your matrix data and option -ksp_view</div>
<div>e.g., petsc/src/ksp/ksp/examples/tutorials/ex10.c</div><div>mpiexec -n 2 ./ex10 -f <matrix binary data file> -pc_type lu -pc_factor_mat_solver_package mumps -ksp_view</div><div>...</div><div>then you'll see memory info provided by mumps.<span style="color:rgb(31,73,125);font-family:Calibri,sans-serif;font-size:11pt"> </span></div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div><div>
<p><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><span>3.<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">The program could not solve the
<b>large</b> size problem (m=640k with a dense matrix at the order of n=178k included in the sparse matrix). The full sparse matrix is sized at 511GB.<u></u><u></u></span></p>
<p style="margin-left:1.0in">
<u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><span>a.<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">We used another software to generate sparse matrix by using 900 cores:
<u></u><u></u></span></p>
<p style="margin-left:1.5in">
<u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><span><span style="font:7.0pt "Times New Roman"">
</span>i.<span style="font:7.0pt "Times New Roman""> </span></span></span><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">When I used the resource from 900 cores (75 nodes with 48GB/node), it could not provide
the solution. There was a complain on the memory violation.<u></u><u></u></span></p>
<p style="margin-left:1.5in">
<u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><span><span style="font:7.0pt "Times New Roman"">
</span>ii.<span style="font:7.0pt "Times New Roman""> </span></span></span><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">When I used the resource from 2400 cores (200 nodes with 48GB/node), it STILL COULD NOT
provide the solution. </span></p></div></div></div></blockquote><div>Your computer system and software have limits. Find the answers to your 'medium size' problems first. </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple"><div><div><p style="margin-left:1.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> <u></u><u></u></span></p>
<p class="MsoNormal"><br></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">My confusion starts from the medium size problem:<u></u><u></u></span></p>
<p style="margin-left:41.45pt">
<u></u><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">It seems something was not right in the default setting in ex125.c for these problems.
<u></u><u></u></span></p>
<p style="margin-left:41.45pt">
<u></u><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I got the info that METIS was used instead of ParMETIS in solving these problems. </span></p>
</div></div></div></blockquote><div>By default, petsc-mumps interface uses sequential symbolic factorization (analysis phase). Use '-mat_mumps_icntl_28 2' to switch to parallel.</div><div>I tested it, but seems parmetis is still not used. Check mumps manual </div>
<div>or contact mumps developer on how to use parmetis.</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div><div><p style="margin-left:41.45pt">
<span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">
<u></u><u></u></span></p>
<p style="margin-left:41.45pt">
<u></u><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Furthermore, it appears that there was unreasonable demand on the solver even on the medium size problem. <u></u><u></u></span></p>
<p style="margin-left:41.45pt">
<u></u><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d"><span>·<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I suspect one rank was trying to collect all data from other ranks. </span></p></div></div></div>
</blockquote><div>Yes, analysis is sequential, and rhs vector must be in the host :-( </div><div>In general, direct solvers cannot be scaled to very large num of cores.</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple"><div><div><p style="margin-left:41.45pt"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">What other addition setting is needed for mumps such that it could deal with medium and large size problems?</span></p>
</div></div></div></blockquote><div> </div><div>Run your code with option '-help |grep mumps' and experiment with</div><div>various options, e.g., matrix orderings, nonzero-fills etc. </div><div>You may also try superlu_dist. Good luck!</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div><div><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Do you guys have similar experience on that?</span></p></div></div></div></blockquote><div>I personally never used mumps or superlu_dist for such large matrices.</div>
<div>Consult mumps developers.</div><div><br></div><div>Hong</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div><div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Thanks,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Jinquan<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
</div>
</div>
</div>
</blockquote></div><br>