<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Wed, Feb 18, 2015 at 10:31 AM, Sun, Hui <span dir="ltr"><<a href="mailto:hus003@ucsd.edu" target="_blank">hus003@ucsd.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div style="direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt">So far I just try around, I haven't looked into literature yet.
<div><br>
</div>
<div>However, both MATLAB's ilu+gmres and ilu+bcgs work. Is it possible that some parameter or options need to be tuned in using PETSc's ilu or hypre's ilu? Besides, is there a way to view how good the performance of the pc is and output the matrices L and
U, so that I can do some test in MATLAB? </div></div></div></blockquote><div><br></div><div>1) Its not clear exactly what Matlab is doing</div><div><br></div><div>2) PETSc uses ILU(0) by default (you can set it to use ILU(k))</div><div><br></div><div>3) I don't know what Hypre's ILU can do</div><div><br></div><div>I would really discourage from using ILU. I cannot imagine it is faster than sparse direct factorization</div><div>for your system, such as from SuperLU or MUMPS.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt"><div>
<div>Hui</div>
<div><br>
</div>
<div><br>
<div style="font-family:Times New Roman;color:#000000;font-size:16px">
<hr>
<div style="direction:ltr"><font face="Tahoma" color="#000000"><b>From:</b> Matthew Knepley [<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>]<br>
<b>Sent:</b> Wednesday, February 18, 2015 8:09 AM<br>
<b>To:</b> Sun, Hui<br>
<b>Cc:</b> <a href="mailto:hong@aspiritech.org" target="_blank">hong@aspiritech.org</a>; <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a><br>
<b>Subject:</b> Re: [petsc-users] Question concerning ilu and bcgs<br>
</font><br>
</div>
<div></div>
<div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Wed, Feb 18, 2015 at 10:02 AM, Sun, Hui <span dir="ltr">
<<a href="mailto:hus003@ucsd.edu" target="_blank">hus003@ucsd.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div style="direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt">Yes I've tried other solvers, gmres/ilu does not work, neither does bcgs/ilu. Here are the options:
<div>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">-pc_type ilu -pc_factor_nonzeros_along_diagonal -pc_factor_levels 0 -pc_factor_reuse_ordering -ksp_ty\</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">pe bcgs -ksp_rtol 1e-6 -ksp_max_it 10 -ksp_monitor_short -ksp_view</p>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Note here that ILU(0) is an unreliable and generally crappy preconditioner. Have you looked in the</div>
<div>literature for the kinds of preconditioners that are effective for your problem?</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div style="direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt">
<div>
<p style="margin-right:0px;margin-left:0px"><font>Here is the output:</font></p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 0 KSP Residual norm 211292 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 1 KSP Residual norm 13990.2 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 2 KSP Residual norm 9870.08 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 3 KSP Residual norm 9173.9 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 4 KSP Residual norm 9121.94 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 5 KSP Residual norm 7386.1 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 6 KSP Residual norm 6222.55 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 7 KSP Residual norm 7192.94 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 8 KSP Residual norm 33964 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 9 KSP Residual norm 33960.4 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 10 KSP Residual norm 1068.54 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">KSP Object: 1 MPI processes</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> type: bcgs</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> maximum iterations=10, initial guess is zero</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> tolerances: relative=1e-06, absolute=1e-50, divergence=10000</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> left preconditioning</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> using PRECONDITIONED norm type for convergence test</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">PC Object: 1 MPI processes</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> type: ilu</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> ILU: out-of-place factorization</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> ILU: Reusing reordering from past factorization</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 0 levels of fill</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> tolerance for zero pivot 2.22045e-14</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> using diagonal shift on blocks to prevent zero pivot [INBLOCKS]</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> matrix ordering: natural</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> factor fill ratio given 1, needed 1</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> Factored matrix follows:</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> Mat Object: 1 MPI processes</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> type: seqaij</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> rows=62500, cols=62500</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> package used to perform factorization: petsc</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> total: nonzeros=473355, allocated nonzeros=473355</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> total number of mallocs used during MatSetValues calls =0</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> not using I-node routines</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> linear system matrix = precond matrix:</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> Mat Object: 1 MPI processes</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> type: seqaij</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> rows=62500, cols=62500</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> total: nonzeros=473355, allocated nonzeros=7.8125e+06</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> total number of mallocs used during MatSetValues calls =0</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> not using I-node routines</p>
<p style="margin-right:0px;margin-left:0px"></p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">Time cost: 0.307149, 0.268402, 0.0990018</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"><br>
</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"><br>
</p>
</div>
<div><br>
<div style="font-family:Times New Roman;color:#000000;font-size:16px">
<hr>
<div style="direction:ltr"><font face="Tahoma" color="#000000"><b>From:</b> <a href="mailto:hong@aspiritech.org" target="_blank">
hong@aspiritech.org</a> [<a href="mailto:hong@aspiritech.org" target="_blank">hong@aspiritech.org</a>]<br>
<b>Sent:</b> Wednesday, February 18, 2015 7:49 AM<br>
<b>To:</b> Sun, Hui<br>
<b>Cc:</b> Matthew Knepley; <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">
petsc-users@mcs.anl.gov</a><br>
<b>Subject:</b> Re: [petsc-users] Question concerning ilu and bcgs<br>
</font><br>
</div>
<div></div>
<div>
<div dir="ltr"> Have you tried other solvers, e.g., PETSc default gmres/ilu, bcgs/ilu etc.
<div>The matrix is small. If it is ill-conditioned, then pc_type lu would work the best.</div>
<div><br>
</div>
<div>Hong<br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, Feb 18, 2015 at 9:34 AM, Sun, Hui <span dir="ltr">
<<a href="mailto:hus003@ucsd.edu" target="_blank">hus003@ucsd.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div style="direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt">With options:
<div><span style="font-family:Menlo;font-size:11px"><br>
</span></div>
<div><span style="font-family:Menlo;font-size:11px">-pc_type hypre -pc_hypre_type pilut -pc_hypre_pilut_maxiter 1000 -pc_hypre_pilut_tol 1e-3 -ksp_type b</span><span style="font-family:Menlo;font-size:11px">cgs -ksp_rtol 1e-10 -ksp_max_it 10 -ksp_monitor_short
-ksp_converged_reason -ksp_view</span>
<div><br>
</div>
<div>Here is the full output:</div>
<div><br>
</div>
<div><span>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 0 KSP Residual norm 1404.62 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 1 KSP Residual norm 88.9068 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 2 KSP Residual norm 64.73 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 3 KSP Residual norm 71.0224 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 4 KSP Residual norm 69.5044 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 5 KSP Residual norm 455.458 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 6 KSP Residual norm 174.876 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 7 KSP Residual norm 183.031 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 8 KSP Residual norm 650.675 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 9 KSP Residual norm 79.2441 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 10 KSP Residual norm 84.1985 </p>
</span>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">Linear solve did not converge due to DIVERGED_ITS iterations 10</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">KSP Object: 1 MPI processes</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> type: bcgs</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> maximum iterations=10, initial guess is zero</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> tolerances: relative=1e-10, absolute=1e-50, divergence=10000</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> left preconditioning</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> using PRECONDITIONED norm type for convergence test</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">PC Object: 1 MPI processes</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> type: hypre</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> HYPRE Pilut preconditioning</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> HYPRE Pilut: maximum number of iterations 1000</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> HYPRE Pilut: drop tolerance 0.001</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> HYPRE Pilut: default factor row size </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> linear system matrix = precond matrix:</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> Mat Object: 1 MPI processes</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> type: seqaij</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> rows=62500, cols=62500</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> total: nonzeros=473355, allocated nonzeros=7.8125e+06</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> total number of mallocs used during MatSetValues calls =0</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> not using I-node routines</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">Time cost: 0.756198, 0.662984, 0.105672 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"><br>
</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"><br>
</p>
</div>
<div><br>
<div style="font-family:Times New Roman;color:#000000;font-size:16px">
<hr>
<div style="direction:ltr"><font face="Tahoma" color="#000000"><b>From:</b> Matthew Knepley [<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>]<br>
<b>Sent:</b> Wednesday, February 18, 2015 3:30 AM<br>
<b>To:</b> Sun, Hui<br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a><br>
<b>Subject:</b> Re: [petsc-users] Question concerning ilu and bcgs<br>
</font><br>
</div>
<div>
<div>
<div></div>
<div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Wed, Feb 18, 2015 at 12:33 AM, Sun, Hui <span dir="ltr">
<<a href="mailto:hus003@ucsd.edu" target="_blank">hus003@ucsd.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div style="direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt">I have a matrix system Ax = b, A is of type MatSeqAIJ or MatMPIAIJ, depending on the number of cores.
<div><br>
</div>
<div>I try to solve this problem by pc_type ilu and ksp_type bcgs, it does not converge. The options I specify are:</div>
<div>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">-pc_type hypre -pc_hypre_type pilut -pc_hypre_pilut_maxiter 1000 -pc_hypre_pilut_tol 1e-3 -ksp_type b\</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo">cgs -ksp_rtol 1e-10 -ksp_max_it 1000 -ksp_monitor_short -ksp_converged_reason</p>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>1) Run with -ksp_view, so we can see exactly what was used</div>
<div><br>
</div>
<div>2) ILUT is unfortunately not a well-defined algorithm, and I believe the parallel version makes different decisions</div>
<div> than the serial version.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div style="direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt">
<div>
<p style="margin-right:0px;margin-left:0px"><font>The first a few lines of the output are:</font></p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 0 KSP Residual norm 1404.62 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 1 KSP Residual norm 88.9068 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 2 KSP Residual norm 64.73 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 3 KSP Residual norm 71.0224 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 4 KSP Residual norm 69.5044 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 5 KSP Residual norm 455.458 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 6 KSP Residual norm 174.876 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 7 KSP Residual norm 183.031 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 8 KSP Residual norm 650.675 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 9 KSP Residual norm 79.2441 </p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"> 10 KSP Residual norm 84.1985</p>
<p style="margin-right:0px;margin-left:0px;font-size:11px;font-family:Menlo"><br>
</p>
<p style="margin-right:0px;margin-left:0px"><font>This clearly indicates non-convergence. However, I output the sparse matrix A and vector b to MATLAB, and run the following command:</font></p>
<p style="margin-right:0px;margin-left:0px">[L,U] = ilu(A,struct('type','ilutp','droptol',1e-3));</p>
<p style="margin-right:0px;margin-left:0px">[ux1,fl1,rr1,it1,rv1] = bicgstab(A,b,1e-10,1000,L,U);</p>
<p style="margin-right:0px;margin-left:0px"><br>
</p>
<p style="margin-right:0px;margin-left:0px">And it converges in MATLAB, with flag fl1=0, relative residue rr1=8.2725e-11, and iteration it1=89.5. I'm wondering how can I figure out what's wrong. </p>
<p style="margin-right:0px;margin-left:0px"><br>
</p>
<p style="margin-right:0px;margin-left:0px">Best, </p>
<span><font color="#888888">
<p style="margin-right:0px;margin-left:0px">Hui</p>
</font></span></div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all"><span class="HOEnZb"><font color="#888888">
<span><font color="#888888">
<div><br>
</div>
-- <br>
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
</font></span></font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
<br>
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
</font></span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</div></div>