<div dir="ltr"><div><div><div>First of all, "-pc_type hypre" does not take "-mg_levels_*". The parameters you set may not affect anything.<br><br></div>And Hypre with the default parameters set by PETSc does not scale well as far as we know. We find the following parameters are important:<br><br>-pc_hypre_boomeramg_strong_threshold <br><br>-pc_hypre_boomeramg_max_levels<br><br>-pc_hypre_boomeramg_coarsen_type<br><br>-pc_hypre_boomeramg_agg_nl <br><br>-pc_hypre_boomeramg_agg_num_paths<br><br></div>You could go into the hypre user manual for more details on these parameters.<br><br></div>Fande,<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jun 26, 2018 at 2:25 PM, Junchao Zhang <span dir="ltr"><<a href="mailto:jczhang@mcs.anl.gov" target="_blank">jczhang@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Mark,<div> I re-do the -pc_type hypre experiment without openmp. Now the job finishes instead of running out of time. I have results with 216 processors (see below). The 1728-processor job is still in the queue so I don't know how it scales. But for the 216-processor one, the execution time is 245 seconds. With -pc_type gamg, the time is 107 seconds. My options are </div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div>-ksp_norm_type unpreconditioned </div>-ksp_rtol 1E-6<br>-ksp_type cg<br>-log_view<br>-mesh_size 1E-4<br>-mg_levels_esteig_ksp_max_it 10<br>-mg_levels_esteig_ksp_type cg<br>-mg_levels_ksp_max_it 1<br>-mg_levels_ksp_norm_type none<br>-mg_levels_ksp_type richardson<br>-mg_levels_pc_sor_its 1<br>-mg_levels_pc_type sor<br>-nodes_per_proc 30<div><div>-pc_type hypre</div></div></blockquote><div><br></div><div>It is a 7-point stencil code. Do you know other hypre options that I can try to improve it? Thanks. </div><span class=""><div><br></div><div><span style="font-family:monospace,monospace">--- Event Stage 2: Remaining Solves</span><br></div></span><div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">KSPSolve 1000 1.0 2.4574e+02 1.0 4.48e+09 1.0 7.6e+06 7.2e+03 2.0e+04 97100100100100 100100100100100 3928</font></div><div><font face="monospace, monospace">VecTDot 12000 1.0 6.5646e+00 2.2 6.48e+08 1.0 0.0e+00 0.0e+00 1.2e+04 2 14 0 0 60 2 14 0 0 60 21321</font></div><div><font face="monospace, monospace">VecNorm 8000 1.0 9.7144e-01 1.2 4.32e+08 1.0 0.0e+00 0.0e+00 8.0e+03 0 10 0 0 40 0 10 0 0 40 96055</font></div><div><font face="monospace, monospace">VecCopy 1000 1.0 7.9706e-02 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</font></div><div><font face="monospace, monospace">VecSet 6000 1.0 1.7941e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</font></div><div><font face="monospace, monospace">VecAXPY 12000 1.0 7.5738e-01 1.2 6.48e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 14 0 0 0 0 14 0 0 0 184806</font></div><div><font face="monospace, monospace">VecAYPX 6000 1.0 4.6802e-01 1.3 2.97e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 7 0 0 0 0 7 0 0 0 137071</font></div><div><font face="monospace, monospace">VecScatterBegin 7000 1.0 4.7924e-01 2.3 0.00e+00 0.0 7.6e+06 7.2e+03 0.0e+00 0 0100100 0 0 0100100 0 0</font></div><div><font face="monospace, monospace">VecScatterEnd 7000 1.0 7.9303e-01 2.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0</font></div><div><font face="monospace, monospace">MatMult 7000 1.0 6.0762e+00 1.1 2.46e+09 1.0 7.6e+06 7.2e+03 0.0e+00 2 55100100 0 2 55100100 0 86894</font></div><div><font face="monospace, monospace">PCApply 6000 1.0 2.3429e+02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 92 0 0 0 0 95 0 0 0 0 0</font></div></div><span class="HOEnZb"><font color="#888888"><div><br></div></font></span></div><div class="gmail_extra"><span class="HOEnZb"><font color="#888888"><br clear="all"><div><div class="m_-9080878473384440017gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div></font></span><div><div class="h5">
<br><div class="gmail_quote">On Thu, Jun 14, 2018 at 5:45 PM, Junchao Zhang <span dir="ltr"><<a href="mailto:jczhang@mcs.anl.gov" target="_blank">jczhang@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>I tested <span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">-pc_gamg_</span><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">repartition with 216 processors again. First I tested with these options</span></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-log_view \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-ksp_rtol 1E-6 \</div></span></div><span><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-ksp_type cg \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-ksp_norm_type unpreconditioned \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-mg_levels_ksp_type richardson \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-mg_levels_ksp_norm_type none \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-mg_levels_pc_type sor \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-mg_levels_ksp_max_it 1 \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-mg_levels_pc_sor_its 1 \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-mg_levels_esteig_ksp_type cg \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-mg_levels_esteig_ksp_max_it 10 \</div></span></div></span><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-pc_type gamg \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-pc_gamg_type agg \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-pc_gamg_threshold 0.05 \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-pc_gamg_type classical \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-gamg_est_ksp_type cg \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-pc_gamg_square_graph 10 \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-pc_gamg_threshold 0.0</div></span></div></blockquote><div><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">then I tested with an extra -pc_gamg_repartition. With repartition, the time <span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">increased from 120s to 140s. <span> </span></span></span><span style="font-size:12.8px">The code measures first KSPSolve and the remaining in separate stages, so the repartition time was not counted in the stage of interest. Actually, log_view says GMAG :repartition time (in the first event stage) is about 1.5 sec., so it is not a big deal. I also tested -<span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">pc_gamg_square_graph 4. It did not change the time.</span></span></div><div>I tested hypre with options "-log_view -ksp_rtol 1E-6 -ksp_type cg -ksp_norm_type unpreconditioned -pc_type hypre" and nothing else. The code ran out of time. In old tests, a job (1000 KSPSolve with 7 KSP iterations each) took 4 minutes. With hypre, 1 KSPSolve + 6 KSP iterations each, takes 6 minutes.<br></div><div>I will test and profile the code on a single node, and apply some vecscatter optimizations I recently did to see what happens.</div><span class="m_-9080878473384440017HOEnZb"><font color="#888888"><div><span style="font-size:small;text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div></font></span></div><div class="gmail_extra"><span class="m_-9080878473384440017HOEnZb"><font color="#888888"><br clear="all"><div><div class="m_-9080878473384440017m_-5678824141310144083gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div></font></span><div><div class="m_-9080878473384440017h5">
<br><div class="gmail_quote">On Thu, Jun 14, 2018 at 11:03 AM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">And with 7-point stensils and no large material discontinuities you probably want <span style="color:rgb(80,0,80);font-size:12.8px;text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">-pc_gamg_square_graph 10 </span><span style="color:rgb(80,0,80);font-size:12.8px;text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">-pc_gamg_threshold 0.0 and you could test the square graph parameter (eg, 1,2,3,4).</span><div><span style="color:rgb(80,0,80);font-size:12.8px;text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="color:rgb(80,0,80);font-size:12.8px;text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">And I would definitely test hypre.</span></div></div><div class="m_-9080878473384440017m_-5678824141310144083HOEnZb"><div class="m_-9080878473384440017m_-5678824141310144083h5"><br><div class="gmail_quote"><div dir="ltr">On Thu, Jun 14, 2018 at 8:54 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div><br></div><div>Just -pc_type hypre instead of -pc_type gamg.</div><div> </div></div></div></blockquote><div>And you need to have configured PETSc with hypre. </div><div> </div></div></div>
</blockquote></div>
</div></div></blockquote></div><br></div></div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div>