<div dir="ltr"><div>I tested <span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">-pc_gamg_</span><wbr style="font-size:12.8px;text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">repartition with 216 processors again. First I tested with these options</span></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-log_view \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-ksp_rtol 1E-6 \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-ksp_type cg \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-ksp_norm_type unpreconditioned \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-mg_levels_ksp_type richardson \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-mg_levels_ksp_norm_type none \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-mg_levels_pc_type sor \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-mg_levels_ksp_max_it 1 \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-mg_levels_pc_sor_its 1 \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-mg_levels_esteig_ksp_type cg \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-mg_levels_esteig_ksp_max_it 10 \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-pc_type gamg \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-pc_gamg_type agg \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-pc_gamg_threshold 0.05 \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-pc_gamg_type classical \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-gamg_est_ksp_type cg \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-pc_gamg_square_graph 10 \</div></span></div><div><span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline;font-size:12.8px"><div>-pc_gamg_threshold 0.0</div></span></div></blockquote><div><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">then I tested with an extra -pc_gamg_repartition. With repartition, the time <span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">increased from 120s to 140s. <span> </span></span></span><span style="font-size:12.8px">The code measures first KSPSolve and the remaining in separate stages, so the repartition time was not counted in the stage of interest. Actually, log_view says GMAG :repartition time (in the first event stage) is about 1.5 sec., so it is not a big deal. I also tested -<span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">pc_gamg_square_graph 4. It did not change the time.</span></span></div><div>I tested hypre with options "-log_view -ksp_rtol 1E-6 -ksp_type cg -ksp_norm_type unpreconditioned -pc_type hypre"  and nothing else. The code ran out of time. In old tests, a job (1000 KSPSolve with 7 KSP iterations each) took 4 minutes. With hypre, 1 KSPSolve + 6 KSP iterations each, takes 6 minutes.<br></div><div>I will test and profile the code on a single node, and apply some vecscatter optimizations I recently did to see what happens.</div><div><span style="font-size:small;text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div>
<br><div class="gmail_quote">On Thu, Jun 14, 2018 at 11:03 AM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">And with 7-point stensils and no large material discontinuities you probably want <span style="color:rgb(80,0,80);font-size:12.8px;text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">-pc_gamg_square_graph 10 </span><span style="color:rgb(80,0,80);font-size:12.8px;text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">-pc_gamg_threshold 0.0 and you could test the square graph parameter (eg, 1,2,3,4).</span><div><span style="color:rgb(80,0,80);font-size:12.8px;text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="color:rgb(80,0,80);font-size:12.8px;text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">And I would definitely test hypre.</span></div></div><div class="HOEnZb"><div class="h5"><br><div class="gmail_quote"><div dir="ltr">On Thu, Jun 14, 2018 at 8:54 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div><br></div><div>Just -pc_type hypre instead of -pc_type gamg.</div><div> </div></div></div></blockquote><div>And you need to have configured PETSc with hypre. </div><div> </div></div></div>
</blockquote></div>
</div></div></blockquote></div><br></div>