<div dir="ltr"><div>BTW, this is from that super slow with-openmp run on 8 procs. Barrier looks sad.</div><div><br></div><div>========================================================================================================================</div><div>Average time to get PetscTime(): 5.00679e-07</div><div>Average time for MPI_Barrier(): 0.1064</div><div>Average time for zero size MPI_Send(): 0.00800014</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Jun 26, 2018 at 10:36 AM Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Interesting, I am seeing the same thing with ksp/ex56 (elasticity) with 30^3 grid on each process. One process runs fine (1.5 sec) but 8 processes with 30^3 on each process took 156 sec.<div><br></div><div>And, PETSc's log_view is running extremely slow. I have the total time (156) but each event is taking like a minute or more to come out.</div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Jun 26, 2018 at 10:13 AM Junchao Zhang <<a href="mailto:jczhang@mcs.anl.gov" target="_blank">jczhang@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><div class="gmail_quote">On Tue, Jun 26, 2018 at 8:26 AM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><br><div class="gmail_quote"><span><div dir="ltr">On Tue, Jun 26, 2018 at 12:19 AM Junchao Zhang <<a href="mailto:jczhang@mcs.anl.gov" target="_blank">jczhang@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Mark,<div> Your email reminded me my recent experiments. My PETSc was configured <span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">--with-openmp=1. With hypre, my job ran out of time. That was on an Argonne Xeon cluster.</span></div></div></blockquote><div><br></div></span><div>Interesting. I tested on Cori's Haswell nodes and it looked fine. I did not time it but seemed OK.</div><span><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"> I repeated the experiments on Cori's Haswell nodes. <span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">--with-openmp=1, "Linear solve converged due to CONVERGED_RTOL iterations 5". But it took very long time (10 mins). Without <span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">--with-openmp=1, it took less than 1 second.</span></span></span></div></div></blockquote><div><br></div></span><div>Humm. I seemed to run OK on Cori's Haswell nodes. Where you running a significant sized job? I was test small serial runs.</div></div></div></blockquote><div><br></div><div> I ran with 27 processors and each had 30^3 unknowns.</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><span><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="gmail_extra"><br clear="all"><div><div class="m_-8388408162126975118m_6687375029577693804m_-3983514530010265515m_8979616536021922214gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div>
<br><div class="gmail_quote">On Fri, Jun 22, 2018 at 3:33 PM, Mark Adams <span dir="ltr"><<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">We are using KNL (Cori) and hypre is not working when configured with '--with-openmp=1', even when not using threads (as far as I can tell, I never use threads).<div><br></div><div>Hypre is not converging, for instance with an optimized build:</div><div><br></div><div><div>srun -n 1 ./ex56 -pc_type hypre -ksp_monitor -ksp_converged_reason -ksp_type cg -pc_hypre_type boomeramg</div><div>OMP: Warning #239: KMP_AFFINITY: granularity=fine will be used.</div><div> 0 KSP Residual norm 7.366251922394e+22 </div><div> 1 KSP Residual norm 3.676434682799e+22 </div><div>Linear solve did not converge due to DIVERGED_INDEFINITE_PC iterations 2</div></div><div><br></div><div>Interestingly in debug mode it almost looks good but it is dying:</div><div><br></div><div><div>05:09 nid02516 maint *= ~/petsc_install/petsc/src/ksp/ksp/examples/tutorials$ make PETSC_DIR=/global/homes/m/madams/petsc_install/petsc-cori-knl-dbg64-intel-omp PETSC_ARCH="" run </div><div>srun -n 1 ./ex56 -pc_type hypre -ksp_monitor -ksp_converged_reason -ksp_type cg -pc_hypre_type boomeramg</div><div>OMP: Warning #239: KMP_AFFINITY: granularity=fine will be used.</div><div> 0 KSP Residual norm 7.882081712007e+02 </div><div> 1 KSP Residual norm 2.500214073037e+02 </div><div> 2 KSP Residual norm 3.371746347713e+01 </div><div> 3 KSP Residual norm 2.918759396143e+00 </div><div> 4 KSP Residual norm 9.006505495017e-01 </div><div>Linear solve did not converge due to DIVERGED_INDEFINITE_PC iterations 5</div></div><div><br></div><div>This test runs fine on Xeon nodes. I assume that Hypre has been tested on KNL. GAMG runs fine, of coarse and the initial residual is similar to this debug run.</div><div><br></div><div>Could PETSc be messing up the matrix conversion to hypre <span style="background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">'--with-openmp=1'</span> ?</div><div><br></div><div>Any ideas?</div><div><br></div><div>Thanks,</div><div>Mark</div><div><br></div></div>
</blockquote></div><br></div>
</blockquote></span></div></div>
</blockquote></div><br></div></div>
</blockquote></div>
</blockquote></div>