<div dir="ltr"><br><br><div class="gmail_quote"><div dir="ltr">On Tue, Jun 26, 2018 at 12:29 AM Balay, Satish <<a href="mailto:balay@mcs.anl.gov">balay@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div class="m_-619367928269439975WordSection1">
<p class="MsoNormal">Perhaps petsc built with openmp is triggering the problem.</p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">You might want to install hyper separately with openmp. And then install petsc with this prebuilt hypre and see if the problem persists</p></div></div></blockquote><div><br></div><div>I am not finding the newest version of hypre on their web page (I see 2.11.2 but the latest is 2.14.0). Could we just build PETSc's download and point to it? Or copy the whole thing out?</div><div><br></div><div>We've never build Hypre AFAIK. Does anyone have any scripts or something for KNL?</div><div><br></div><div>Thanks,</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div class="m_-619367928269439975WordSection1">
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Satish</p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<hr style="display:inline-block;width:98%">
<div id="m_-619367928269439975divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> petsc-dev <<a href="mailto:petsc-dev-bounces@mcs.anl.gov" target="_blank">petsc-dev-bounces@mcs.anl.gov</a>> on behalf of Yang, Ulrike Meier <<a href="mailto:yang11@llnl.gov" target="_blank">yang11@llnl.gov</a>><br>
<b>Sent:</b> Monday, June 25, 2018 8:20:58 PM<br>
<b>To:</b> Mark Adams; Smith, Barry F.<br>
<b>Cc:</b> For users of the development version of PETSc; <a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a> Trebotich<br>
<b>Subject:</b> Re: [petsc-dev] problem with hypre with '--with-openmp=1'</font>
<div> </div>
</div>
<div>
<div class="m_-619367928269439975WordSection1">
<p class="MsoNormal">Hi Mark,<u></u><u></u></p>
<p class="MsoNormal">I don’t have an account on KNL and have not tested hypre there with OpenMP. I believe other people have used it however, but I haven’t heard about any non-convergence issues before now.<u></u><u></u></p>
<p class="MsoNormal">Ulrike<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><b>From:</b> Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> <br>
<b>Sent:</b> Friday, June 22, 2018 4:28 PM<br>
<b>To:</b> Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>><br>
<b>Cc:</b> For users of the development version of PETSc <<a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a>>; Yang, Ulrike Meier <<a href="mailto:yang11@llnl.gov" target="_blank">yang11@llnl.gov</a>>; <a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a> Trebotich <<a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a>><br>
<b>Subject:</b> Re: [petsc-dev] problem with hypre with '--with-openmp=1'<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Fri, Jun 22, 2018 at 4:39 PM Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal"><br>
<br>
> On Jun 22, 2018, at 3:33 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
> <br>
> We are using KNL (Cori) and hypre is not working when configured with '--with-openmp=1', even when not using threads (as far as I can tell, I never use threads).<br>
<br>
It does seem to run correctly without the --with-openmp option?<u></u><u></u></p>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Yes. And I've run it through valgrind.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Ulrike: have you tested on KNL with openMP?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
PETSc code does not know about --with-openmp at all so my guess is some bug in the compiler when building hypre with OpenMP.<br>
<br>
Barry<br>
<br>
> <br>
> Hypre is not converging, for instance with an optimized build:<br>
> <br>
> srun -n 1 ./ex56 -pc_type hypre -ksp_monitor -ksp_converged_reason -ksp_type cg -pc_hypre_type boomeramg<br>
> OMP: Warning #239: KMP_AFFINITY: granularity=fine will be used.<br>
> 0 KSP Residual norm 7.366251922394e+22 <br>
> 1 KSP Residual norm 3.676434682799e+22 <br>
> Linear solve did not converge due to DIVERGED_INDEFINITE_PC iterations 2<br>
> <br>
> Interestingly in debug mode it almost looks good but it is dying:<br>
> <br>
> 05:09 nid02516 maint *= ~/petsc_install/petsc/src/ksp/ksp/examples/tutorials$ make PETSC_DIR=/global/homes/m/madams/petsc_install/petsc-cori-knl-dbg64-intel-omp PETSC_ARCH="" run
<br>
> srun -n 1 ./ex56 -pc_type hypre -ksp_monitor -ksp_converged_reason -ksp_type cg -pc_hypre_type boomeramg<br>
> OMP: Warning #239: KMP_AFFINITY: granularity=fine will be used.<br>
> 0 KSP Residual norm 7.882081712007e+02 <br>
> 1 KSP Residual norm 2.500214073037e+02 <br>
> 2 KSP Residual norm 3.371746347713e+01 <br>
> 3 KSP Residual norm 2.918759396143e+00 <br>
> 4 KSP Residual norm 9.006505495017e-01 <br>
> Linear solve did not converge due to DIVERGED_INDEFINITE_PC iterations 5<br>
> <br>
> This test runs fine on Xeon nodes. I assume that Hypre has been tested on KNL. GAMG runs fine, of coarse and the initial residual is similar to this debug run.<br>
> <br>
> Could PETSc be messing up the matrix conversion to hypre '--with-openmp=1' ?<br>
> <br>
> Any ideas?<br>
> <br>
> Thanks,<br>
> Mark<br>
> <u></u><u></u></p>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</blockquote></div></div>