<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
<div class="WordSection1">
<p class="MsoNormal">Perhaps petsc built with openmp is triggering the problem.</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">You might want to install hyper separately with openmp. And then install petsc with this prebuilt hypre and see if the problem persists</p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Satish</p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> petsc-dev <petsc-dev-bounces@mcs.anl.gov> on behalf of Yang, Ulrike Meier <yang11@llnl.gov><br>
<b>Sent:</b> Monday, June 25, 2018 8:20:58 PM<br>
<b>To:</b> Mark Adams; Smith, Barry F.<br>
<b>Cc:</b> For users of the development version of PETSc; treb@lbl.gov Trebotich<br>
<b>Subject:</b> Re: [petsc-dev] problem with hypre with '--with-openmp=1'</font>
<div> </div>
</div>
<div>
<div class="WordSection1">
<p class="MsoNormal">Hi Mark,<o:p></o:p></p>
<p class="MsoNormal">I don’t have an account on KNL and have not tested hypre there with OpenMP. I believe other people have used it however, but I haven’t heard about any non-convergence issues before now.<o:p></o:p></p>
<p class="MsoNormal">Ulrike<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>From:</b> Mark Adams <mfadams@lbl.gov> <br>
<b>Sent:</b> Friday, June 22, 2018 4:28 PM<br>
<b>To:</b> Smith, Barry F. <bsmith@mcs.anl.gov><br>
<b>Cc:</b> For users of the development version of PETSc <petsc-dev@mcs.anl.gov>; Yang, Ulrike Meier <yang11@llnl.gov>; treb@lbl.gov Trebotich <treb@lbl.gov><br>
<b>Subject:</b> Re: [petsc-dev] problem with hypre with '--with-openmp=1'<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Fri, Jun 22, 2018 at 4:39 PM Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal"><br>
<br>
> On Jun 22, 2018, at 3:33 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
> <br>
> We are using KNL (Cori) and hypre is not working when configured with '--with-openmp=1', even when not using threads (as far as I can tell, I never use threads).<br>
<br>
It does seem to run correctly without the --with-openmp option?<o:p></o:p></p>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Yes. And I've run it through valgrind.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Ulrike: have you tested on KNL with openMP?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
PETSc code does not know about --with-openmp at all so my guess is some bug in the compiler when building hypre with OpenMP.<br>
<br>
Barry<br>
<br>
> <br>
> Hypre is not converging, for instance with an optimized build:<br>
> <br>
> srun -n 1 ./ex56 -pc_type hypre -ksp_monitor -ksp_converged_reason -ksp_type cg -pc_hypre_type boomeramg<br>
> OMP: Warning #239: KMP_AFFINITY: granularity=fine will be used.<br>
> 0 KSP Residual norm 7.366251922394e+22 <br>
> 1 KSP Residual norm 3.676434682799e+22 <br>
> Linear solve did not converge due to DIVERGED_INDEFINITE_PC iterations 2<br>
> <br>
> Interestingly in debug mode it almost looks good but it is dying:<br>
> <br>
> 05:09 nid02516 maint *= ~/petsc_install/petsc/src/ksp/ksp/examples/tutorials$ make PETSC_DIR=/global/homes/m/madams/petsc_install/petsc-cori-knl-dbg64-intel-omp PETSC_ARCH="" run
<br>
> srun -n 1 ./ex56 -pc_type hypre -ksp_monitor -ksp_converged_reason -ksp_type cg -pc_hypre_type boomeramg<br>
> OMP: Warning #239: KMP_AFFINITY: granularity=fine will be used.<br>
> 0 KSP Residual norm 7.882081712007e+02 <br>
> 1 KSP Residual norm 2.500214073037e+02 <br>
> 2 KSP Residual norm 3.371746347713e+01 <br>
> 3 KSP Residual norm 2.918759396143e+00 <br>
> 4 KSP Residual norm 9.006505495017e-01 <br>
> Linear solve did not converge due to DIVERGED_INDEFINITE_PC iterations 5<br>
> <br>
> This test runs fine on Xeon nodes. I assume that Hypre has been tested on KNL. GAMG runs fine, of coarse and the initial residual is similar to this debug run.<br>
> <br>
> Could PETSc be messing up the matrix conversion to hypre '--with-openmp=1' ?<br>
> <br>
> Any ideas?<br>
> <br>
> Thanks,<br>
> Mark<br>
> <o:p></o:p></p>
</blockquote>
</div>
</div>
</div>
</div>
</body>
</html>