<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Note, you need to damp jacobi for a smoother with something like this:<div><br></div><div><div>-mg_levels_ksp_type chebyshev</div><div>-mg_levels_ksp_chebyshev_estimate_eigenvalues 0,0.1,0,1.05 </div><div><br></div><div><br></div><div><div>On Apr 18, 2013, at 9:49 AM, "Christon, Mark A" <<a href="mailto:christon@lanl.gov">christon@lanl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Helvetica, sans-serif; ">
<div>
<div>
<div>HI Mark,</div>
<div><br>
</div>
<div>Thanks for the information. We thought something had changed and could see it the effect, but couldn't quite pin it down.</div>
<div><br>
</div>
<div>To be clear our pressure-Poisson equation is a warm and fluffy Laplacian, but typically quite stiff from a spectral point of view when dealing with boundary-layer meshes and complex geometry — our norm. This is the first-order computational cost in our
flow solver, so hit's like the recent change are very problematic, and particularly so when they break a number of regression tests that run nightly across multiple platflorms.</div>
<div><br>
</div>
<div>So, unfortunately, while I'd like to use something as Jacobi, it's completely ineffective in for the operator (and RHS) in question.</div>
<div><br>
</div>
<div>Thanks.</div>
<div><br>
</div>
<div>
<div>
<div>- Mark</div>
<div><br>
</div>
<div>-- </div>
<div>
<div style="font-family: Calibri; "><div style="margin: 0in 0in 0.0001pt; font-family: 'Times New Roman', serif; ">
<span style="font-family: Helvetica, sans-serif; ">Mark A. Christon<o:p></o:p></span></div>
</div>
<div style="font-family: Calibri; "><div style="margin: 0in 0in 0.0001pt; font-family: 'Times New Roman', serif; ">
<span style="font-family: Helvetica, sans-serif; ">Computational Physics Group (CCS-2)<o:p></o:p></span></div>
</div>
<div style="font-family: Calibri; "><div style="margin: 0in 0in 0.0001pt; font-family: 'Times New Roman', serif; ">
<span style="font-family: Helvetica, sans-serif; ">Computer, Computational and Statistical Sciences Division<o:p></o:p></span></div>
</div>
<div style="font-family: Calibri; "><div style="margin: 0in 0in 0.0001pt; font-family: 'Times New Roman', serif; ">
<span style="font-family: Helvetica, sans-serif; ">Los Alamos National Laboratory<o:p></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-family: 'Times New Roman', serif; ">
<span style="font-family: Helvetica, sans-serif; ">MS D413, P.O. Box 1663</span></div><div style="margin: 0in 0in 0.0001pt; font-family: 'Times New Roman', serif; ">
<span style="font-family: Helvetica, sans-serif; ">Los Alamos, NM 87545</span></div><div style="margin: 0in 0in 0.0001pt; font-family: 'Times New Roman', serif; ">
<span style="font-family: Helvetica, sans-serif; "><br>
</span></div>
</div>
<div style="font-family: Calibri; "><div style="margin: 0in 0in 0.0001pt; font-family: 'Times New Roman', serif; ">
<span style="font-family: Helvetica, sans-serif; ">E-mail: <a href="mailto:christon@lanl.gov" style="color: blue; text-decoration: underline; ">christon@lanl.gov</a><o:p></o:p></span></div>
</div>
<div style="font-family: Calibri; "><div style="margin: 0in 0in 0.0001pt; font-family: 'Times New Roman', serif; ">
<span style="font-family: Helvetica, sans-serif; ">Phone: (505) 663-5124 </span></div><div style="margin: 0in 0in 0.0001pt; font-family: 'Times New Roman', serif; ">
<span style="font-family: Helvetica, sans-serif; ">Mobile: (505) 695-5649 (voice mail)<o:p></o:p></span></div>
</div>
<div style="font-family: Calibri; "><div style="margin: 0in 0in 0.0001pt; font-family: 'Times New Roman', serif; ">
<span style="font-family: Helvetica, sans-serif; "> </span></div>
</div>
<div style="font-family: Calibri; font-size: medium; "><div style="margin: 0in 0in 0.0001pt; font-family: 'Times New Roman', serif; ">
<span style="font-family: Helvetica, sans-serif; "><a href="http://wileyonlinelibrary.com/journal/fld" style="color: blue; text-decoration: underline; "><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px; ">International
Journal for Numerical Methods in Fluids</span></font></a></span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family: Calibri; font-size: 11pt; text-align: left; border-width: 1pt medium medium; border-style: solid none none; padding: 3pt 0in 0in; border-top-color: rgb(181, 196, 223); ">
<span style="font-weight:bold">From: </span>"Mark F. Adams" <<a href="mailto:mark.adams@columbia.edu">mark.adams@columbia.edu</a>><br>
<span style="font-weight:bold">Reply-To: </span>PETSc users list <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>><br>
<span style="font-weight:bold">Date: </span>Wed, 17 Apr 2013 19:42:47 -0400<br>
<span style="font-weight:bold">To: </span>PETSc users list <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [petsc-users] Fwd: Any changes in ML usage between 3.1-p8 -> 3.3-p6?<br>
</div>
<div><br>
</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;" type="cite">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
In looking at the logs for icc it looks like Hong has done a little messing around with the shifting tolerance:
<div><br>
</div>
<div>
<div class="deletion" data-fnum="199" style="color: rgb(51, 51, 51); font-family: Arial, sans-serif; font-size: 0px; line-height: 0px; background-color: rgb(255, 255, 255); ">
<pre class="source" style="font-family: 'Bitstream Vera Sans Mono', 'DejaVu Sans Mono', Monaco, monospace; font-size: 11px; line-height: 18px; margin-top: 0px; margin-bottom: 0px; background-color: rgb(255, 221, 221); color: rgb(57, 57, 57); height: 18px; padding-left: 72px; ">- ((PC_Factor*)icc)->info.shiftamount = 1.e-12;</pre>
</div>
<div class="deletion" data-fnum="200" style="color: rgb(51, 51, 51); font-family: Arial, sans-serif; font-size: 0px; line-height: 0px; background-color: rgb(255, 255, 255); ">
<a class="line-numbers" id="Lsrc/ksp/pc/impls/factor/icc/icc.cF200" href="https://bitbucket.org/petsc/petsc/src/0ed735ceb4be/src/ksp/pc/impls/factor/icc/icc.c#Lsrc/ksp/pc/impls/factor/icc/icc.cF200" style="color: rgb(57, 57, 57); text-decoration: none; font-family: 'Bitstream Vera Sans Mono', 'DejaVu Sans Mono', Monaco, monospace; font-size: 11px; height: 18px; line-height: 18px; position: absolute; left: 1px; width: 70px; background-color: rgb(245, 245, 245); border-right-width: 1px; border-right-style: solid; border-right-color: rgb(204, 204, 204); -webkit-user-select: none; "></a>
<pre class="source" style="font-family: 'Bitstream Vera Sans Mono', 'DejaVu Sans Mono', Monaco, monospace; font-size: 11px; line-height: 18px; margin-top: 0px; margin-bottom: 0px; background-color: rgb(255, 221, 221); color: rgb(57, 57, 57); height: 18px; padding-left: 72px; ">- ((PC_Factor*)icc)->info.zeropivot = 1.e-12;</pre>
</div>
<div class="addition" data-tnum="199" style="color: rgb(51, 51, 51); font-family: Arial, sans-serif; font-size: 0px; line-height: 0px; background-color: rgb(255, 255, 255); ">
<a class="line-numbers" id="Lsrc/ksp/pc/impls/factor/icc/icc.cT199" href="https://bitbucket.org/petsc/petsc/src/0ed735ceb4be/src/ksp/pc/impls/factor/icc/icc.c#Lsrc/ksp/pc/impls/factor/icc/icc.cT199" style="color: rgb(57, 57, 57); text-decoration: none; font-family: 'Bitstream Vera Sans Mono', 'DejaVu Sans Mono', Monaco, monospace; font-size: 11px; height: 18px; line-height: 18px; position: absolute; left: 1px; width: 70px; background-color: rgb(245, 245, 245); border-right-width: 1px; border-right-style: solid; border-right-color: rgb(204, 204, 204); -webkit-user-select: none; "></a>
<pre class="source" style="font-family: 'Bitstream Vera Sans Mono', 'DejaVu Sans Mono', Monaco, monospace; font-size: 11px; line-height: 18px; margin-top: 0px; margin-bottom: 0px; background-color: rgb(204, 255, 204); color: rgb(57, 57, 57); height: 18px; padding-left: 72px; ">+ ((PC_Factor*)icc)->info.shiftamount = 100.0*PETSC_MACHINE_EPSILON;</pre>
</div>
<div class="addition" data-tnum="200" style="color: rgb(51, 51, 51); font-family: Arial, sans-serif; font-size: 0px; line-height: 0px; background-color: rgb(255, 255, 255); ">
<a class="line-numbers" id="Lsrc/ksp/pc/impls/factor/icc/icc.cT200" href="https://bitbucket.org/petsc/petsc/src/0ed735ceb4be/src/ksp/pc/impls/factor/icc/icc.c#Lsrc/ksp/pc/impls/factor/icc/icc.cT200" style="color: rgb(57, 57, 57); text-decoration: none; font-family: 'Bitstream Vera Sans Mono', 'DejaVu Sans Mono', Monaco, monospace; font-size: 11px; height: 18px; line-height: 18px; position: absolute; left: 1px; width: 70px; background-color: rgb(245, 245, 245); border-right-width: 1px; border-right-style: solid; border-right-color: rgb(204, 204, 204); -webkit-user-select: none; "></a>
<pre class="source" style="font-family: 'Bitstream Vera Sans Mono', 'DejaVu Sans Mono', Monaco, monospace; font-size: 11px; line-height: 18px; margin-top: 0px; margin-bottom: 0px; background-color: rgb(204, 255, 204); color: rgb(57, 57, 57); height: 18px; padding-left: 72px; ">+ ((PC_Factor*)icc)->info.zeropivot = 100.0*PETSC_MACHINE_EPSILON;</pre>
</div>
<div><br>
</div>
<div><br>
</div>
<div>This looks like it would lower the shifting and drop tolerance. You might set these back to 1e-12. </div>
<div><br>
</div>
<div><a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFactorSetZeroPivot.html">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFactorSetZeroPivot.html</a></div>
<div><br>
</div>
<div><a href="http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFactorSetShiftAmount.html">http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFactorSetShiftAmount.html</a><br>
<div><br>
</div>
<div>BTW, using an indefinite preconditioner, that has to be fixed with is-this-a-small-number kind of code, on a warm and fluffy Laplacian is not recommended. As I said before I would just use jacobi -- god gave you an easy problem. Exploit it.</div>
<div><br>
<div>
<div>On Apr 17, 2013, at 7:22 PM, "Mark F. Adams" <<a href="mailto:mark.adams@columbia.edu">mark.adams@columbia.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<br>
<div><br>
<div>Begin forwarded message:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;">
<span style="font-family: Helvetica; font-size: medium; "><b>From: </b></span><span style="font-family:'Helvetica'; font-size:medium;">"Christon, Mark A" <<a href="mailto:christon@lanl.gov">christon@lanl.gov</a>><br>
</span></div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;">
<span style="font-family: Helvetica; font-size: medium; "><b>Subject: </b></span><span style="font-family:'Helvetica'; font-size:medium;"><b>Re: [petsc-users] Any changes in ML usage between 3.1-p8 -> 3.3-p6?</b><br>
</span></div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;">
<span style="font-family: Helvetica; font-size: medium; "><b>Date: </b></span><span style="font-family:'Helvetica'; font-size:medium;">April 17, 2013 7:06:11 PM EDT<br>
</span></div>
<div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;">
<span style="font-family: Helvetica; font-size: medium; "><b>To: </b></span><span style="font-family:'Helvetica'; font-size:medium;">"Mark F. Adams" <<a href="mailto:mark.adams@columbia.edu">mark.adams@columbia.edu</a>>, "Bakosi, Jozsef" <<a href="mailto:jbakosi@lanl.gov">jbakosi@lanl.gov</a>><br>
</span></div>
<br>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; font-size: 14px; font-family: Helvetica, sans-serif; ">
<div>
<div>
<div>Hi Mark,</div>
<div><br>
</div>
<div>Yes, looks like the new version does a little better after 2 iterations, but at the 8th iteration, the residuals increase:(</div>
<div><br>
</div>
<div>I suspect this is why PETSc is whining about an indefinite preconditioner. </div>
<div><br>
</div>
<div>Something definitely changes as we've had about 6-8 regression tests start failing that have been running flawlessly with ML + PETSc 3.1-p8 for almost two years.</div>
<div><br>
</div>
<div>If we can understand what changed, we probably have a fighting chance of correcting it — assuming it's some solver setting for PETSc that we're not currently using.</div>
<div><br>
</div>
<div>
<div>
<div>- Mark</div>
<div><br>
</div>
<div>-- </div>
<div>
<div style="font-family: Calibri; ">
<div style="margin: 0in 0in 0.0001pt; font-family: 'Times New Roman', serif; "><span style="font-family: Helvetica, sans-serif; ">Mark A. Christon<o:p></o:p></span></div>
</div>
<div style="font-family: Calibri; ">
<div style="margin: 0in 0in 0.0001pt; font-family: 'Times New Roman', serif; "><span style="font-family: Helvetica, sans-serif; ">Computational Physics Group (CCS-2)<o:p></o:p></span></div>
</div>
<div style="font-family: Calibri; ">
<div style="margin: 0in 0in 0.0001pt; font-family: 'Times New Roman', serif; "><span style="font-family: Helvetica, sans-serif; ">Computer, Computational and Statistical Sciences Division<o:p></o:p></span></div>
</div>
<div style="font-family: Calibri; ">
<div style="margin: 0in 0in 0.0001pt; font-family: 'Times New Roman', serif; "><span style="font-family: Helvetica, sans-serif; ">Los Alamos National Laboratory<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-family: 'Times New Roman', serif; "><span style="font-family: Helvetica, sans-serif; ">MS D413, P.O. Box 1663</span></div>
<div style="margin: 0in 0in 0.0001pt; font-family: 'Times New Roman', serif; "><span style="font-family: Helvetica, sans-serif; ">Los Alamos, NM 87545</span></div>
<div style="margin: 0in 0in 0.0001pt; font-family: 'Times New Roman', serif; "><span style="font-family: Helvetica, sans-serif; "><br>
</span></div>
</div>
<div style="font-family: Calibri; ">
<div style="margin: 0in 0in 0.0001pt; font-family: 'Times New Roman', serif; "><span style="font-family: Helvetica, sans-serif; ">E-mail: <a href="mailto:christon@lanl.gov" style="color: blue; text-decoration: underline; ">christon@lanl.gov</a><o:p></o:p></span></div>
</div>
<div style="font-family: Calibri; ">
<div style="margin: 0in 0in 0.0001pt; font-family: 'Times New Roman', serif; "><span style="font-family: Helvetica, sans-serif; ">Phone: (505) 663-5124 </span></div>
<div style="margin: 0in 0in 0.0001pt; font-family: 'Times New Roman', serif; "><span style="font-family: Helvetica, sans-serif; ">Mobile: (505) 695-5649 (voice mail)<o:p></o:p></span></div>
</div>
<div style="font-family: Calibri; ">
<div style="margin: 0in 0in 0.0001pt; font-family: 'Times New Roman', serif; "><span style="font-family: Helvetica, sans-serif; "> </span></div>
</div>
<div style="font-family: Calibri; font-size: medium; ">
<div style="margin: 0in 0in 0.0001pt; font-family: 'Times New Roman', serif; "><span style="font-family: Helvetica, sans-serif; "><a href="http://wileyonlinelibrary.com/journal/fld" style="color: blue; text-decoration: underline; "><font class="Apple-style-span" size="4"><span class="Apple-style-span" style="font-size: 14px; ">International
Journal for Numerical Methods in Fluids</span></font></a></span></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family: Calibri; font-size: 11pt; text-align: left; border-width: 1pt medium medium; border-style: solid none none; padding: 3pt 0in 0in; border-top-color: rgb(181, 196, 223); ">
<span style="font-weight:bold">From: </span>"Mark F. Adams" <<a href="mailto:mark.adams@columbia.edu">mark.adams@columbia.edu</a>><br>
<span style="font-weight:bold">Date: </span>Wed, 17 Apr 2013 18:51:02 -0400<br>
<span style="font-weight:bold">To: </span>PETSc users list <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>><br>
<span style="font-weight:bold">Cc: </span>"Mark A. Christon" <<a href="mailto:christon@lanl.gov">christon@lanl.gov</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [petsc-users] Any changes in ML usage between 3.1-p8 -> 3.3-p6?<br>
</div>
<div><br>
</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;" type="cite">
<div>
<div>
<div>I see you are using icc. Perhaps our icc changed a bit between versions. These results look like both solves are working and the old does a little better (after two iterations).</div>
<div><br>
</div>
<div>Try using jacobi instead of icc.</div>
<div><br>
</div>
<div><br>
</div>
<div>On Apr 17, 2013, at 6:21 PM, Jozsef Bakosi <<a href="mailto:jbakosi@lanl.gov">jbakosi@lanl.gov</a>> wrote:</div>
<div><br>
</div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;" type="cite">
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;" type="cite">
<div>On 04.17.2013 15:38, Matthew Knepley wrote:</div>
<div></div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;" type="cite">
<div>On 04.17.2013 14:26, Jozsef Bakosi wrote:</div>
<div></div>
<blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;" type="cite">
<div>Mark F. Adams mark.adams at <a href="http://columbia.edu/">columbia.edu</a></div>
<div>Wed Apr 17 14:25:04 CDT 2013</div>
<div></div>
<div>2) If you get "Indefinite PC" (I am guessing from using CG) it is because the</div>
<div>preconditioner</div>
<div> really is indefinite (or possible non-symmetric). We improved the checking</div>
<div> for this in one</div>
<div> of those releases.</div>
<div></div>
<div>AMG does not guarantee an SPD preconditioner so why persist in trying to use</div>
<div>CG?</div>
<div></div>
<div></div>
<div>AMG is positive if everything is working correctly.</div>
<div></div>
<div>Are these problems only semidefinite? Singular systems can give erratic</div>
<div>behavior.</div>
</blockquote>
<div></div>
<div>It is a Laplace operator from Galerkin finite elements. And the PC is fine on</div>
<div>ranks 1, 2, 3, and 5 -- indefinite only on 4. I think we can safely say that the</div>
<div>same PC should be positive on 4 as well.</div>
</blockquote>
<div></div>
<div>Why is it safe? Because it sounds plausible? Mathematics is replete with things</div>
<div>that sound plausible and are false. Are there proofs that suggest this? Is there</div>
<div>computational evidence? Why would I believe you?</div>
</blockquote>
<div></div>
<div>Okay, so here is some additional information:</div>
<div></div>
<div>I tried both old and new PETSc versions again, but now only taking 2 iterations</div>
<div>(both with 4 CPUs) and checked the residuals. I get the same exact PC from ML in</div>
<div>both cases, however, the residuals are different after both iterations:</div>
<div></div>
<div>Please do a diff on the attached files and you can verify that the ML</div>
<div>diagnostics are exactly the same: same max eigenvalues, nodes aggregated, etc,</div>
<div>while the norm coming out of the solver at the end at both iterations are</div>
<div>different.</div>
<div></div>
<div>We reproduced the same exact behavior on two different linux platforms.</div>
<div></div>
<div>Once again: same application source code, same ML source code, different PETSc:</div>
<div>3.1-p8 vs. 3.3-p6.</div>
<div><old.out><new.out></div>
</blockquote>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</blockquote>
</span></div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</span>
</div>
</blockquote></div><br></div></body></html>