<div dir="ltr"><div dir="ltr">On Thu, Nov 17, 2022 at 2:34 PM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk">karthikeyan.chockalingam@stfc.ac.uk</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg-1279333881495980509">
<div lang="EN-GB" style="overflow-wrap: break-word;">
<div class="m_-1279333881495980509WordSection1">
<p class="MsoNormal"><span style="font-size:11pt">I have tried and not sure how to set
</span><span class="m_-1279333881495980509s4">"-pc_svd_monitor” </span><span style="font-size:11pt">since, I have not yet setup the command line option. (I am currently using PETSc within another framework).</span></p></div></div></div></blockquote><div><br></div><div>Sorry this email got lost for me. You just call</div><div><br></div><div>PetscOptionsSetValue(NULL, "-pc_svd_monitor", "1");</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg-1279333881495980509"><div lang="EN-GB" style="overflow-wrap: break-word;"><div class="m_-1279333881495980509WordSection1"><p class="MsoNormal"> </p></div></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg-1279333881495980509"><div lang="EN-GB" style="overflow-wrap: break-word;"><div class="m_-1279333881495980509WordSection1"><p class="MsoNormal"></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9pt;font-family:Menlo;color:black"> </span><span style="font-size:9pt;font-family:Menlo;color:rgb(57,0,160)">KSP</span><span style="font-size:9pt;font-family:Menlo;color:black">
ksp;<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9pt;font-family:Menlo;color:black">
</span><span style="font-size:9pt;font-family:Menlo;color:rgb(57,0,160)">PC</span><span style="font-size:9pt;font-family:Menlo;color:black"> pc;<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9pt;font-family:Menlo;color:rgb(108,54,169)"> KSPCreate(PETSC_COMM_WORLD, &ksp);<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9pt;font-family:Menlo;color:black">
</span><span style="font-size:9pt;font-family:Menlo;color:rgb(108,54,169)">KSPSetOperators</span><span style="font-size:9pt;font-family:Menlo;color:black">(ksp, A, A);<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<p class="m_-1279333881495980509p1"><span class="m_-1279333881495980509apple-converted-space"> </span>ierr = <span class="m_-1279333881495980509s1">
KSPSetType</span>(ksp,<span class="m_-1279333881495980509s2">KSPPREONLY</span>);<span class="m_-1279333881495980509s2">CHKERRQ</span>(ierr);<u></u><u></u></p>
<p class="m_-1279333881495980509p1"><span class="m_-1279333881495980509apple-converted-space"> </span>ierr = <span class="m_-1279333881495980509s1">
KSPGetPC</span>(ksp,&pc);<span class="m_-1279333881495980509s2">CHKERRQ</span>(ierr);<u></u><u></u></p>
<p class="m_-1279333881495980509p1"><span class="m_-1279333881495980509apple-converted-space"> </span>ierr = <span class="m_-1279333881495980509s1">
PCSetType</span>(pc,<span class="m_-1279333881495980509s2">PCSVD</span>);<span class="m_-1279333881495980509s2">CHKERRQ</span>(ierr);<u></u><u></u></p>
<p class="m_-1279333881495980509p1"><span class="m_-1279333881495980509apple-converted-space"> </span>ierr = <span class="m_-1279333881495980509s1">
PetscOptionsSetValue</span>(<span class="m_-1279333881495980509s3"><b>NULL</b></span>,<span class="m_-1279333881495980509s4">"-pc_svd_monitor"</span>,
<span class="m_-1279333881495980509s3"><b>NULL</b></span>); <span class="m_-1279333881495980509s2">CHKERRQ</span>(ierr); <span style="font-family:Wingdings">
</span>-- is this right?<u></u><u></u></p>
<p class="m_-1279333881495980509p1"><span class="m_-1279333881495980509apple-converted-space"> </span>ierr = <span class="m_-1279333881495980509s1">
PCFactorSetMatSolverType</span>(pc,<span class="m_-1279333881495980509s2">MATSOLVERMUMPS</span>);<span class="m_-1279333881495980509s2">CHKERRQ</span>(ierr);<u></u><u></u></p>
<p class="m_-1279333881495980509p2"><span class="m_-1279333881495980509apple-converted-space"> </span><span class="m_-1279333881495980509s5">ierr =
</span><span class="m_-1279333881495980509s1">KSPView</span><span class="m_-1279333881495980509s5">(ksp, </span>PETSC_VIEWER_STDOUT_SELF<span class="m_-1279333881495980509s5">);</span>CHKERRQ<span class="m_-1279333881495980509s5">(ierr);<u></u><u></u></span></p>
<p class="m_-1279333881495980509p2"><span class="m_-1279333881495980509s5"><u></u> <u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9pt;font-family:Menlo;color:black"> </span><span style="font-size:9pt;font-family:Menlo;color:rgb(108,54,169)">KSPSolve</span><span style="font-size:9pt;font-family:Menlo;color:black">(ksp,
b, x);<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9pt;font-family:Menlo;color:black"> <u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9pt;font-family:Menlo;color:black"> ierr =
</span><span style="font-size:9pt;font-family:Menlo;color:rgb(108,54,169)">KSPMonitorTrueResidual</span><span style="font-size:9pt;font-family:Menlo;color:black">(ksp,its,norm,vf);</span><span style="font-size:9pt;font-family:Menlo;color:rgb(100,56,32)">CHKERRQ</span><span style="font-size:9pt;font-family:Menlo;color:black">(ierr);<u></u><u></u></span></p>
<p class="MsoNormal" style="background:white"><span style="font-size:9pt;font-family:Menlo;color:black"> ierr =
</span><span style="font-size:9pt;font-family:Menlo;color:rgb(108,54,169)">KSPMonitorSingularValue</span><span style="font-size:9pt;font-family:Menlo;color:black">(ksp, its, norm, vf);</span><span style="font-size:9pt;font-family:Menlo;color:rgb(100,56,32)">CHKERRQ</span><span style="font-size:9pt;font-family:Menlo;color:black">(ierr);<u></u><u></u></span></p>
<p class="m_-1279333881495980509p2"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">Yes, now using PCSVD both the serial and parallel version produce the same result.
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<p class="m_-1279333881495980509MsoListParagraph" style="margin-left:54pt">
<u></u><span style="font-size:11pt"><span>(i)<span style="font:7pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11pt">What does this imply?</span></p></div></div></div></blockquote><div><br></div><div>Your matrix is singular</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg-1279333881495980509"><div lang="EN-GB" style="overflow-wrap: break-word;"><div class="m_-1279333881495980509WordSection1"><p class="m_-1279333881495980509MsoListParagraph" style="margin-left:54pt"><span style="font-size:11pt">
<u></u><u></u></span></p>
<p class="m_-1279333881495980509MsoListParagraph" style="margin-left:54pt">
<u></u><span style="font-size:11pt"><span>(ii)<span style="font:7pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11pt">Would I be able to solve using CG preconditioned using Hypre as I scale the problem?</span></p></div></div></div></blockquote><div><br></div><div>Only if Hypre uses SVD on the coarse grid. I am not sure they can do that.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg-1279333881495980509"><div lang="EN-GB" style="overflow-wrap: break-word;"><div class="m_-1279333881495980509WordSection1"><p class="m_-1279333881495980509MsoListParagraph" style="margin-left:54pt"><span style="font-size:11pt"><u></u><u></u></span></p>
<p class="m_-1279333881495980509MsoListParagraph" style="margin-left:54pt">
<u></u><span style="font-size:11pt"><span>(iii)<span style="font:7pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11pt">I have not built PETSc with SLEPc – can I still use PCSVD?</span></p></div></div></div></blockquote><div><br></div><div>Yes.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg-1279333881495980509"><div lang="EN-GB" style="overflow-wrap: break-word;"><div class="m_-1279333881495980509WordSection1"><p class="m_-1279333881495980509MsoListParagraph" style="margin-left:54pt"><span style="font-size:11pt"><u></u><u></u></span></p>
<p class="m_-1279333881495980509MsoListParagraph" style="margin-left:54pt">
<u></u><span style="font-size:11pt"><span>(iv)<span style="font:7pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11pt">Can I set ksp type, pc type, ksp monitor etc using PETScOptionsSetValue instead of code? In that case how would the above code translate to? That will be very helpful.</span></p></div></div></div></blockquote><div><br></div><div>Yes, you can use SetValue(). I do not understand the rest of the question.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg-1279333881495980509"><div lang="EN-GB" style="overflow-wrap: break-word;"><div class="m_-1279333881495980509WordSection1">
<p class="MsoNormal"><span style="font-size:11pt">Many thanks.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">Best,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">Karthik.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0cm 0cm">
<p class="MsoNormal" style="margin-bottom:12pt"><b><span style="font-size:12pt;color:black">From:
</span></b><span style="font-size:12pt;color:black">Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br>
<b>Date: </b>Thursday, 17 November 2022 at 17:48<br>
<b>To: </b>Chockalingam, Karthikeyan (STFC,DL,HC) <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>><br>
<b>Cc: </b>Zhang, Hong <<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>>, <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject: </b>Re: [petsc-users] Different solution while running in parallel<u></u><u></u></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11pt">On Thu, Nov 17, 2022 at 11:13 AM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>> wrote:<u></u><u></u></span></p>
</div>
<div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11pt">Hi Matt,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">I tested two sizes manually for the Poisson problem with homogenous Dirichlet boundary conditions (on all nodes on the boundary) and they both produced
the right result when run serially using PCLU<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
<ol start="1" type="1">
<li class="m_-1279333881495980509m-7699120777846829930msolistparagraph">
2 elements x 2 elements (total nodes 9 but 1 dof)<u></u><u></u></li></ol>
<p class="m_-1279333881495980509m-7699120777846829930msolistparagraph" style="margin-left:72pt">A = 10.6667 b = 4 x = 0.375<u></u><u></u></p>
<ol start="2" type="1">
<li class="m_-1279333881495980509m-7699120777846829930msolistparagraph">
3 elements x 3 elements (total nodes 16 but 4 dof)<u></u><u></u></li></ol>
<p class="MsoNormal" style="margin-left:36pt">
<span style="font-size:11pt"> A = 10.6667 -1.33333 -1.33333 -1.33333<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt">
<span style="font-size:11pt"> -1.33333 10.6667 -1.33333 -1.33333<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt">
<span style="font-size:11pt"> -1.33333 -1.33333 10.6667 -1.33333<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt">
<span style="font-size:11pt"> -1.33333 -1.33333 -1.33333 10.6667<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt">
<span style="font-size:11pt"> <u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt">
<span style="font-size:11pt"> b = {4 4 4 4}^T<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt">
<span style="font-size:11pt"> x = (0.6 0.6 0.6 0.6)<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-left:36pt">
<span style="font-size:11pt"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> Since, it is solvable not sure if the system can be singular?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">I have attached the runs for case (2) run on one and two cores. Parallel run produces zero vector for x.
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">I used MatZeroRowsColumns to set the Dirichlet boundary conditions by zeroing the entries in the matrix corresponding to the boundary nodes.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt">Please please please run the original thing with the options I suggested:<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"> -pc_type svd -pc_svd_monitor<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt">This will print out all the singular values of the matrix and solve it using SVD.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"> Thanks,<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"> Matt<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11pt">Best,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">Karthik.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
<p class="m_-1279333881495980509m-7699120777846829930msolistparagraph"> <u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0cm 0cm">
<p class="MsoNormal" style="margin-bottom:12pt"><b><span style="font-size:12pt;color:black">From:
</span></b><span style="font-size:12pt;color:black">Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br>
<b>Date: </b>Thursday, 17 November 2022 at 15:16<br>
<b>To: </b>Chockalingam, Karthikeyan (STFC,DL,HC) <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>><br>
<b>Cc: </b>Zhang, Hong <<a href="mailto:hzhang@mcs.anl.gov" target="_blank">hzhang@mcs.anl.gov</a>>,
<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject: </b>Re: [petsc-users] Different solution while running in parallel</span><span style="font-size:11pt"><u></u><u></u></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11pt">Using options instead of code will make your life much easier.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11pt">Two thing are wrong here:<u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"> 1) Your solver is doing no iterates because the initial residual is very small, 5.493080158227e-15. The LU does not matter.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"> In order to check the condition number of your system, run with -pc_type svd -pc_svd_monitor<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"> 2) Your parallel run also does no iterates<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"> 0 KSP none resid norm 6.951601853367e-310 true resid norm 1.058300524426e+01 ||r(i)||/||b|| 8.819171036882e-01<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt">but the true residual is not small. That means that your system is singular, but you have given a consistent RHS.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"> Thanks,<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"> Matt<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11pt">On Thu, Nov 17, 2022 at 9:37 AM Karthikeyan Chockalingam - STFC UKRI <<a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank">karthikeyan.chockalingam@stfc.ac.uk</a>>
wrote:<u></u><u></u></span></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt">
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11pt">Hi Matt and Hong,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">Thank you for your response.
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">I made the following changes, to get the desired output
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
<p class="m_-1279333881495980509m-7699120777846829930m-6088207111319744988p1"><span class="m_-1279333881495980509m-7699120777846829930m-6088207111319744988apple-converted-space">
</span><span class="m_-1279333881495980509m-7699120777846829930m-6088207111319744988s2">PetscReal</span><span class="m_-1279333881495980509m-7699120777846829930m-6088207111319744988s1"> norm;
</span>/* norm of solution error */<u></u><u></u></p>
<p class="m_-1279333881495980509m-7699120777846829930m-6088207111319744988p2"><span class="m_-1279333881495980509m-7699120777846829930m-6088207111319744988apple-converted-space">
</span><span class="m_-1279333881495980509m-7699120777846829930m-6088207111319744988s2">PetscInt</span><span class="m_-1279333881495980509m-7699120777846829930m-6088207111319744988apple-converted-space">
</span>its;<u></u><u></u></p>
<p class="m_-1279333881495980509m-7699120777846829930m-6088207111319744988p3"><span class="m_-1279333881495980509m-7699120777846829930m-6088207111319744988apple-converted-space">
</span>KSPConvergedReason<span class="m_-1279333881495980509m-7699120777846829930m-6088207111319744988s1"> reason;</span><u></u><u></u></p>
<p class="m_-1279333881495980509m-7699120777846829930m-6088207111319744988p3"><span class="m_-1279333881495980509m-7699120777846829930m-6088207111319744988apple-converted-space">
</span>PetscViewerAndFormat<span class="m_-1279333881495980509m-7699120777846829930m-6088207111319744988s1"> *vf;</span><u></u><u></u></p>
<p class="m_-1279333881495980509m-7699120777846829930m-6088207111319744988p4"><span class="m_-1279333881495980509m-7699120777846829930m-6088207111319744988apple-converted-space">
</span>PetscViewerAndFormatCreate<span class="m_-1279333881495980509m-7699120777846829930m-6088207111319744988s1">(</span><span class="m_-1279333881495980509m-7699120777846829930m-6088207111319744988s3">PETSC_VIEWER_STDOUT_WORLD</span><span class="m_-1279333881495980509m-7699120777846829930m-6088207111319744988s1">,
</span>PETSC_VIEWER_DEFAULT<span class="m_-1279333881495980509m-7699120777846829930m-6088207111319744988s1">, &vf);</span><u></u><u></u></p>
<p class="MsoNormal" style="background:white">
<span style="font-size:9pt;font-family:Menlo;color:rgb(100,56,32)"> </span><span style="font-size:11pt"><u></u><u></u></span></p>
<p class="MsoNormal" style="background:white">
<span style="font-size:9pt;font-family:Menlo;color:rgb(100,56,32)"> ierr = </span><span style="font-size:9pt;font-family:Menlo;color:rgb(108,54,169)">KSPView</span><span style="font-size:9pt;font-family:Menlo;color:rgb(100,56,32)">(ksp, PETSC_VIEWER_STDOUT_SELF);CHKERRQ(ierr);</span><span style="font-size:11pt"><u></u><u></u></span></p>
<p class="MsoNormal" style="background:white">
<span style="font-size:9pt;font-family:Menlo;color:black"> </span><span style="font-size:11pt"><u></u><u></u></span></p>
<p class="MsoNormal" style="background:white">
<span style="font-size:9pt;font-family:Menlo;color:black"> </span><span style="font-size:9pt;font-family:Menlo;color:rgb(108,54,169)">KSPSolve</span><span style="font-size:9pt;font-family:Menlo;color:black">(ksp, b, x);</span><span style="font-size:11pt"><u></u><u></u></span></p>
<p class="MsoNormal" style="background:white">
<span style="font-size:9pt;font-family:Menlo;color:black"> </span><span style="font-size:11pt"><u></u><u></u></span></p>
<p class="MsoNormal" style="background:white">
<span style="font-size:9pt;font-family:Menlo;color:black"> ierr = </span><span style="font-size:9pt;font-family:Menlo;color:rgb(108,54,169)">KSPMonitorTrueResidual</span><span style="font-size:9pt;font-family:Menlo;color:black">(ksp,its,norm,vf);</span><span style="font-size:9pt;font-family:Menlo;color:rgb(100,56,32)">CHKERRQ</span><span style="font-size:9pt;font-family:Menlo;color:black">(ierr);</span><span style="font-size:11pt"><u></u><u></u></span></p>
<p class="MsoNormal" style="background:white">
<span style="font-size:9pt;font-family:Menlo;color:black"> ierr = </span><span style="font-size:9pt;font-family:Menlo;color:rgb(108,54,169)">KSPMonitorSingularValue</span><span style="font-size:9pt;font-family:Menlo;color:black">(ksp, its, norm, vf);</span><span style="font-size:9pt;font-family:Menlo;color:rgb(100,56,32)">CHKERRQ</span><span style="font-size:9pt;font-family:Menlo;color:black">(ierr);</span><span style="font-size:11pt"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">I have attached the outputs from both the runs. As before, I am also printing A, b, and x.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">I wonder if it is a memory issue related to mpi library employed. I am currently using openmpi – should I instead use mpich?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">Kind regards,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt">Karthik.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0cm 0cm">
<p class="MsoNormal" style="margin-bottom:12pt"><b><span style="font-size:12pt;color:black">From:
</span></b><span style="font-size:12pt;color:black">Matthew Knepley <</span><span style="font-size:11pt"><a href="mailto:knepley@gmail.com" target="_blank"><span style="font-size:12pt">knepley@gmail.com</span></a></span><span style="font-size:12pt;color:black">><br>
<b>Date: </b>Thursday, 17 November 2022 at 12:19<br>
<b>To: </b>Zhang, Hong <</span><span style="font-size:11pt"><a href="mailto:hzhang@mcs.anl.gov" target="_blank"><span style="font-size:12pt">hzhang@mcs.anl.gov</span></a></span><span style="font-size:12pt;color:black">><br>
<b>Cc: </b></span><span style="font-size:11pt"><a href="mailto:petsc-users@mcs.anl.gov" target="_blank"><span style="font-size:12pt">petsc-users@mcs.anl.gov</span></a></span><span style="font-size:12pt;color:black"> <</span><span style="font-size:11pt"><a href="mailto:petsc-users@mcs.anl.gov" target="_blank"><span style="font-size:12pt">petsc-users@mcs.anl.gov</span></a></span><span style="font-size:12pt;color:black">>,
Chockalingam, Karthikeyan (STFC,DL,HC) <</span><span style="font-size:11pt"><a href="mailto:karthikeyan.chockalingam@stfc.ac.uk" target="_blank"><span style="font-size:12pt">karthikeyan.chockalingam@stfc.ac.uk</span></a></span><span style="font-size:12pt;color:black">><br>
<b>Subject: </b>Re: [petsc-users] Different solution while running in parallel</span><span style="font-size:11pt"><u></u><u></u></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11pt">On Wed, Nov 16, 2022 at 9:07 PM Zhang, Hong via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>>
wrote:<u></u><u></u></span></p>
</div>
<div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt">
<div>
<div>
<div>
<p class="MsoNormal" style="background:white">
<span style="font-size:9pt;font-family:Menlo;color:black;background:white">Karhik,</span><span style="font-size:11pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white">
<span style="font-size:9pt;font-family:Menlo;color:black;background:white">Can you find out the condition number of your matrix?</span><span style="font-size:11pt"><u></u><u></u></span></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt">Also, run using<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"> -ksp_view -ksp_monitor_true_residual -ksp_converged_reason<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt">and send the two outputs.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"> Thanks,<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"> Matt<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt">
<div>
<div>
<div>
<p class="MsoNormal" style="background:white">
<span style="font-size:9pt;font-family:Menlo;color:black;background:white">Hong</span><span style="font-size:11pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12pt;color:black"> </span><span style="font-size:11pt"><u></u><u></u></span></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-size:11pt">
<hr size="0" width="100%" align="center">
</span></div>
<div id="m_-1279333881495980509m_-7699120777846829930m_-6088207111319744988m_-7607706715142363214divRplyFwdMsg">
<p class="MsoNormal"><b><span style="font-size:11pt;color:black">From:</span></b><span style="font-size:11pt;color:black"> petsc-users <</span><span style="font-size:11pt"><a href="mailto:petsc-users-bounces@mcs.anl.gov" target="_blank">petsc-users-bounces@mcs.anl.gov</a><span style="color:black">>
on behalf of Karthikeyan Chockalingam - STFC UKRI via petsc-users <</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a><span style="color:black">><br>
<b>Sent:</b> Wednesday, November 16, 2022 6:04 PM<br>
<b>To:</b> </span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a><span style="color:black"> <</span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a><span style="color:black">><br>
<b>Subject:</b> [petsc-users] Different solution while running in parallel</span>
<u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
</div>
</div>
<div>
<div>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:black"> Hello,</span><u></u><u></u></p>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:black"> </span><u></u><u></u></p>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:black"> I tried to solve a (FE discretized) Poisson equation using PCLU. For some reason I am getting different solutions while running the problem on one and two
cores. I have attached the output file (out.txt) from both the runs. I am printing A, b and x from both the runs – while A and b are the same but the solution seems is different.</span><u></u><u></u></p>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:black"> </span><u></u><u></u></p>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:black">I am not sure what I doing wrong.</span><u></u><u></u></p>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:black"> </span><u></u><u></u></p>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:black">Below is my matrix, vector, and solve setup.
</span><u></u><u></u></p>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:black"> </span><u></u><u></u></p>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:black"> </span><u></u><u></u></p>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:black"> </span><span style="font-size:9pt;font-family:Menlo;color:rgb(57,0,160)">Mat</span><span style="font-size:9pt;font-family:Menlo;color:black"> A;</span><u></u><u></u></p>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:black">
</span><span style="font-size:9pt;font-family:Menlo;color:rgb(57,0,160)">Vec</span><span style="font-size:9pt;font-family:Menlo;color:black"> b, x;</span><u></u><u></u></p>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:black"> </span><u></u><u></u></p>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:black"> ierr =
</span><span style="font-size:9pt;font-family:Menlo;color:rgb(108,54,169)">MatCreate</span><span style="font-size:9pt;font-family:Menlo;color:black">(</span><span style="font-size:9pt;font-family:Menlo;color:rgb(108,54,169)">PETSC_COMM_WORLD</span><span style="font-size:9pt;font-family:Menlo;color:black">,
&A); </span><span style="font-size:9pt;font-family:Menlo;color:rgb(100,56,32)">CHKERRQ</span><span style="font-size:9pt;font-family:Menlo;color:black">(ierr);</span><u></u><u></u></p>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:black"> ierr =
</span><span style="font-size:9pt;font-family:Menlo;color:rgb(108,54,169)">MatSetType</span><span style="font-size:9pt;font-family:Menlo;color:black">(A,
</span><span style="font-size:9pt;font-family:Menlo;color:rgb(100,56,32)">MATMPIAIJ</span><span style="font-size:9pt;font-family:Menlo;color:black">);
</span><span style="font-size:9pt;font-family:Menlo;color:rgb(100,56,32)">CHKERRQ</span><span style="font-size:9pt;font-family:Menlo;color:black">(ierr);</span><u></u><u></u></p>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:black"> ierr =
</span><span style="font-size:9pt;font-family:Menlo;color:rgb(108,54,169)">MatSetSizes</span><span style="font-size:9pt;font-family:Menlo;color:black">(A,
</span><span style="font-size:9pt;font-family:Menlo;color:rgb(100,56,32)">PETSC_DECIDE</span><span style="font-size:9pt;font-family:Menlo;color:black">,
</span><span style="font-size:9pt;font-family:Menlo;color:rgb(100,56,32)">PETSC_DECIDE</span><span style="font-size:9pt;font-family:Menlo;color:black">, N, N);
</span><span style="font-size:9pt;font-family:Menlo;color:rgb(100,56,32)">CHKERRQ</span><span style="font-size:9pt;font-family:Menlo;color:black">(ierr);</span><u></u><u></u></p>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:black"> ierr =
</span><span style="font-size:9pt;font-family:Menlo;color:rgb(108,54,169)">MatMPIAIJSetPreallocation</span><span style="font-size:9pt;font-family:Menlo;color:black">(A,d_nz,
</span><b><span style="font-size:9pt;font-family:Menlo;color:rgb(155,35,147)">NULL</span></b><span style="font-size:9pt;font-family:Menlo;color:black">, o_nz,
</span><b><span style="font-size:9pt;font-family:Menlo;color:rgb(155,35,147)">NULL</span></b><span style="font-size:9pt;font-family:Menlo;color:black">);
</span><span style="font-size:9pt;font-family:Menlo;color:rgb(100,56,32)">CHKERRQ</span><span style="font-size:9pt;font-family:Menlo;color:black">(ierr);</span><u></u><u></u></p>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:rgb(108,54,169)"> ierr = MatSetOption(A,MAT_SYMMETRIC,PETSC_TRUE);
</span><span style="font-size:9pt;font-family:Menlo;color:rgb(100,56,32)">CHKERRQ</span><span style="font-size:9pt;font-family:Menlo;color:rgb(108,54,169)">(ierr);</span><u></u><u></u></p>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:black"> ierr =
</span><span style="font-size:9pt;font-family:Menlo;color:rgb(108,54,169)">MatCreateVecs</span><span style="font-size:9pt;font-family:Menlo;color:black">(A, &b, &x);
</span><span style="font-size:9pt;font-family:Menlo;color:rgb(100,56,32)">CHKERRQ</span><span style="font-size:9pt;font-family:Menlo;color:black">(ierr);</span><u></u><u></u></p>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:black"> </span><u></u><u></u></p>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:black">
</span><span style="font-size:9pt;font-family:Menlo;color:rgb(57,0,160)">KSP</span><span style="font-size:9pt;font-family:Menlo;color:black"> ksp;</span><u></u><u></u></p>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:black">
</span><span style="font-size:9pt;font-family:Menlo;color:rgb(57,0,160)">PC</span><span style="font-size:9pt;font-family:Menlo;color:black"> pc;</span><u></u><u></u></p>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:rgb(108,54,169)"> KSPCreate(PETSC_COMM_WORLD, &ksp);</span><u></u><u></u></p>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:black">
</span><span style="font-size:9pt;font-family:Menlo;color:rgb(108,54,169)">KSPSetOperators</span><span style="font-size:9pt;font-family:Menlo;color:black">(ksp, A, A);</span><u></u><u></u></p>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:black"> ierr =
</span><span style="font-size:9pt;font-family:Menlo;color:rgb(108,54,169)">KSPSetType</span><span style="font-size:9pt;font-family:Menlo;color:black">(ksp,</span><span style="font-size:9pt;font-family:Menlo;color:rgb(100,56,32)">KSPPREONLY</span><span style="font-size:9pt;font-family:Menlo;color:black">);</span><span style="font-size:9pt;font-family:Menlo;color:rgb(100,56,32)">CHKERRQ</span><span style="font-size:9pt;font-family:Menlo;color:black">(ierr);</span><u></u><u></u></p>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:black"> ierr =
</span><span style="font-size:9pt;font-family:Menlo;color:rgb(108,54,169)">KSPGetPC</span><span style="font-size:9pt;font-family:Menlo;color:black">(ksp,&pc);</span><span style="font-size:9pt;font-family:Menlo;color:rgb(100,56,32)">CHKERRQ</span><span style="font-size:9pt;font-family:Menlo;color:black">(ierr);</span><u></u><u></u></p>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:black"> ierr =
</span><span style="font-size:9pt;font-family:Menlo;color:rgb(108,54,169)">PCSetType</span><span style="font-size:9pt;font-family:Menlo;color:black">(pc,</span><span style="font-size:9pt;font-family:Menlo;color:rgb(100,56,32)">PCLU</span><span style="font-size:9pt;font-family:Menlo;color:black">);</span><span style="font-size:9pt;font-family:Menlo;color:rgb(100,56,32)">CHKERRQ</span><span style="font-size:9pt;font-family:Menlo;color:black">(ierr);</span><u></u><u></u></p>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:rgb(108,54,169)"> ierr = PCFactorSetMatSolverType(pc,</span><span style="font-size:9pt;font-family:Menlo;color:rgb(100,56,32)">MATSOLVERMUMPS</span><span style="font-size:9pt;font-family:Menlo;color:rgb(108,54,169)">);</span><span style="font-size:9pt;font-family:Menlo;color:rgb(100,56,32)">CHKERRQ</span><span style="font-size:9pt;font-family:Menlo;color:rgb(108,54,169)">(ierr);</span><u></u><u></u></p>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:black">
</span><span style="font-size:9pt;font-family:Menlo;color:rgb(108,54,169)">KSPSolve</span><span style="font-size:9pt;font-family:Menlo;color:black">(ksp, b, x);</span><u></u><u></u></p>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:black"> </span><u></u><u></u></p>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:black">Thank you for your help.</span><u></u><u></u></p>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:black"> </span><u></u><u></u></p>
<p style="margin:0cm;background:white"><span style="font-size:9pt;font-family:Menlo;color:black">Karhik.</span><u></u><u></u></p>
<p style="margin:0cm"> <u></u><u></u></p>
</div>
<p><span style="font-size:6pt">This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify
the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own
virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. </span><u></u><u></u></p>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span style="font-size:11pt"><br clear="all">
<u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11pt">--
<u></u><u></u></span></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11pt">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments
lead.<br>
-- Norbert Wiener<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><u></u><u></u></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span style="font-size:11pt"><br clear="all">
<u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11pt">--
<u></u><u></u></span></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11pt">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments
lead.<br>
-- Norbert Wiener<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><u></u><u></u></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span style="font-size:11pt"><br clear="all">
<u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11pt">-- <u></u><u></u></span></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11pt">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt"><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><u></u><u></u></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>