<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBannerihx6zai { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtonihx6zai:hover, .pfptPrimaryButtonihx6zai:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButtonihx6zai:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
</head><!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
On Sun, Jun 2, 2024 at 10: 27 AM Matthew Knepley <knepley@ gmail. com> wrote: On Sat, Jun 1, 2024 at 11: 39 PM Carpenter, Mark H. (LARC-D302) via petsc-users <petsc-users@ mcs. anl. gov> wrote: Mark Carpenter, NASA Langley. I am a novice
</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBannerihx6zai" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBannerihx6zai" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBannerihx6zai" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBannerihx6zai" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<div dir="ltr"><div dir="ltr">On Sun, Jun 2, 2024 at 10:27 AM Matthew Knepley <<a href="mailto:knepley@gmail.com">knepley@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Sat, Jun 1, 2024 at 11:39 PM Carpenter, Mark H. (LARC-D302) via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="font-size:1px;color:rgb(255,255,255);line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;display:none">
Mark Carpenter, NASA Langley. I am a novice PETSC user of about 10 years. I’ve build a DG-FEM code with petsc as one of the solver paths (I have my own as well). Furthermore, I use petsc for MPI communication. I’m running the DG-FEM
</div>
<div style="font-size:1px;color:rgb(255,255,255);line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;display:none">ZjQcmQRYFpfptBannerStart</div>
<u></u>
<div dir="ltr" id="m_-6766692149393245880m_3021406515665623195pfptBanner3ppl1oc" style="display:block;text-align:left;margin:16px 0px;padding:8px 16px;border-radius:4px;min-width:200px;background-color:rgb(208,216,220);border-top:4px solid rgb(144,164,174)">
<div id="m_-6766692149393245880m_3021406515665623195pfptBanner3ppl1oc" style="float:left;display:block;margin:0px 0px 1px;max-width:600px">
<div id="m_-6766692149393245880m_3021406515665623195pfptBanner3ppl1oc" style="display:block;background-color:rgb(208,216,220);color:rgb(0,0,0);font-family:Arial,sans-serif;font-weight:bold;font-size:14px;line-height:18px">
This Message Is From an External Sender
</div>
<div id="m_-6766692149393245880m_3021406515665623195pfptBanner3ppl1oc" style="font-weight:normal;display:block;background-color:rgb(208,216,220);color:rgb(0,0,0);font-family:Arial,sans-serif;font-size:12px;line-height:18px;margin-top:2px">
This message came from outside your organization.
</div>
</div>
<div style="height:0px;clear:both;display:block;line-height:0;font-size:0.01px"> </div>
</div>
<u></u>
<div style="font-size:1px;color:rgb(255,255,255);line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;display:none">ZjQcmQRYFpfptBannerEnd</div>
<div lang="EN-US">
<div>
<p class="MsoNormal">Mark Carpenter, NASA Langley.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I am a novice PETSC user of about 10 years. I’ve build a DG-FEM code with petsc as one of the solver paths (I have my own as well). Furthermore, I use petsc for MPI communication.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I’m running the DG-FEM code on our NAS supercomputer. Everything works when my integer sizes are small. When I exceed the 2^32 limit of integer arithmetic the code fails in very strange ways.<u></u><u></u></p>
<p class="MsoNormal">The users that originally set up the petsc infrastructure in the code are no longer at NASA and I’m “dead in the water”.</p></div></div></blockquote></div></div></blockquote><div><br></div><div>One additional point. I have looked at the error message. When you make PETSc calls, each call should be wrapped in PetscCall(). Here is a Fortran example:</div><div><br></div><div> <a href="https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/ksp/tutorials/ex22f.F90?ref_type=heads__;!!G_uCfscf7eWS!eOkbaTOpui-YHhrX_HYLmYerXOaaGtlJn04-tdLvQzfRqa6gaCs2x-YtPn7xNTWzRRgD-wze7GkX5hkXqc8i$">https://gitlab.com/petsc/petsc/-/blob/main/src/ksp/ksp/tutorials/ex22f.F90?ref_type=heads</a></div><div><br></div><div>This checks the return value after each call and ends early if there is an error. It would make your</div><div>error output much more readable.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div lang="EN-US"><div><p class="MsoNormal"> </p>
<p class="MsoNormal">I think I’ve promoted all the integers that are problematic in my code (F95). On PETSC side: I’ve tried<u></u><u></u></p>
<ol style="margin-top:0in" start="1" type="1">
<li style="margin-left:0in">Reinstall petsc with –with-64-bit-integers (no luck)<br></li></ol></div></div></blockquote><div><br></div><div>That option does not exist, so this will not work.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div lang="EN-US"><div><ol style="margin-top:0in" start="1" type="1"><li style="margin-left:0in"><u></u><u></u></li><li style="margin-left:0in">Reinstall petsc with –with-64-bit-integers and –with-64-bit-indices (code will not compile with these options. Additional variables on F90 side require promotion and then the errors
cascade through code when making PETSC calls.</li></ol></div></div></div></blockquote><div><br></div><div>We should fix this. I feel confident we can get the code to compile.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div lang="EN-US"><div><ol style="margin-top:0in" start="1" type="1"><li style="margin-left:0in"><u></u><u></u></li><li style="margin-left:0in">It’s possible that I’ve missed offending integers, but the petsc error messages are so cryptic that I can’t even tell where it is failing.<u></u><u></u></li></ol>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Further complicating matters: <u></u><u></u></p>
<p class="MsoNormal">The problem by definition needs to be HUGE. Problem sizes requiring 1000 cores (10^6 elements at P5) are needed to experience the errors, which involves waiting in queues for ½ day at least.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Attached are the <u></u><u></u></p>
<ol style="margin-top:0in" start="1" type="1">
<li style="margin-left:0in">Install script used to install PETSC on our machine<u></u><u></u></li><li style="margin-left:0in">The Makefile used on the fortran side<u></u><u></u></li><li style="margin-left:0in">A data dump from an offending simulation (which is huge and I can’t see any useful information.)<u></u><u></u></li></ol>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">How do I attack this problem.<u></u><u></u></p>
<p class="MsoNormal">(I’ve never gotten debugging working properly).</p></div></div></div></blockquote><div><br></div><div>Let's get the install for 64-bit indices to work. So we</div><div><br></div><div>1) Configure PETSc adding --with-64bit-indices to the configure line. Does this work? If not, send configure.log</div><div><br></div><div>2) Compile PETSc. Does this work? If not, send make.log</div><div><br></div><div>3) Compile your code. Does this work? If not, send all output.</div><div><br></div><div>4) Do one of the 1/2 day runs and let us know what happens. An alternative is to run a small number</div><div> of processes on a large memory workstation. We do this to test at the lab.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div lang="EN-US"><div>
<p class="MsoNormal">Mark<u></u><u></u></p>
</div>
</div>
</div></blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eOkbaTOpui-YHhrX_HYLmYerXOaaGtlJn04-tdLvQzfRqa6gaCs2x-YtPn7xNTWzRRgD-wze7GkX5r8KJKDw$" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!eOkbaTOpui-YHhrX_HYLmYerXOaaGtlJn04-tdLvQzfRqa6gaCs2x-YtPn7xNTWzRRgD-wze7GkX5r8KJKDw$" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>