<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBannerz5vgegv { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtonz5vgegv:hover, .pfptPrimaryButtonz5vgegv:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButtonz5vgegv:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
</head><!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
Yikes, it looks like we have been off the list this whole time. I am not the only PETSC developer nor the only person that knows about PETSc! These folks have some strange behavior with GAMG going from 1 to 2 cores, using lots of memory, but
</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBannerz5vgegv" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBannerz5vgegv" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBannerz5vgegv" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBannerz5vgegv" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<div dir="ltr"><div dir="ltr">Yikes, it looks like we have been off the list this whole time.<div>I am not the only PETSC developer nor the only person that knows about PETSc!</div><div><br></div><div>These folks have some strange behavior with GAMG going from 1 to 2 cores, using lots of memory, but one question that they have, that I don't understand either is this:</div><div><br></div><div><div>>> Yea, my interpretation of these methods is also that "<span style="color:rgb(0,0,0);font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:16px">PetscMemoryGetMaximumUsage</span>" should be >= "<span style="color:rgb(0,0,0);font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:16px">PetscMallocGetMaximumUsage</span>". </div><div>>> But you are seeing the opposite.</div><br class="gmail-Apple-interchange-newline"></div><div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div dir="ltr"><div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><br class="gmail-Apple-interchange-newline">We are using PETSc main and have found a case where memory consumption explodes in parallel.</div><div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;color:rgb(0,0,0)"><span style="font-size:12pt">Also, we see a non-negligible difference between </span><span style="font-size:16px">PetscMemoryGetMaximumUsage</span><span style="font-size:12pt">() and </span><span style="font-size:16px">PetscMallocGetMaximumUsage</span><span style="font-size:12pt">().</span></div><div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;color:rgb(0,0,0)"><span style="font-size:12pt">Running in serial through /usr/bin/time, the max. resident set size matches the </span><span style="font-size:16px">PetscMallocGetMaximumUsage</span><span style="font-size:12pt">() result.</span></div><div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;color:rgb(0,0,0)"><span style="font-size:12pt">I would have expected it to match </span><span style="font-size:16px">PetscMemoryGetMaximumUsage</span><span style="font-size:12pt">() instead.</span></div></div></div></blockquote><br class="gmail-Apple-interchange-newline"></div><div><br class="gmail-Apple-interchange-newline"><table id="m_8494545079624546237m_-7618095157115570734x_m_8419638774470603419x_table_0" style="direction:ltr;height:282px;border-collapse:collapse;border-spacing:0px"><tbody><tr><td style="width:71.4688px;height:23px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box"><div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt"><br></div></td><td style="width:233.844px;height:23px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box"><div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:16px;color:rgb(0,0,0)">PetscMemoryGetMaximumUsage</div></td><td style="width:224px;height:23px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box"><div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:16px;color:rgb(0,0,0)">PetscMallocGetMaximumUsage</div></td><td style="width:71px;height:23px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;word-break:break-word;box-sizing:border-box"><div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> Time</div></td></tr><tr><td style="width:71.4688px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box"><div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Serial + Option 1</div></td><td style="width:233.844px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box"><div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> 4.8 GB</div></td><td style="width:224px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box"><div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> 7.4 GB</div></td><td style="width:71px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;word-break:break-word;box-sizing:border-box"><div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">112 sec</div></td></tr><tr><td style="width:71.4688px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box"><div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:16px;color:rgb(0,0,0)">2 core + Option1</div></td><td style="width:233.844px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box"><div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">15.2 GB</div></td><td style="width:224px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box"><div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">45.5 GB</div></td><td style="width:71px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;word-break:break-word;box-sizing:border-box"><div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">150 sec</div></td></tr><tr><td style="width:71.4688px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box"><div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Serial + Option 2</div></td><td style="width:233.844px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box"><div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">3.1 GB</div></td><td style="width:224px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box"><div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">3.8 GB</div></td><td style="width:71px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;word-break:break-word;box-sizing:border-box"><div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> 167 sec</div></td></tr><tr><td style="width:71.4688px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box"><div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">2 core + Option2</div></td><td style="width:233.844px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box"><div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">13.1 GB</div></td><td style="width:224px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box"><div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">17.4 GB</div></td><td style="width:71px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;word-break:break-word;box-sizing:border-box"><div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">89 sec</div></td></tr><tr><td style="width:71.4688px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box"><div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">Serial + Option 3</div></td><td style="width:233.844px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box"><div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">4.7GB</div></td><td style="width:224px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box"><div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">5.2GB</div></td><td style="width:71px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;word-break:break-word;box-sizing:border-box"><div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">693 sec</div></td></tr><tr><td style="width:71.4688px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box"><div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">2 core + Option 3</div></td><td style="width:233.844px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box"><div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">23.2 GB</div></td><td style="width:224px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box"><div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">26.4 GB</div></td><td style="width:71px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;word-break:break-word;box-sizing:border-box"><div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">411 sec</div></td></tr></tbody></table><br class="gmail-Apple-interchange-newline"></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Apr 18, 2024 at 4:13 PM Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">The next thing you might try is not using the null space argument.<div>Hypre does not use it, but GAMG does.</div><div>You could also run with -malloc_view to see some info on mallocs. It is probably in the Mat objects.</div><div>You can also run with "-info" and grep on GAMG in the output and send that.</div><div><br></div><div>Mark</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Apr 18, 2024 at 12:03 PM Ashish Patel <<a href="mailto:ashish.patel@ansys.com" target="_blank">ashish.patel@ansys.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi Mark,</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Thanks for your response and suggestion. With hypre both memory and time looks good, here is the data for that</div>
<table id="m_8494545079624546237m_-7618095157115570734table_0" style="width:594.5px;box-sizing:border-box;border-collapse:collapse;border-spacing:0px">
<tbody>
<tr>
<td style="width:68.9219px;height:36px;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;word-break:break-word;box-sizing:border-box">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
</td>
<td style="width:234.656px;height:36px;direction:ltr;text-align:left;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;word-break:break-word;box-sizing:border-box">
<div style="direction:ltr;text-align:left;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:16px;color:black">
<span style="background-color:white">PetscMemoryGetMaximumUsage</span></div>
</td>
<td style="width:225.922px;height:36px;direction:ltr;text-align:left;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;word-break:break-word;box-sizing:border-box">
<div style="direction:ltr;text-align:left;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:16px;color:black">
<span style="background-color:white">PetscMallocGetMaximumUsage</span></div>
</td>
<td style="width:64px;height:36px;direction:ltr;text-align:left;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;word-break:break-word;box-sizing:border-box">
<div style="direction:ltr;text-align:left;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:black">
Time</div>
</td>
</tr>
<tr>
<td style="width:68.9219px;height:43px;direction:ltr;text-align:left;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box">
<div style="direction:ltr;text-align:left;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:black">
Serial + Option <span style="color:rgb(0,0,0)">4</span></div>
</td>
<td style="width:234.656px;height:43px;direction:ltr;text-align:left;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;word-break:break-word;box-sizing:border-box">
<div style="direction:ltr;text-align:left;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:black">
5.55 GB</div>
</td>
<td style="width:225.922px;height:43px;direction:ltr;text-align:left;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;word-break:break-word;box-sizing:border-box">
<div style="direction:ltr;text-align:left;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:black">
5.17 GB</div>
</td>
<td style="width:64px;height:43px;direction:ltr;text-align:left;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;word-break:break-word;box-sizing:border-box">
<div style="direction:ltr;text-align:left;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:black">
15.7 sec</div>
</td>
</tr>
<tr>
<td style="width:68.9219px;height:43px;direction:ltr;text-align:left;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box">
<div style="direction:ltr;text-align:left;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:16px;color:black">
<span style="background-color:white">2 core + Option 4</span></div>
</td>
<td style="width:234.656px;height:43px;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;word-break:break-word;box-sizing:border-box">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
5.85 GB</div>
</td>
<td style="width:225.922px;height:43px;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;word-break:break-word;box-sizing:border-box">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
4.69 GB</div>
</td>
<td style="width:64px;height:43px;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;word-break:break-word;box-sizing:border-box">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
21.9 sec</div>
</td>
</tr>
</tbody>
</table>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Option 4</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
mpirun -n _ ./ex1 -A_name matrix.dat -b_name vector.dat -n_name _null_space.dat -num_near_nullspace 6 -ksp_type cg -pc_type hypre -pc_hypre_boomeramg_strong_threshold 0.9 -ksp_view -log_view -log_view_memory -info :pc</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-size:12pt;color:rgb(0,0,0)"><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif">I am also attaching a standalone program to reproduce these options and the link to matrix, rhs and
near null spaces (<span style="display:inline-block"><span><a href="https://urldefense.us/v3/__https://ansys-my.sharepoint.com/:u:/p/ashish_patel/EbUM5Ahp-epNi4xDxR9mnN0B1dceuVzGhVXQQYJzI5Py2g__;!!G_uCfscf7eWS!ar7t_MsQ-W6SXcDyEWpSDZP_YngFSqVsz2D-8chGJHSz7IZzkLBvN4UpJ1GXrRBGyhEHqmDUQGBfqTKf5x_BPXo$" rel="noopener noreferrer" id="m_8494545079624546237m_-7618095157115570734OLK_Beautified_04d9dff3-2630-3fc0-947a-e16ee02d906c" style="padding:0px 1px;border-radius:2px;background-color:rgb(243,242,241)" target="_blank"><img alt="" role="presentation" style="width: 16px; height: 16px; vertical-align: middle; padding: 1px 2px 2px 0px;" src="https://res.cdn.office.net/assets/mail/file-icon/png/generic_16x16.png">serial.tar
2.xz</a></span></span></span><span style="background-color:rgb(243,242,241)">)</span><span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif"> if
you would like to try as well. Please let me know if you have trouble accessing the link.</span></div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Ashish</div>
<div id="m_8494545079624546237m_-7618095157115570734appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="m_8494545079624546237m_-7618095157115570734divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>><br>
<b>Sent:</b> Wednesday, April 17, 2024 7:52 PM<br>
<b>To:</b> Jeremy Theler (External) <<a href="mailto:jeremy.theler-ext@ansys.com" target="_blank">jeremy.theler-ext@ansys.com</a>><br>
<b>Cc:</b> Ashish Patel <<a href="mailto:ashish.patel@ansys.com" target="_blank">ashish.patel@ansys.com</a>>; Scott McClennan <<a href="mailto:scott.mcclennan@ansys.com" target="_blank">scott.mcclennan@ansys.com</a>><br>
<b>Subject:</b> Re: About recent changes in GAMG</font>
<div> </div>
</div>
<div>
<div style="border:1pt solid rgb(156,101,0);padding:2pt">
<p style="line-height:12pt;background:rgb(255,235,156)"><b><span lang="EN-US" style="font-size:10pt;color:red">[External Sender]</span></b></p>
</div>
<div>
<div dir="ltr">
<div dir="ltr"><br>
</div>
<br>
<div>
<div dir="ltr">On Wed, Apr 17, 2024 at 7:20 AM Jeremy Theler (External) <<a href="mailto:jeremy.theler-ext@ansys.com" target="_blank">jeremy.theler-ext@ansys.com</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hey Mark. Long time no see! How are thing going over there?</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
We are using PETSc main and have found a case where memory consumption explodes in parallel.</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;color:rgb(0,0,0)">
<span style="font-size:12pt">Also, we see a non-negligible difference between </span>
<span style="font-size:16px;background-color:rgb(255,255,255)">PetscMemoryGetMaximumUsage</span><span style="font-size:12pt">() and
</span><span style="font-size:16px;background-color:rgb(255,255,255)">PetscMallocGetMaximumUsage</span><span style="font-size:12pt">().</span></div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;color:rgb(0,0,0)">
<span style="font-size:12pt">Running in serial through /usr/bin/time, the max. resident set size matches the
</span><span style="font-size:16px;background-color:rgb(255,255,255)">PetscMallocGetMaximumUsage</span><span style="font-size:12pt">() result.</span></div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;color:rgb(0,0,0)">
<span style="font-size:12pt">I would have expected it to match </span><span style="font-size:16px;background-color:rgb(255,255,255)">PetscMemoryGetMaximumUsage</span><span style="font-size:12pt">() instead.</span></div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Yea, my interpretation of these methods is also that "Memory" should be >= "Malloc". But you are seeing the opposite.</div>
<div><br>
</div>
<div>I don't have any idea what is going on with your big memory penalty going from 1 to 2 cores on this test, but the first thing to do is try other solvers and see how that behaves. Hypre in particular would be a good thing to try because it is a similar
algorithm.</div>
<div><br>
</div>
<div>Mark</div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
The matrix size is around 1 million. We can share it with you if you want, along with the RHS and the 6 near nullspace vectors and a modified ex1.c which will read these files and show the following behavior.</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Observations using latest main for elastic matrix with a block size of 3 (after removing bonded glue-like DOFs with direct elimination) and near null space provided</div>
<div style="margin:0px 0px 0px 7px;min-width:370px">
<div style="margin:0px 0px 8px 8px;min-width:0px">
<div style="margin:2px 2px 8px;padding-right:12px;padding-bottom:12px;padding-left:12px;border-width:1px;border-style:solid;border-color:transparent;border-radius:4px">
<div style="margin:12px 16px 0px 52px;padding-bottom:2px">
<ul style="direction:ltr;margin-top:0px;margin-bottom:0px">
<li style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);list-style-type:"- "">
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Big memory penalty going from serial to parallel (2 core)</div>
</li><li style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);list-style-type:"- "">
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;color:rgb(0,0,0)">
<span style="font-size:12pt">Big difference between </span><span style="font-size:16px;background-color:rgb(255,255,255)">PetscMemoryGetMaximumUsage</span><span style="font-size:12pt"> and
</span><span style="font-size:16px;background-color:rgb(255,255,255)">PetscMallocGetMaximumUsage</span><span style="font-size:12pt">, why?</span></div>
</li><li style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);list-style-type:"- "">
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;color:rgb(0,0,0)">
<span style="font-size:12pt">The memory penalty decreases with -</span><span style="font-size:16px;background-color:rgb(255,255,255)">pc_gamg_aggressive_square_graph</span><span style="font-size:12pt"> false (option 2)</span></div>
</li><li style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);list-style-type:"- "">
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;color:rgb(0,0,0)">
<span style="font-size:12pt">The difference between </span><span style="font-size:16px;background-color:rgb(255,255,255)">PetscMemoryGetMaximumUsage</span><span style="font-size:12pt;background-color:rgb(255,255,255)"> and
</span><span style="font-size:16px;background-color:rgb(255,255,255)">PetscMallocGetMaximumUsage</span><span style="font-size:12pt"> reduces when -</span><span style="font-size:16px;background-color:rgb(255,255,255)">pc_gamg_threshold</span><span style="font-size:12pt"> is
increased from 0 to 0.01 (option 3), the solve time increase a lot though.</span></div>
</li></ul>
<div style="direction:ltr;margin:0px;font-size:15px"><br>
<br>
<br>
</div>
<table id="m_8494545079624546237m_-7618095157115570734x_m_8419638774470603419x_table_0" style="direction:ltr;height:282px;box-sizing:border-box;border-collapse:collapse;border-spacing:0px">
<tbody>
<tr>
<td style="width:71.4688px;height:23px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box">
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt">
<br>
</div>
</td>
<td style="width:233.844px;height:23px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box">
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:16px;color:rgb(0,0,0)">
<span style="background-color:rgb(255,255,255)">PetscMemoryGetMaximumUsage</span></div>
</td>
<td style="width:224px;height:23px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box">
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:16px;color:rgb(0,0,0)">
<span style="background-color:rgb(255,255,255)">PetscMallocGetMaximumUsage</span></div>
</td>
<td style="width:71px;height:23px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;word-break:break-word;box-sizing:border-box">
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Time</div>
</td>
</tr>
<tr>
<td style="width:71.4688px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box">
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Serial + Option 1</div>
</td>
<td style="width:233.844px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box">
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
4.8 GB</div>
</td>
<td style="width:224px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box">
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
7.4 GB</div>
</td>
<td style="width:71px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;word-break:break-word;box-sizing:border-box">
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
112 sec</div>
</td>
</tr>
<tr>
<td style="width:71.4688px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box">
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:16px;color:rgb(0,0,0)">
<span style="background-color:rgb(255,255,255)">2 core + Option1</span></div>
</td>
<td style="width:233.844px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box">
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
15.2 GB</div>
</td>
<td style="width:224px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box">
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
45.5 GB</div>
</td>
<td style="width:71px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;word-break:break-word;box-sizing:border-box">
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
150 sec</div>
</td>
</tr>
<tr>
<td style="width:71.4688px;height:43px;direction:ltr;text-align:left;text-indent:0px;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box">
<div style="direction:ltr;text-align:left;text-indent:0px;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Serial + Option 2</div>
</td>
<td style="width:233.844px;height:43px;direction:ltr;text-align:left;text-indent:0px;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box">
<div style="direction:ltr;text-align:left;text-indent:0px;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
3.1 GB</div>
</td>
<td style="width:224px;height:43px;direction:ltr;text-align:left;text-indent:0px;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box">
<div style="direction:ltr;text-align:left;text-indent:0px;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
3.8 GB</div>
</td>
<td style="width:71px;height:43px;direction:ltr;text-align:left;text-indent:0px;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;word-break:break-word;box-sizing:border-box">
<div style="direction:ltr;text-align:left;text-indent:0px;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
167 sec</div>
</td>
</tr>
<tr>
<td style="width:71.4688px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box">
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
2 core + Option2</div>
</td>
<td style="width:233.844px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box">
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
13.1 GB</div>
</td>
<td style="width:224px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box">
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
17.4 GB</div>
</td>
<td style="width:71px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;word-break:break-word;box-sizing:border-box">
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
89 sec</div>
</td>
</tr>
<tr>
<td style="width:71.4688px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box">
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Serial + Option 3</div>
</td>
<td style="width:233.844px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box">
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
4.7GB</div>
</td>
<td style="width:224px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box">
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
5.2GB</div>
</td>
<td style="width:71px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;word-break:break-word;box-sizing:border-box">
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
693 sec</div>
</td>
</tr>
<tr>
<td style="width:71.4688px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box">
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
2 core + Option 3</div>
</td>
<td style="width:233.844px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box">
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
23.2 GB</div>
</td>
<td style="width:224px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;box-sizing:border-box">
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
26.4 GB</div>
</td>
<td style="width:71px;height:43px;direction:ltr;border-width:1px;border-style:solid;border-color:rgb(171,171,171);vertical-align:top;word-break:break-word;box-sizing:border-box">
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
411 sec</div>
</td>
</tr>
</tbody>
</table>
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
Option 1 </div>
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
mpirun -n _ ./ex1 -A_name matrix.dat -b_name vector.dat -n_name _null_space.dat -num_near_nullspace 6 -ksp_type cg -pc_type gamg -pc_gamg_coarse_eq_limit 1000 -ksp_view -log_view -log_view_memory -pc_gamg_aggressive_square_graph true -pc_gamg_threshold 0.0
-info :pc</div>
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Option 2</div>
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:16px;color:rgb(0,0,0)">
<span style="background-color:rgb(255,255,255)">mpirun -n _ ./ex1 -A_name matrix.dat -b_name vector.dat -n_name _null_space.dat -num_near_nullspace 6 -ksp_type cg -pc_type gamg -pc_gamg_coarse_eq_limit 1000 -ksp_view -log_view -log_view_memory -pc_gamg_aggressive_square_graph
<b>false</b> -pc_gamg_threshold 0.0 -info :pc</span></div>
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:16px;color:rgb(0,0,0)">
<span style="background-color:rgb(255,255,255)"><br>
</span></div>
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:16px;color:rgb(0,0,0)">
<span style="background-color:rgb(255,255,255)">Option 3</span></div>
<div style="direction:ltr;margin:0px;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:16px;color:rgb(0,0,0)">
<span style="background-color:rgb(255,255,255)">mpirun -n _ ./ex1 -A_name matrix.dat -b_name vector.dat -n_name _null_space.dat -num_near_nullspace 6 -ksp_type cg -pc_type gamg -pc_gamg_coarse_eq_limit 1000 -ksp_view -log_view -log_view_memory -pc_gamg_aggressive_square_graph
true -pc_gamg_threshold <b>0.01</b> -info :pc</span></div>
<div style="direction:ltr;margin:0px;font-size:14px;color:unset"><button type="button" style="font-weight:600;font-size:14px;margin:4px 0px 0px 4px;vertical-align:middle;color:unset;box-sizing:border-box;display:inline-flex;text-decoration-line:none;overflow:hidden;min-width:0px;border-radius:4px;height:32px;white-space:nowrap;width:fit-content">
<div style="direction:ltr;margin-right:0px;margin-left:0px"></div>
</button></div>
</div>
</div>
</div>
<div style="margin:0px;padding-top:4px">
<div style="margin:0px 8px;border-radius:2px 2px 0px 0px;max-width:200px;min-width:80px">
<div style="margin:0px;padding-right:10px;padding-left:8px">
<div style="white-space:nowrap;font-size:14px"><span style="font-weight:600"><button type="button" title="Close tab" style="border:none;font-size:14px;margin:0px;outline:transparent;border-radius:2px;box-sizing:border-box;display:inline-block;text-decoration:none;text-align:center;width:28px;height:28px">
<div style="white-space:nowrap;margin:0px"></div>
</button></span></div>
</div>
</div>
</div>
</div>
<hr style="display:inline-block;width:98%">
<div style="direction:ltr;font-family:Calibri,sans-serif;font-size:11pt;color:rgb(0,0,0)">
<b>From:</b> Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>><br>
<b>Sent:</b> Tuesday, November 14, 2023 11:28 AM<br>
<b>To:</b> Jeremy Theler (External) <<a href="mailto:jeremy.theler-ext@ansys.com" target="_blank">jeremy.theler-ext@ansys.com</a>><br>
<b>Cc:</b> Ashish Patel <<a href="mailto:ashish.patel@ansys.com" target="_blank">ashish.patel@ansys.com</a>><br>
<b>Subject:</b> Re: About recent changes in GAMG</div>
<div style="direction:ltr"> </div>
<div style="padding:2pt;border-width:1pt;border-style:solid;border-color:rgb(156,101,0)">
<p style="line-height:12pt;background-color:rgb(255,235,156);margin-top:0px;margin-bottom:0px">
<span style="font-size:10pt;color:red"><b>[External Sender]</b></span></p>
</div>
<div style="direction:ltr">Sounds good,</div>
<div style="direction:ltr"><br>
</div>
<div style="direction:ltr">I think the not-square graph "aggressive" coarsening is only issue that I see and you can fix this by using:</div>
<div style="direction:ltr"><br>
</div>
<div style="direction:ltr">-mat_coarsen_type mis </div>
<div style="direction:ltr"><br>
</div>
<div style="direction:ltr">Aside, '-pc_gamg_aggressive_square_graph' should do it also, and you can use both and they will be ignored in earlier versions.</div>
<div style="direction:ltr"><br>
</div>
<div style="direction:ltr">If you see a difference then the first thing to do is run with '-info :pc' and send that to me (you can grep on 'GAMG' and send that if you like to reduce the data).</div>
<div style="direction:ltr"><br>
</div>
<div style="direction:ltr">Thanks,</div>
<div style="direction:ltr">Mark </div>
<div style="direction:ltr"><br>
</div>
<br>
<div style="direction:ltr">On Tue, Nov 14, 2023 at 8:49 AM Jeremy Theler (External) <<a href="mailto:jeremy.theler-ext@ansys.com" id="m_8494545079624546237m_-7618095157115570734x_m_8419638774470603419OWA491c5dae-b978-48e7-11ec-485d9c27a865" target="_blank">jeremy.theler-ext@ansys.com</a>> wrote:</div>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left:1px solid rgb(204,204,204)">
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi Mark.</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Thanks for reaching out. For now, we are going to stick to 3.19 for our production code because the changes in 3.20 impact in our tests in different ways (some of them perform better, some perform worse).</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I now switched to another task about investigating structural elements in DMplex.</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I'll go back to analyzing the new changes in GAMG in a couple of weeks so we can then see if we upgrade to 3.20 or we wait until 3.21.</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Thanks for your work and your kindness.</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
--</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
jeremy</div>
<hr style="direction:ltr;display:inline-block;width:98%">
<div style="direction:ltr;font-family:Calibri,sans-serif;font-size:11pt;color:rgb(0,0,0)">
<b>From:</b> Mark Adams <<a href="mailto:mfadams@lbl.gov" id="m_8494545079624546237m_-7618095157115570734x_m_8419638774470603419OWAf43c4bcb-90f2-4f02-6ea8-3c2b174df861" target="_blank">mfadams@lbl.gov</a>><br>
<b>Sent:</b> Tuesday, November 14, 2023 9:35 AM<br>
<b>To:</b> Jeremy Theler (External) <<a href="mailto:jeremy.theler-ext@ansys.com" id="m_8494545079624546237m_-7618095157115570734x_m_8419638774470603419OWAd080dd5b-ee75-ac41-1aac-1f86ae9af8d2" target="_blank">jeremy.theler-ext@ansys.com</a>><br>
<b>Cc:</b> Ashish Patel <<a href="mailto:ashish.patel@ansys.com" id="m_8494545079624546237m_-7618095157115570734x_m_8419638774470603419OWA2855e631-06a7-4533-914f-66c1f395f52a" target="_blank">ashish.patel@ansys.com</a>><br>
<b>Subject:</b> Re: About recent changes in GAMG</div>
<div style="direction:ltr"> </div>
<div style="padding:2pt;border-width:1pt;border-style:solid;border-color:rgb(156,101,0)">
<p style="direction:ltr;line-height:12pt;background-color:rgb(255,235,156);margin-top:0px;margin-bottom:0px">
<span style="font-size:10pt;color:red"><b>[External Sender]</b></span></p>
</div>
<div style="direction:ltr">Hi Jeremy,</div>
<div style="direction:ltr"><br>
</div>
<div style="direction:ltr">Just following up. </div>
<div style="direction:ltr">I appreciate your digging into performance regressions in GAMG.</div>
<div style="direction:ltr">AMG is really a pain sometimes and we want GAMG to be solid, at least for mainstream options, and your efforts are appreciated.</div>
<div style="direction:ltr">So feel free to start this discussion up.</div>
<div style="direction:ltr"><br>
</div>
<div style="direction:ltr">Thanks,</div>
<div style="direction:ltr">Mark</div>
<div style="direction:ltr"><br>
</div>
<div style="direction:ltr">On Wed, Oct 25, 2023 at 9:52 PM Jeremy Theler (External) <<a href="mailto:jeremy.theler-ext@ansys.com" id="m_8494545079624546237m_-7618095157115570734x_m_8419638774470603419OWAc5f06a48-dcb1-bb1e-fb62-4b466b347516" target="_blank">jeremy.theler-ext@ansys.com</a>> wrote:</div>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left:1px solid rgb(204,204,204)">
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Dear Mark</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Thanks for the follow up and sorry for the delay.</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I'm taking some days off. I'll be back to full throttle next week so can continue the discussion about these changes in GAMG.</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Regards,</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Jeremy</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<hr style="direction:ltr;display:inline-block;width:98%">
<div style="direction:ltr;font-family:Calibri,sans-serif;font-size:11pt;color:rgb(0,0,0)">
<b>From:</b> Mark Adams <<a href="mailto:mfadams@lbl.gov" id="m_8494545079624546237m_-7618095157115570734x_m_8419638774470603419OWA386c51e8-58fd-9a00-f4b6-608874bc2b06" target="_blank">mfadams@lbl.gov</a>><br>
<b>Sent:</b> Wednesday, October 18, 2023 9:15 AM<br>
<b>To:</b> Jeremy Theler (External) <<a href="mailto:jeremy.theler-ext@ansys.com" id="m_8494545079624546237m_-7618095157115570734x_m_8419638774470603419OWAfb6c7aee-fa9d-dac7-c8ca-dd0e4153dff6" target="_blank">jeremy.theler-ext@ansys.com</a>>; PETSc users list <<a href="mailto:petsc-users@mcs.anl.gov" id="m_8494545079624546237m_-7618095157115570734x_m_8419638774470603419OWA8ef1d6ae-5731-071b-32b3-026a7372fcc3" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Cc:</b> Ashish Patel <<a href="mailto:ashish.patel@ansys.com" id="m_8494545079624546237m_-7618095157115570734x_m_8419638774470603419OWA74ed31b1-c54b-7cc7-20d7-e99dcc6115fa" target="_blank">ashish.patel@ansys.com</a>><br>
<b>Subject:</b> Re: About recent changes in GAMG</div>
<div style="direction:ltr"> </div>
<div style="padding:2pt;border-width:1pt;border-style:solid;border-color:rgb(156,101,0)">
<p style="direction:ltr;line-height:12pt;background-color:rgb(255,235,156);margin-top:0px;margin-bottom:0px">
<span style="font-size:10pt;color:red"><b>[External Sender]</b></span></p>
</div>
<div style="direction:ltr">Hi Jeremy,</div>
<div style="direction:ltr"><br>
</div>
<div style="direction:ltr">I hope you don't mind putting this on the list (w/o data), but this is documentation and you are the second user that found regressions. </div>
<div style="direction:ltr">Sorry for the churn. </div>
<div style="direction:ltr"><br>
</div>
<div style="direction:ltr">There is a lot here so we can iterate, but here is a pass at your questions.</div>
<div style="direction:ltr"><br>
</div>
<div style="direction:ltr">*** Using MIS-2 instead of square graph was motivated by setup cost/performance but on GPUs with some recent fixes in Kokkos (in a branch) square graph seems OK.</div>
<div style="direction:ltr">My experience was that square graph is better in terms of quality and we have a power user, like you all, that found this also.</div>
<div style="direction:ltr">So I switched the default back to square graph.</div>
<div style="direction:ltr"><br>
</div>
<div style="direction:ltr">Interesting that you found that MIS-2 (new method) could be faster, but it might be because the two methods coarsen at different rates and that can make a big difference.</div>
<div style="direction:ltr">(the way to test would be to adjust parameters to get similar coarsen rates, but I digress)</div>
<div style="direction:ltr">It's hard to understand the differences between these two methods in terms of aggregate quality so we need to just experiment and have options.</div>
<div style="direction:ltr"><br>
</div>
<div style="direction:ltr">*** As far as your thermal problem. There was a complaint that the eigen estimates for chebyshev smoother were not recomputed for nonlinear problems and I added an option to do that and turned it on by default:</div>
<div style="direction:ltr">Use '-pc_gamg_recompute_esteig false' to get back to the original.</div>
<div style="direction:ltr">(I should have turned it off by default)</div>
<div style="direction:ltr"><br>
</div>
<div style="direction:ltr">Now, if your problem is symmetric and you use CG to compute the eigen estimates there should be no difference.</div>
<div style="direction:ltr">If you use CG to compute the eigen estimates in GAMG (and have GAMG give them to cheby, the default) that when you recompute the eigen estimates the cheby eigen estimator is used and that will use gmres by default unless you set the
SPD property in your matrix.</div>
<div style="direction:ltr">So if you set '-pc_gamg_esteig_ksp_type cg' you want to also set '-mg_levels_esteig_ksp_type cg' (verify with -ksp_view and -options_left)</div>
<div style="direction:ltr">CG is a much better estimator for SPD.</div>
<div style="direction:ltr"><br>
</div>
<div style="direction:ltr">And I found that the cheby eigen estimator uses an LAPACK *eigen* method to compute the eigen bounds and GAMG uses a *singular value* method.</div>
<div style="direction:ltr">The two give very different results on the lid driven cavity test (ex19). </div>
<div style="direction:ltr">eigen is lower, which is safer but not optimal if it is too low.</div>
<div style="direction:ltr">I have a branch to have cheby use the singular value method, but I don't plan on merging it (enough churn and I don't understand these differences).</div>
<div style="direction:ltr"><br>
</div>
<div style="direction:ltr">*** '-pc_gamg_low_memory_threshold_filter false' recovers the old filtering method. </div>
<div style="direction:ltr">This is the default now because there is a bug in the (new) low memory filter.</div>
<div style="direction:ltr">This bug is very rare and catastrophic.</div>
<div style="direction:ltr">We are working on it and will turn it on by default when it's fixed.</div>
<div style="direction:ltr">This does not affect the semantics of the solver, just work and memory complexity.</div>
<div style="direction:ltr"><br>
</div>
<div style="direction:ltr">*** As far as tet4 vs tet10, I would guess that tet4 wants more aggressive coarsening.</div>
<div style="direction:ltr">The default is to do aggressive on one (1) level.</div>
<div style="direction:ltr">You might want more levels for tet4.</div>
<div style="direction:ltr">And the new MIS-k coarsening can use any k (default is 2) wth '-mat_coarsen_misk_distance k' (eg, k=3)</div>
<div style="direction:ltr">I have not added hooks to have a more complex schedule to specify the method on each level.</div>
<div style="direction:ltr"><br>
</div>
<div style="direction:ltr">Thanks,</div>
<div style="direction:ltr">Mark</div>
<div style="direction:ltr"><br>
</div>
<div style="direction:ltr">On Tue, Oct 17, 2023 at 9:33 PM Jeremy Theler (External) <<a href="mailto:jeremy.theler-ext@ansys.com" id="m_8494545079624546237m_-7618095157115570734x_m_8419638774470603419OWA130dcdd8-8a43-730d-f93c-4429567ff9b5" target="_blank">jeremy.theler-ext@ansys.com</a>> wrote:</div>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left:1px solid rgb(204,204,204)">
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hey Mark</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Regarding the changes in the coarsening algorithm in 3.20 with respect to 3.19 in general we see that for some problems the MIS strategy gives and overall performance which is slightly better and for some others it is slightly worse than the "baseline" from
3.19.</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
We also saw that current main has switched back to the old square coarsening algorithm by default, which again, in some cases is better and in others is worse than 3.19 without any extra command-line option.</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Now what seems weird to us is that we have a test case which is a heat conduction problem with radiation boundary conditions (so it is non linear) using tet10 and we see</div>
<ol start="1" style="direction:ltr">
<li style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);direction:ltr;list-style-type:"1. "">
<span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">that in parallel v3.20 is way worse than v3.19, although the memory usage is similar</span></li><li style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);direction:ltr;list-style-type:"2. "">
<span style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">that petsc main (with no extra flags, just the defaults) recover the 3.19 performance but memory usage is significantly larger</span></li></ol>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I tried using the -pc_gamg_low_memory_threshold_filter flag and the results were the same.</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Find attached the log and snes views of 3.19, 3.20 and main using 4 MPI ranks.</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Is there any explanation about these two points we are seeing?</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Another weird finding is that if we use tet4 instead of tet10, v3.20 is only 10% slower than the other two and main does not need more memory than the other two.</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
BTW, I have dozens of other log view outputs comparing 3.19, 3.20 and main should you be interested.</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Let me know if it is better to move this discussion into the PETSc mailing list.</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Regards,</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
jeremy theler</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="direction:ltr;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
</blockquote>
</blockquote>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div></blockquote></div>
</blockquote></div></div>