<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBannerxgsywhc { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtonxgsywhc:hover, .pfptPrimaryButtonxgsywhc:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButtonxgsywhc:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
</head><!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
These options are prefixed by the equation name, and it won't work. You should add the option via the environment variable PETSC_OPTIONS="-log_view" On Fri, Jun 14, 2024, 23: 01 Barry Smith <bsmith@ petsc. dev> wrote: Are you
</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBannerxgsywhc" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBannerxgsywhc" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBannerxgsywhc" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBannerxgsywhc" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<div dir="auto">These options are prefixed by the equation name, and it won't work. You should add the option via the environment variable<div dir="auto"> PETSC_OPTIONS="-log_view"</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jun 14, 2024, 23:01 Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="display:none!important;display:none;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden">
Are you using https: //develop. openfoam. com/modules/external-solver If so you might be able to add it like petsc { options { log_view; ksp_type cg; pc_type bjacobi; sub_pc_type icc; } as discussed on the webpage for setting PETSc solver optionsOn
</div>
<div style="display:none!important;display:none;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden">ZjQcmQRYFpfptBannerStart</div>
<u></u>
<div dir="ltr" id="m_-7257113826661780366pfptBanneryb0y531" style="display:block!important;text-align:left!important;margin:16px 0px 16px 0px!important;padding:8px 16px 8px 16px!important;border-radius:4px!important;min-width:200px!important;background-color:#d0d8dc!important;background-color:#d0d8dc;border-top:4px solid #90a4ae!important;border-top:4px solid #90a4ae">
<div id="m_-7257113826661780366pfptBanneryb0y531" style="float:left!important;display:block!important;margin:0px 0px 1px 0px!important;max-width:600px!important">
<div id="m_-7257113826661780366pfptBanneryb0y531" style="display:block!important;background-color:#d0d8dc!important;color:#000000!important;color:#000000;font-family:'Arial',sans-serif!important;font-family:'Arial',sans-serif;font-weight:bold!important;font-weight:bold;font-size:14px!important;line-height:18px!important;line-height:18px">
This Message Is From an External Sender
</div>
<div id="m_-7257113826661780366pfptBanneryb0y531" style="display:block!important;background-color:#d0d8dc!important;color:#000000!important;color:#000000;font-weight:normal;font-family:'Arial',sans-serif!important;font-family:'Arial',sans-serif;font-size:12px!important;line-height:18px!important;line-height:18px;margin-top:2px!important">
This message came from outside your organization.
</div>
</div>
<div style="clear:both!important;display:block!important;line-height:0!important;font-size:0.01px!important;height:0px"> </div>
</div>
<u></u>
<div style="display:none!important;display:none;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden">ZjQcmQRYFpfptBannerEnd</div>
<div style="line-break:after-white-space"><div><br></div> Are you using <a href="https://urldefense.us/v3/__https://develop.openfoam.com/modules/external-solver__;!!G_uCfscf7eWS!YgXDdUKnfJdPEbKAQzR8TYlQzciWTcMmU_6jrKexdBB6UajcOHUvJhfaBjBtMN2cX6EUFDrcF4LcFItaSLxbAb0$" target="_blank" rel="noreferrer">https://develop.openfoam.com/modules/external-solver</a><div><br></div><div>If so you might be able to add it like</div><div><br></div><div><pre lang="plaintext">petsc
{
options
{</pre><pre lang="plaintext"> log_view;
ksp_type cg;
pc_type bjacobi;
sub_pc_type icc;
}
</pre><pre lang="plaintext">as discussed on the webpage for setting PETSc solver options</pre><pre lang="plaintext"><br></pre><div><br><blockquote type="cite"><div>On Jun 14, 2024, at 3:44 PM, Sircar, Arpan <<a href="mailto:sircara@ornl.gov" target="_blank" rel="noreferrer">sircara@ornl.gov</a>> wrote:</div><br><div><div style="font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt">Hi Junchao and Barry,</div><div style="font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt"><br></div><div style="font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt">I tried adding -log_view to my OpenFOAM command since that is what I am running the entire package through. However, it does not recognize that as a valid option. I am not sure where to put that tag in this setup.</div><div style="font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt">I am however using a Kokkos profiling tool, the output of which (for the GPU run) is attached with this email. Do let me know if this is useful or if you have ideas of where to put the -log_view tag.</div><div style="font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt"><br></div><div style="font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt">Junchao - Great nice running into you here.</div><div style="font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt"><br></div><div style="font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt">Thanks,</div><div style="font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt">Arpan</div><div id="m_-7257113826661780366appendonsend" style="font-family:Helvetica;font-size:18px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"></div><hr style="font-family:Helvetica;font-size:18px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;display:inline-block;width:1120.125px"><span style="font-family:Helvetica;font-size:18px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;float:none;display:inline!important"></span><div id="m_-7257113826661780366divRplyFwdMsg" dir="ltr" style="font-family:Helvetica;font-size:18px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><font face="Calibri, sans-serif" style="font-size:11pt"><b>From:</b><span> </span>Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank" rel="noreferrer">junchao.zhang@gmail.com</a>><br><b>Sent:</b><span> </span>Friday, June 14, 2024 3:16 PM<br><b>To:</b><span> </span>Sircar, Arpan <<a href="mailto:sircara@ornl.gov" target="_blank" rel="noreferrer">sircara@ornl.gov</a>><br><b>Cc:</b><span> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" rel="noreferrer">bsmith@petsc.dev</a>>;<span> </span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a><span> </span><<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a>>; Gottiparthi, Kalyan <<a href="mailto:gottiparthik@ornl.gov" target="_blank" rel="noreferrer">gottiparthik@ornl.gov</a>><br><b>Subject:</b><span> </span>Re: [petsc-users] [EXTERNAL] Re: Running PETSc with a Kokkos backend on OLCF Frontier</font><div> </div></div><div style="font-family:Helvetica;font-size:18px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div dir="ltr">Arpan,<div> Did you add -log_view ?</div><div><br clear="all"><div><div dir="ltr" data-smartmail="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div><br></div></div><br><div><div dir="ltr">On Fri, Jun 14, 2024 at 2:00 PM Sircar, Arpan via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a>> wrote:<br></div><blockquote style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div><u></u><div dir="ltr" id="m_-7257113826661780366x_m_8085757438764483939pfptBannerreuum1h" style="display:block;text-align:left;margin:16px 0px;padding:8px 16px;border-radius:4px;min-width:200px;background-color:rgb(208,216,220);border-top-width:4px;border-top-style:solid;border-top-color:rgb(144,164,174)"><div id="m_-7257113826661780366x_m_8085757438764483939pfptBannerreuum1h" style="float:left;display:block;margin:0px 0px 1px;max-width:600px"><div id="m_-7257113826661780366x_m_8085757438764483939pfptBannerreuum1h" style="display:block;background-color:rgb(208,216,220);font-family:Arial,sans-serif;font-weight:bold;font-size:14px;line-height:18px">This Message Is From an External Sender<span> </span></div><div id="m_-7257113826661780366x_m_8085757438764483939pfptBannerreuum1h" style="font-weight:normal;display:block;background-color:rgb(208,216,220);font-family:Arial,sans-serif;font-size:12px;line-height:18px;margin-top:2px">This message came from outside your organization.<span> </span></div></div><div style="height:0px;clear:both;display:block;line-height:0;font-size:0.01px"> </div></div><u></u><div dir="ltr"><div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt">Hi Barry,</div><div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt"><br></div><div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt">Thanks for your prompt response. These are run with with the same PETSc solvers but the one on GPUs (log_pKok) uses mataijkokkos while the other one does not.</div><div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt"><br></div><div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt">Please let me know if you need any other information.</div><div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt"><br></div><div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt">Thanks,</div><div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt">Arpan</div><div id="m_-7257113826661780366x_m_8085757438764483939appendonsend"></div><hr style="display:inline-block;width:1102.5625px"><div id="m_-7257113826661780366x_m_8085757438764483939divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt"><b>From:</b><span> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" rel="noreferrer">bsmith@petsc.dev</a>><br><b>Sent:</b><span> </span>Friday, June 14, 2024 1:47 PM<br><b>To:</b><span> </span>Sircar, Arpan <<a href="mailto:sircara@ornl.gov" target="_blank" rel="noreferrer">sircara@ornl.gov</a>><br><b>Cc:</b><span> </span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a><span> </span><<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a>>; Gottiparthi, Kalyan <<a href="mailto:gottiparthik@ornl.gov" target="_blank" rel="noreferrer">gottiparthik@ornl.gov</a>><br><b>Subject:</b><span> </span>[EXTERNAL] Re: [petsc-users] Running PETSc with a Kokkos backend on OLCF Frontier</font><div> </div></div><div><div><br></div> Please run both the CPU solvers and GPU solvers cases with -log_view and send the two outputs.<div><br></div><div> Barry</div><div><br id="m_-7257113826661780366x_m_8085757438764483939x_lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On Jun 14, 2024, at 1:35 PM, Sircar, Arpan via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank" rel="noreferrer">petsc-users@mcs.anl.gov</a>> wrote:</div><br><div><div dir="ltr" id="m_-7257113826661780366x_m_8085757438764483939x_pfptBannervdthixz" style="writing-mode:revert;color:revert;font-family:revert;font-feature-settings:revert;font-kerning:revert;font-size:revert;font-stretch:revert;font-variant-alternates:revert;font-variant-caps:revert;font-variant-east-asian:revert;font-variant-ligatures:revert;font-variant-numeric:revert;font-weight:revert;text-orientation:revert;zoom:revert;letter-spacing:revert;background-blend-mode:revert;background-image:revert;background-position:revert;background-repeat:revert;background-size:revert;border-collapse:revert;box-sizing:revert;break-after:revert;break-before:revert;break-inside:revert;caption-side:revert;clear:revert;column-width:revert;column-count:revert;column-fill:revert;column-gap:revert;column-rule-width:revert;column-rule-style:revert;column-rule-color:revert;column-span:revert;empty-cells:revert;float:revert;image-orientation:revert;isolation:revert;line-break:revert;line-height:revert;list-style:revert;mix-blend-mode:revert;object-fit:revert;object-position:revert;outline:revert;overflow:revert;quotes:revert;table-layout:revert;text-align:left;text-decoration:revert;text-emphasis-style:revert;text-emphasis-color:revert;text-indent:revert;text-overflow:revert;text-transform:revert;text-underline-position:revert;vertical-align:revert;white-space:revert;word-break:revert;border-spacing:revert"><div id="m_-7257113826661780366x_m_8085757438764483939x_pfptBannervdthixz" style="writing-mode:unset;color:unset;font-family:unset;font-feature-settings:unset;font-kerning:unset;font-size:unset;font-stretch:unset;font-variant-alternates:unset;font-variant-caps:unset;font-variant-east-asian:unset;font-variant-ligatures:unset;font-variant-numeric:unset;font-weight:unset;text-orientation:unset;zoom:unset;letter-spacing:unset;background:unset;background-blend-mode:unset;border-collapse:unset;border:unset;box-sizing:unset;break-after:unset;break-before:unset;break-inside:unset;caption-side:unset;clear:unset;column-width:unset;column-count:unset;column-fill:unset;column-gap:unset;column-rule-width:unset;column-rule-style:unset;column-rule-color:unset;column-span:unset;empty-cells:unset;float:left;image-orientation:unset;isolation:unset;line-break:unset;line-height:unset;list-style:unset;mix-blend-mode:unset;object-fit:unset;object-position:unset;outline:unset;overflow:unset;quotes:unset;table-layout:unset;text-align:unset;text-decoration:unset;text-emphasis-style:unset;text-emphasis-color:unset;text-indent:unset;text-overflow:unset;text-transform:unset;text-underline-position:unset;vertical-align:unset;white-space:unset;word-break:unset;border-spacing:unset"><div id="m_-7257113826661780366x_m_8085757438764483939x_pfptBannervdthixz" style="writing-mode:unset;font-family:Arial,sans-serif;font-feature-settings:unset;font-kerning:unset;font-size:14px;font-stretch:unset;font-variant-alternates:unset;font-variant-caps:unset;font-variant-east-asian:unset;font-variant-ligatures:unset;font-variant-numeric:unset;font-weight:bold;text-orientation:unset;zoom:unset;letter-spacing:unset;background-blend-mode:unset;background-image:unset;background-position:unset;background-repeat:unset;background-size:unset;border-collapse:unset;border:unset;box-sizing:unset;break-after:unset;break-before:unset;break-inside:unset;caption-side:unset;clear:unset;column-width:unset;column-count:unset;column-fill:unset;column-gap:unset;column-rule-width:unset;column-rule-style:unset;column-rule-color:unset;column-span:unset;empty-cells:unset;float:unset;image-orientation:unset;isolation:unset;line-break:unset;line-height:18px;list-style:unset;mix-blend-mode:unset;object-fit:unset;object-position:unset;outline:unset;overflow:unset;quotes:unset;table-layout:unset;text-align:unset;text-decoration:unset;text-emphasis-style:unset;text-emphasis-color:unset;text-indent:unset;text-overflow:unset;text-transform:unset;text-underline-position:unset;vertical-align:unset;white-space:unset;word-break:unset;border-spacing:unset;word-spacing:unset;background-clip:unset;background-origin:unset;text-combine-upright:unset">This Message Is From an External Sender</div><div id="m_-7257113826661780366x_m_8085757438764483939x_pfptBannervdthixz" style="writing-mode:unset;font-family:Arial,sans-serif;font-feature-settings:unset;font-kerning:unset;font-size:12px;font-stretch:unset;font-variant-alternates:unset;font-variant-caps:unset;font-variant-east-asian:unset;font-variant-ligatures:unset;font-variant-numeric:unset;font-weight:unset;text-orientation:unset;zoom:unset;letter-spacing:unset;background-blend-mode:unset;background-image:unset;background-position:unset;background-repeat:unset;background-size:unset;border-collapse:unset;border:unset;box-sizing:unset;break-after:unset;break-before:unset;break-inside:unset;caption-side:unset;clear:unset;column-width:unset;column-count:unset;column-fill:unset;column-gap:unset;column-rule-width:unset;column-rule-style:unset;column-rule-color:unset;column-span:unset;empty-cells:unset;float:unset;image-orientation:unset;isolation:unset;line-break:unset;line-height:18px;list-style:unset;mix-blend-mode:unset;object-fit:unset;object-position:unset;outline:unset;overflow:unset;quotes:unset;table-layout:unset;text-align:unset;text-decoration:unset;text-emphasis-style:unset;text-emphasis-color:unset;text-indent:unset;text-overflow:unset;text-transform:unset;text-underline-position:unset;vertical-align:unset;white-space:unset;word-break:unset;border-spacing:unset;word-spacing:unset;background-clip:unset;background-origin:unset;text-combine-upright:unset">This message came from outside your organization.</div></div><div style="height:0px;clear:both;display:block;line-height:0;font-size:0.01px"></div></div><div style="font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt">Hi,</div><div style="font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt"><br></div><div style="font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt">We have been working with OpenFOAM (an open-source CFD software) which can transfer its matrices to PETSc to use its linear solvers. This has been tested and is working well on OCLF's Frontier machine. Next we are trying to use the Kokkos backend to run it on Frontier GPUs. While the OpenFOAM+PETSc+Kokkos environment built correctly on Frontier using the modules sourced (attached file<span> </span><i>bash_petsc4foam_gpu</i>) and configuring PETSc correctly (attached file<span> </span><i>config_gpu</i>), the GPU solve seems to take more time than the CPU solve.</div><div style="font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt"><br></div><div style="font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt">The PETSc run-time options we are using are attached to this email (file<span> </span><i>fvSolution_petsc_pKok_Uof</i>). Could you please take a look and let us know if this combination of options is fine? In this approach we are trying to solve the pressure equation only on the GPUs.</div><div style="font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt"><br></div><div style="font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt">Thanks,</div><div style="font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt">Arpan</div><div style="font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none;font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt"><br></div><div id="m_-7257113826661780366x_m_8085757438764483939x_Signature" style="font-family:Helvetica;font-size:18px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:11pt;color:rgb(102,102,102)"><span style="background-color:rgb(255,255,255)"><b>Arpan Sircar</b></span></div><div style="text-align:left;background-color:rgb(255,255,255);margin:0px;font-family:Calibri,Arial,Helvetica,sans-serif;font-size:10pt;color:rgb(102,102,102)"><span style="background-color:rgb(255,255,255)">R&D Associate Staff</span></div><div style="text-align:left;background-color:rgb(255,255,255);margin:0px;font-family:Calibri,Arial,Helvetica,sans-serif;font-size:10pt;color:rgb(102,102,102)"><span style="background-color:rgb(255,255,255)">Thermal Hydraulics Group</span></div><div style="text-align:left;background-color:rgb(255,255,255);margin:0px;font-family:Calibri,Arial,Helvetica,sans-serif;font-size:10pt;color:rgb(102,102,102)">Nuclear Energy and Fuel Cycle Division</div><div style="text-align:left;background-color:rgb(255,255,255);margin:0px;font-family:Calibri,Arial,Helvetica,sans-serif;font-size:10pt;color:rgb(102,102,102)"><b>Oak Ridge National Laboratory</b></div><div style="text-align:left;background-color:rgb(255,255,255);margin:0px;font-size:15px;color:rgb(36,36,36)"><br></div></div><span id="m_-7257113826661780366x_m_8085757438764483939x_cid:30E5FD77-BD68-4987-9F1C-4EF987DA4B33"><fvSolution_petsc_pKok_Uof></span><span id="m_-7257113826661780366x_m_8085757438764483939x_cid:F8894B7F-0EC7-4EB0-8607-4434E292F59C"><bash_petsc4foam_gpu></span><span id="m_-7257113826661780366x_m_8085757438764483939x_cid:9F0AA58D-E203-4872-8D80-814D21CBB4C1"><config_gpu></span></div></blockquote></div><br></div></div></div></div></blockquote></div></div><span id="m_-7257113826661780366cid:E776427D-88AA-4C0A-9CB2-B25B923C70D7"><log.kok></span></div></blockquote></div><br></div></div></blockquote></div>