<!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
We need the output when run with -log_view to see where the time is being spent and what communication between the CPU and GPU is occurring. Barry On Jun 14, 2024, at 2: 59 PM, Sircar, Arpan <sircara@ ornl. gov> wrote: Hi Barry,Thanks for
</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBannerngqlv14" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBannerngqlv14" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBannerngqlv14" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBannerngqlv14" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<html><head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBannerngqlv14 { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtonngqlv14:hover, .pfptPrimaryButtonngqlv14:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButtonngqlv14:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
<meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div><br></div> We need the output when run with -log_view to see where the time is being spent and what communication between the CPU and GPU is occurring.<div><br></div><div> Barry</div><div><br id="lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On Jun 14, 2024, at 2:59 PM, Sircar, Arpan <sircara@ornl.gov> wrote:</div><br class="Apple-interchange-newline"><div><meta charset="UTF-8"><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">Hi Barry,</div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">Thanks for your prompt response. These are run with with the same PETSc solvers but the one on GPUs (log_pKok) uses mataijkokkos while the other one does not.</div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">Please let me know if you need any other information.</div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">Thanks,</div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">Arpan</div><div id="appendonsend" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"></div><hr tabindex="-1" style="font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; display: inline-block; width: 1120.125px;"><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;"></span><div id="divRplyFwdMsg" dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><font face="Calibri, sans-serif" style="font-size: 11pt;"><b>From:</b><span class="Apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>><br><b>Sent:</b><span class="Apple-converted-space"> </span>Friday, June 14, 2024 1:47 PM<br><b>To:</b><span class="Apple-converted-space"> </span>Sircar, Arpan <<a href="mailto:sircara@ornl.gov">sircara@ornl.gov</a>><br><b>Cc:</b><span class="Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a><span class="Apple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>>; Gottiparthi, Kalyan <<a href="mailto:gottiparthik@ornl.gov">gottiparthik@ornl.gov</a>><br><b>Subject:</b><span class="Apple-converted-space"> </span>[EXTERNAL] Re: [petsc-users] Running PETSc with a Kokkos backend on OLCF Frontier</font><div> </div></div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; line-break: after-white-space;"><div><br></div> Please run both the CPU solvers and GPU solvers cases with -log_view and send the two outputs.<div><br></div><div> Barry</div><div><br id="x_lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On Jun 14, 2024, at 1:35 PM, Sircar, Arpan via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>> wrote:</div><br class="x_Apple-interchange-newline"><div><div dir="ltr" id="x_pfptBannervdthixz" style="writing-mode: revert !important; color: revert !important; font-family: revert !important; font-feature-settings: revert !important; font-kerning: revert !important; font-size: revert !important; font-size-adjust: revert !important; font-stretch: revert !important; font-style: revert !important; font-variant-alternates: revert !important; font-variant-caps: revert !important; font-variant-east-asian: revert !important; font-variant-ligatures: revert !important; font-variant-numeric: revert !important; font-weight: revert !important; text-orientation: revert !important; zoom: revert !important; letter-spacing: revert !important; align-content: revert !important; align-items: revert !important; align-self: revert !important; background-attachment: revert !important; background-blend-mode: revert !important; background-image: revert !important; background-position: revert !important; background-repeat: revert !important; background-size: revert !important; border-collapse: revert !important; box-sizing: revert !important; break-after: revert !important; break-before: revert !important; break-inside: revert !important; caption-side: revert !important; clear: revert !important; clip: revert !important; column-count: revert !important; column-fill: revert !important; column-gap: revert !important; column-rule-color: revert !important; column-rule-style: revert !important; column-rule-width: revert !important; column-span: revert !important; column-width: revert !important; content: revert !important; counter-increment: revert !important; counter-reset: revert !important; empty-cells: revert !important; flex: revert !important; flex-flow: revert !important; float: revert !important; image-orientation: revert !important; isolation: revert !important; justify-content: revert !important; justify-items: revert !important; line-break: revert !important; line-height: revert !important; list-style: revert !important; mix-blend-mode: revert !important; object-fit: revert !important; object-position: revert !important; order: revert !important; orphans: revert !important; outline: revert !important; overflow: revert !important; position: revert !important; quotes: revert !important; tab-size: revert !important; table-layout: revert !important; text-align: left !important; text-decoration-color: revert !important; text-decoration: revert !important; text-decoration-style: revert !important; text-emphasis-color: revert !important; text-emphasis-style: revert !important; text-indent: revert !important; text-overflow: revert !important; text-transform: revert !important; text-underline-position: revert !important; vertical-align: revert !important; white-space: revert !important; widows: revert !important; word-break: revert !important; z-index: revert !important; border-spacing: revert !important;"><div id="x_pfptBannervdthixz" style="writing-mode: unset !important; color: unset !important; font-family: unset !important; font-feature-settings: unset !important; font-kerning: unset !important; font-size: unset !important; font-size-adjust: unset !important; font-stretch: unset !important; font-style: unset !important; font-variant-alternates: unset !important; font-variant-caps: unset !important; font-variant-east-asian: unset !important; font-variant-ligatures: unset !important; font-variant-numeric: unset !important; font-weight: unset !important; text-orientation: unset !important; zoom: unset !important; letter-spacing: unset !important; align-content: unset !important; align-items: unset !important; align-self: unset !important; background: unset !important; background-blend-mode: unset !important; border-collapse: unset !important; border: unset !important; box-sizing: unset !important; break-after: unset !important; break-before: unset !important; break-inside: unset !important; caption-side: unset !important; clear: unset !important; clip: unset !important; column-count: unset !important; column-fill: unset !important; column-gap: unset !important; column-rule-color: unset !important; column-rule-style: unset !important; column-rule-width: unset !important; column-span: unset !important; column-width: unset !important; content: unset !important; counter-increment: unset !important; counter-reset: unset !important; empty-cells: unset !important; flex: unset !important; flex-flow: unset !important; float: left !important; image-orientation: unset !important; isolation: unset !important; justify-content: unset !important; justify-items: unset !important; line-break: unset !important; line-height: unset !important; list-style: unset !important; mix-blend-mode: unset !important; object-fit: unset !important; object-position: unset !important; order: unset !important; orphans: unset !important; outline: unset !important; overflow: unset !important; position: unset !important; quotes: unset !important; tab-size: unset !important; table-layout: unset !important; text-align: unset !important; text-decoration-color: unset !important; text-decoration: unset !important; text-decoration-style: unset !important; text-emphasis-color: unset !important; text-emphasis-style: unset !important; text-indent: unset !important; text-overflow: unset !important; text-transform: unset !important; text-underline-position: unset !important; vertical-align: unset !important; white-space: unset !important; widows: unset !important; word-break: unset !important; z-index: unset !important; border-spacing: unset !important;"><div id="x_pfptBannervdthixz" style="writing-mode: unset !important; font-family: Arial, sans-serif !important; font-feature-settings: unset !important; font-kerning: unset !important; font-size: 14px !important; font-size-adjust: unset !important; font-stretch: unset !important; font-style: unset !important; font-variant-alternates: unset !important; font-variant-caps: unset !important; font-variant-east-asian: unset !important; font-variant-ligatures: unset !important; font-variant-numeric: unset !important; font-weight: bold !important; text-orientation: unset !important; zoom: unset !important; letter-spacing: unset !important; align-content: unset !important; align-items: unset !important; align-self: unset !important; background-attachment: unset !important; background-blend-mode: unset !important; background-image: unset !important; background-position: unset !important; background-repeat: unset !important; background-size: unset !important; border-collapse: unset !important; border: unset !important; box-sizing: unset !important; break-after: unset !important; break-before: unset !important; break-inside: unset !important; caption-side: unset !important; clear: unset !important; clip: unset !important; column-count: unset !important; column-fill: unset !important; column-gap: unset !important; column-rule-color: unset !important; column-rule-style: unset !important; column-rule-width: unset !important; column-span: unset !important; column-width: unset !important; content: unset !important; counter-increment: unset !important; counter-reset: unset !important; empty-cells: unset !important; flex: unset !important; flex-flow: unset !important; float: unset !important; image-orientation: unset !important; isolation: unset !important; justify-content: unset !important; justify-items: unset !important; line-break: unset !important; line-height: 18px !important; list-style: unset !important; mix-blend-mode: unset !important; object-fit: unset !important; object-position: unset !important; order: unset !important; orphans: unset !important; outline: unset !important; overflow: unset !important; position: unset !important; quotes: unset !important; tab-size: unset !important; table-layout: unset !important; text-align: unset !important; text-decoration-color: unset !important; text-decoration: unset !important; text-decoration-style: unset !important; text-emphasis-color: unset !important; text-emphasis-style: unset !important; text-indent: unset !important; text-overflow: unset !important; text-transform: unset !important; text-underline-position: unset !important; vertical-align: unset !important; white-space: unset !important; widows: unset !important; word-break: unset !important; z-index: unset !important; border-spacing: unset !important; word-spacing: unset !important; background-clip: unset !important; background-origin: unset !important; box-shadow: unset !important; text-combine-upright: unset !important;">This Message Is From an External Sender</div><div id="x_pfptBannervdthixz" style="writing-mode: unset !important; font-family: Arial, sans-serif !important; font-feature-settings: unset !important; font-kerning: unset !important; font-size: 12px !important; font-size-adjust: unset !important; font-stretch: unset !important; font-style: unset !important; font-variant-alternates: unset !important; font-variant-caps: unset !important; font-variant-east-asian: unset !important; font-variant-ligatures: unset !important; font-variant-numeric: unset !important; font-weight: unset !important; text-orientation: unset !important; zoom: unset !important; letter-spacing: unset !important; align-content: unset !important; align-items: unset !important; align-self: unset !important; background-attachment: unset !important; background-blend-mode: unset !important; background-image: unset !important; background-position: unset !important; background-repeat: unset !important; background-size: unset !important; border-collapse: unset !important; border: unset !important; box-sizing: unset !important; break-after: unset !important; break-before: unset !important; break-inside: unset !important; caption-side: unset !important; clear: unset !important; clip: unset !important; column-count: unset !important; column-fill: unset !important; column-gap: unset !important; column-rule-color: unset !important; column-rule-style: unset !important; column-rule-width: unset !important; column-span: unset !important; column-width: unset !important; content: unset !important; counter-increment: unset !important; counter-reset: unset !important; empty-cells: unset !important; flex: unset !important; flex-flow: unset !important; float: unset !important; image-orientation: unset !important; isolation: unset !important; justify-content: unset !important; justify-items: unset !important; line-break: unset !important; line-height: 18px !important; list-style: unset !important; mix-blend-mode: unset !important; object-fit: unset !important; object-position: unset !important; order: unset !important; orphans: unset !important; outline: unset !important; overflow: unset !important; position: unset !important; quotes: unset !important; tab-size: unset !important; table-layout: unset !important; text-align: unset !important; text-decoration-color: unset !important; text-decoration: unset !important; text-decoration-style: unset !important; text-emphasis-color: unset !important; text-emphasis-style: unset !important; text-indent: unset !important; text-overflow: unset !important; text-transform: unset !important; text-underline-position: unset !important; vertical-align: unset !important; white-space: unset !important; widows: unset !important; word-break: unset !important; z-index: unset !important; border-spacing: unset !important; word-spacing: unset !important; background-clip: unset !important; background-origin: unset !important; box-shadow: unset !important; text-combine-upright: unset !important;">This message came from outside your organization.</div></div><div style="height: 0px; clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important;"></div></div><div class="x_elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">Hi,</div><div class="x_elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><div class="x_elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">We have been working with OpenFOAM (an open-source CFD software) which can transfer its matrices to PETSc to use its linear solvers. This has been tested and is working well on OCLF's Frontier machine. Next we are trying to use the Kokkos backend to run it on Frontier GPUs. While the OpenFOAM+PETSc+Kokkos environment built correctly on Frontier using the modules sourced (attached file<span class="x_Apple-converted-space"> </span><i>bash_petsc4foam_gpu</i>) and configuring PETSc correctly (attached file<span class="x_Apple-converted-space"> </span><i>config_gpu</i>), the GPU solve seems to take more time than the CPU solve.</div><div class="x_elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><div class="x_elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">The PETSc run-time options we are using are attached to this email (file<span class="x_Apple-converted-space"> </span><i>fvSolution_petsc_pKok_Uof</i>). Could you please take a look and let us know if this combination of options is fine? In this approach we are trying to solve the pressure equation only on the GPUs.</div><div class="x_elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><div class="x_elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">Thanks,</div><div class="x_elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;">Arpan</div><div style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none; font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><div id="x_Signature" style="font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; text-decoration: none;"><div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(102, 102, 102);"><span style="background-color: rgb(255, 255, 255);"><b>Arpan Sircar</b></span></div><div style="text-align: left; background-color: rgb(255, 255, 255); margin: 0px; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(102, 102, 102);"><span style="background-color: rgb(255, 255, 255);">R&D Associate Staff</span></div><div style="text-align: left; background-color: rgb(255, 255, 255); margin: 0px; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(102, 102, 102);"><span style="background-color: rgb(255, 255, 255);">Thermal Hydraulics Group</span></div><div style="text-align: left; background-color: rgb(255, 255, 255); margin: 0px; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(102, 102, 102);">Nuclear Energy and Fuel Cycle Division</div><div style="text-align: left; background-color: rgb(255, 255, 255); margin: 0px; font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 10pt; color: rgb(102, 102, 102);"><b>Oak Ridge National Laboratory</b></div><div style="text-align: left; background-color: rgb(255, 255, 255); margin: 0px; font-size: 15px; color: rgb(36, 36, 36);"><br></div></div><span id="x_cid:30E5FD77-BD68-4987-9F1C-4EF987DA4B33"><fvSolution_petsc_pKok_Uof></span><span id="x_cid:F8894B7F-0EC7-4EB0-8607-4434E292F59C"><bash_petsc4foam_gpu></span><span id="x_cid:9F0AA58D-E203-4872-8D80-814D21CBB4C1"><config_gpu></span></div></blockquote></div><br></div></div><span id="cid:813F576F-8841-4AD5-AE2B-098628C5FD25"><log_pKok></span><span id="cid:5BD5A85C-513C-4C3A-9E86-58F3153E1242"><log_pPet></span></div></blockquote></div><br></div></body></html>