<head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBannergt96kxi { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtongt96kxi:hover, .pfptPrimaryButtongt96kxi:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButtongt96kxi:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
</head><!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
Thanks for the suggestions; I will try them out. Dense factorization is used as the benchmark for Top500 right? That's why I thought there would be some state-of-the-art multi GPU dense linear solvers out there. I saw this library called
</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBannergt96kxi" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBannergt96kxi" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBannergt96kxi" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBannergt96kxi" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<div dir="auto">Thanks for the suggestions; I will try them out.<div dir="auto"><br></div><div dir="auto">Dense factorization is used as the benchmark for Top500 right? That's why I thought there would be some state-of-the-art multi GPU dense linear solvers out there.</div><div dir="auto"><br></div><div dir="auto">I saw this library called cuSOLVERMp <a href="https://urldefense.us/v3/__https://docs.nvidia.com/cuda/cusolvermp/__;!!G_uCfscf7eWS!ebpPt6OKSu0Ua8y56LhYJM0ol0OAD-aZ4XGMbFPoIIzc0oNqKZryYg0uIRdhObPv7MOrgO1jJFieu5U2hVjtUcOPaA$">https://docs.nvidia.com/cuda/cusolvermp/</a> from NVIDIA. It looks somewhat difficult to integrate with other code, though.</div><div dir="auto"><br></div><div dir="auto">I also found this <a href="https://urldefense.us/v3/__https://github.com/nv-legate/cunumeric__;!!G_uCfscf7eWS!ebpPt6OKSu0Ua8y56LhYJM0ol0OAD-aZ4XGMbFPoIIzc0oNqKZryYg0uIRdhObPv7MOrgO1jJFieu5U2hVg0mdmrig$">https://github.com/nv-legate/cunumeric</a> from NVIDIA which shows some good results for multi GPU Cholesky, but I'm having some trouble getting it set up correctly.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jul 24, 2024, 12:08 PM Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="line-break:after-white-space"><div><br></div><div> For one MPI rank, it looks like you can use -pc_type cholesky -pc_factor_mat_solver_type cupm though it is not documented in <a href="https://urldefense.us/v3/__https://petsc.org/release/overview/linear_solve_table/*direct-solvers__;Iw!!G_uCfscf7eWS!ebpPt6OKSu0Ua8y56LhYJM0ol0OAD-aZ4XGMbFPoIIzc0oNqKZryYg0uIRdhObPv7MOrgO1jJFieu5U2hVhs7ez3Mw$" target="_blank" rel="noreferrer">https://petsc.org/release/overview/linear_solve_table/#direct-solvers</a></div><div><br></div><div> Of if you also ./configure --download-kokkos --download-kokkos-kernels you can use <span style="font-family:Menlo;font-size:14px">-pc_factor_mat_solver_type kokkos </span>if you also this may also work for multiple GPUs but that is not documented in the table either (Junchao) Nor are sparse Kokkos or CUDA stuff documented (if they exist) in the table.</div><div><br></div><div><span style="font-family:Menlo;font-size:14px"><br></span></div><div><span style="font-family:Menlo;font-size:14px"> Barry</span></div><div><span style="font-family:Menlo;font-size:14px"><br></span></div><div><font face="Menlo"><span style="font-size:14px"><br></span></font><div><br><blockquote type="cite"><div>On Jul 24, 2024, at 2:44 PM, Sreeram R Venkat <<a href="mailto:srvenkat@utexas.edu" target="_blank" rel="noreferrer">srvenkat@utexas.edu</a>> wrote:</div><br><div><div dir="ltr" id="m_5785087414262769895pfptBannermd4yide" style="writing-mode:revert!important;color:revert!important;font-family:revert!important;font-feature-settings:revert!important;font-kerning:revert!important;font-size:revert!important;font-size-adjust:revert!important;font-stretch:revert!important;font-variant-alternates:revert!important;font-variant-caps:revert!important;font-variant-east-asian:revert!important;font-variant-ligatures:revert!important;font-variant-numeric:revert!important;font-weight:revert!important;text-orientation:revert!important;zoom:revert!important;letter-spacing:revert!important;text-autospace:revert!important;background-blend-mode:revert!important;background-image:revert!important;background-position:revert!important;background-repeat:revert!important;background-size:revert!important;border-collapse:revert!important;box-sizing:revert!important;break-after:revert!important;break-before:revert!important;break-inside:revert!important;caption-side:revert!important;clear:revert!important;column-count:revert!important;column-fill:revert!important;column-gap:revert!important;column-rule-color:revert!important;column-rule-style:revert!important;column-rule-width:revert!important;column-span:revert!important;column-width:revert!important;empty-cells:revert!important;float:revert!important;image-orientation:revert!important;isolation:revert!important;line-break:revert!important;line-height:revert!important;list-style:revert!important;mix-blend-mode:revert!important;object-fit:revert!important;object-position:revert!important;outline:revert!important;overflow:revert!important;quotes:revert!important;table-layout:revert!important;text-align:left!important;text-align-last:revert!important;text-decoration-color:revert!important;text-decoration:revert!important;text-decoration-style:revert!important;text-emphasis-color:revert!important;text-emphasis-style:revert!important;text-indent:revert!important;text-justify:revert!important;text-overflow:revert!important;text-transform:revert!important;text-underline-position:revert!important;vertical-align:revert!important;white-space:revert!important;word-break:revert!important;border-spacing:revert!important;word-spacing:revert!important;background-clip:revert!important;background-origin:revert!important;text-combine-upright:revert!important;display:block!important;opacity:revert!important;background-color:rgb(208,216,220)!important;border-bottom-color:revert!important;border-bottom-style:revert!important;border-bottom-width:revert!important;border-left-color:revert!important;border-left-style:revert!important;border-left-width:revert!important;border-right-color:revert!important;border-right-style:revert!important;border-right-width:revert!important;height:revert!important;max-height:revert!important;max-width:revert!important;min-height:revert!important;width:revert!important;margin:16px 0px!important;padding:8px 16px!important;border-radius:4px!important;min-width:200px!important;border-top-width:4px!important;border-top-style:solid!important;border-top-color:rgb(144,164,174)!important"><div id="m_5785087414262769895pfptBannermd4yide" style="writing-mode:unset!important;color:unset!important;font-family:unset!important;font-feature-settings:unset!important;font-kerning:unset!important;font-size:unset!important;font-size-adjust:unset!important;font-stretch:unset!important;font-variant-alternates:unset!important;font-variant-caps:unset!important;font-variant-east-asian:unset!important;font-variant-ligatures:unset!important;font-variant-numeric:unset!important;font-weight:unset!important;text-orientation:unset!important;zoom:unset!important;letter-spacing:unset!important;text-autospace:unset!important;background:unset!important;background-blend-mode:unset!important;border-collapse:unset!important;border:unset!important;box-sizing:unset!important;break-after:unset!important;break-before:unset!important;break-inside:unset!important;caption-side:unset!important;clear:unset!important;column-count:unset!important;column-fill:unset!important;column-gap:unset!important;column-rule-color:unset!important;column-rule-style:unset!important;column-rule-width:unset!important;column-span:unset!important;column-width:unset!important;empty-cells:unset!important;float:left!important;image-orientation:unset!important;isolation:unset!important;line-break:unset!important;line-height:unset!important;list-style:unset!important;mix-blend-mode:unset!important;object-fit:unset!important;object-position:unset!important;outline:unset!important;overflow:unset!important;quotes:unset!important;table-layout:unset!important;text-align:unset!important;text-align-last:unset!important;text-decoration-color:unset!important;text-decoration:unset!important;text-decoration-style:unset!important;text-emphasis-color:unset!important;text-emphasis-style:unset!important;text-indent:unset!important;text-justify:unset!important;text-overflow:unset!important;text-transform:unset!important;text-underline-position:unset!important;vertical-align:unset!important;white-space:unset!important;word-break:unset!important;border-spacing:unset!important;word-spacing:unset!important;text-combine-upright:unset!important;display:block!important;opacity:unset!important;border-radius:unset!important;height:unset!important;max-height:unset!important;min-height:unset!important;min-width:unset!important;padding:unset!important;width:unset!important;margin:0px 0px 1px!important;max-width:600px!important"><div id="m_5785087414262769895pfptBannermd4yide" style="writing-mode:unset!important;font-family:Arial,sans-serif!important;font-feature-settings:unset!important;font-kerning:unset!important;font-size:14px!important;font-size-adjust:unset!important;font-stretch:unset!important;font-variant-alternates:unset!important;font-variant-caps:unset!important;font-variant-east-asian:unset!important;font-variant-ligatures:unset!important;font-variant-numeric:unset!important;font-weight:bold!important;text-orientation:unset!important;zoom:unset!important;letter-spacing:unset!important;background-blend-mode:unset!important;background-image:unset!important;background-position:unset!important;background-repeat:unset!important;background-size:unset!important;border-collapse:unset!important;border:unset!important;box-sizing:unset!important;break-after:unset!important;break-before:unset!important;break-inside:unset!important;caption-side:unset!important;clear:unset!important;column-count:unset!important;column-fill:unset!important;column-gap:unset!important;column-rule-color:unset!important;column-rule-style:unset!important;column-rule-width:unset!important;column-span:unset!important;column-width:unset!important;empty-cells:unset!important;float:unset!important;image-orientation:unset!important;isolation:unset!important;line-break:unset!important;line-height:18px!important;list-style:unset!important;mix-blend-mode:unset!important;object-fit:unset!important;object-position:unset!important;outline:unset!important;overflow:unset!important;quotes:unset!important;table-layout:unset!important;text-align:unset!important;text-align-last:unset!important;text-decoration-color:unset!important;text-decoration:unset!important;text-decoration-style:unset!important;text-emphasis-color:unset!important;text-emphasis-style:unset!important;text-indent:unset!important;text-overflow:unset!important;text-transform:unset!important;text-underline-position:unset!important;vertical-align:unset!important;white-space:unset!important;word-break:unset!important;border-spacing:unset!important;word-spacing:unset!important;background-clip:unset!important;background-origin:unset!important;text-combine-upright:unset!important;display:block!important;opacity:unset!important;background-color:rgb(208,216,220)!important;border-radius:unset!important;height:unset!important;margin:unset!important;max-height:unset!important;max-width:unset!important;min-height:unset!important;min-width:unset!important;padding:unset!important;width:unset!important">This Message Is From an External Sender</div><div id="m_5785087414262769895pfptBannermd4yide" style="writing-mode:unset!important;font-family:Arial,sans-serif!important;font-feature-settings:unset!important;font-kerning:unset!important;font-size:12px!important;font-size-adjust:unset!important;font-stretch:unset!important;font-variant-alternates:unset!important;font-variant-caps:unset!important;font-variant-east-asian:unset!important;font-variant-ligatures:unset!important;font-variant-numeric:unset!important;font-weight:unset!important;text-orientation:unset!important;zoom:unset!important;letter-spacing:unset!important;background-blend-mode:unset!important;background-image:unset!important;background-position:unset!important;background-repeat:unset!important;background-size:unset!important;border-collapse:unset!important;border:unset!important;box-sizing:unset!important;break-after:unset!important;break-before:unset!important;break-inside:unset!important;caption-side:unset!important;clear:unset!important;column-count:unset!important;column-fill:unset!important;column-gap:unset!important;column-rule-color:unset!important;column-rule-style:unset!important;column-rule-width:unset!important;column-span:unset!important;column-width:unset!important;empty-cells:unset!important;float:unset!important;image-orientation:unset!important;isolation:unset!important;line-break:unset!important;line-height:18px!important;list-style:unset!important;mix-blend-mode:unset!important;object-fit:unset!important;object-position:unset!important;outline:unset!important;overflow:unset!important;quotes:unset!important;table-layout:unset!important;text-align:unset!important;text-align-last:unset!important;text-decoration-color:unset!important;text-decoration:unset!important;text-decoration-style:unset!important;text-emphasis-color:unset!important;text-emphasis-style:unset!important;text-indent:unset!important;text-overflow:unset!important;text-transform:unset!important;text-underline-position:unset!important;vertical-align:unset!important;white-space:unset!important;word-break:unset!important;border-spacing:unset!important;word-spacing:unset!important;background-clip:unset!important;background-origin:unset!important;text-combine-upright:unset!important;display:block!important;opacity:unset!important;background-color:rgb(208,216,220)!important;border-radius:unset!important;height:unset!important;margin-bottom:unset!important;margin-left:unset!important;margin-right:unset!important;max-height:unset!important;max-width:unset!important;min-height:unset!important;min-width:unset!important;padding:unset!important;width:unset!important;margin-top:2px!important">This message came from outside your organization.</div></div><div style="height:0px;clear:both!important;display:block!important;line-height:0!important;font-size:0.01px!important"></div></div><div dir="ltr" style="font-family:Helvetica;font-size:18px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div style="color:rgb(34,34,34)">I have an SPD dense matrix of size NxN, where N can range from 10^4-10^5. Are there any Cholesky factorization/solve routines for it in PETSc (or in any of the external libraries)? If possible, I want to use GPU acceleration with 1 or more GPUs. The matrix type can be MATSEQDENSE/MATMPIDENSE or MATSEQDENSECUDA/MATMPIDENSECUDA accordingly. If it is possible to do the factorization beforehand and store it to do the triangular solves later, that would be great.</div><div style="color:rgb(34,34,34)"><br></div><div style="color:rgb(34,34,34)">Thanks,</div><div style="color:rgb(34,34,34)">Sreeram</div></div></div></div></div></blockquote></div><br></div></div></blockquote></div>