<!-- BaNnErBlUrFlE-BoDy-start -->
<!-- Preheader Text : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">
I have removed an unnecessary PetscMPIIntCast() on MPI rank zero that was causing your test code to fail. See https: //gitlab. com/petsc/petsc/-/merge_requests/7747 Thanks for reporting the problem. Barry BTW: I don't think we have code to distribute
</div>
<!-- Preheader Text : END -->
<!-- Email Banner : BEGIN -->
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerStart</div>
<!--[if ((ie)|(mso))]>
<table border="0" cellspacing="0" cellpadding="0" width="100%" style="padding: 16px 0px 16px 0px; direction: ltr" ><tr><td>
<table border="0" cellspacing="0" cellpadding="0" style="padding: 0px 10px 5px 6px; width: 100%; border-radius:4px; border-top:4px solid #90a4ae;background-color:#D0D8DC;"><tr><td valign="top">
<table align="left" border="0" cellspacing="0" cellpadding="0" style="padding: 4px 8px 4px 8px">
<tr><td style="color:#000000; font-family: 'Arial', sans-serif; font-weight:bold; font-size:14px; direction: ltr">
This Message Is From an External Sender
</td></tr>
<tr><td style="color:#000000; font-weight:normal; font-family: 'Arial', sans-serif; font-size:12px; direction: ltr">
This message came from outside your organization.
</td></tr>
</table>
</td></tr></table>
</td></tr></table>
<![endif]-->
<![if !((ie)|(mso))]>
<div dir="ltr" id="pfptBannersgw4dxx" style="all: revert !important; display:block !important; text-align: left !important; margin:16px 0px 16px 0px !important; padding:8px 16px 8px 16px !important; border-radius: 4px !important; min-width: 200px !important; background-color: #D0D8DC !important; background-color: #D0D8DC; border-top: 4px solid #90a4ae !important; border-top: 4px solid #90a4ae;">
<div id="pfptBannersgw4dxx" style="all: unset !important; float:left !important; display:block !important; margin: 0px 0px 1px 0px !important; max-width: 600px !important;">
<div id="pfptBannersgw4dxx" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-weight:bold !important; font-weight:bold; font-size:14px !important; line-height:18px !important; line-height:18px">
This Message Is From an External Sender
</div>
<div id="pfptBannersgw4dxx" style="all: unset !important; display:block !important; visibility: visible !important; background-color: #D0D8DC !important; color:#000000 !important; color:#000000; font-weight:normal; font-family: 'Arial', sans-serif !important; font-family: 'Arial', sans-serif; font-size:12px !important; line-height:18px !important; line-height:18px; margin-top:2px !important;">
This message came from outside your organization.
</div>
</div>
<div style="clear: both !important; display: block !important; visibility: hidden !important; line-height: 0 !important; font-size: 0.01px !important; height: 0px"> </div>
</div>
<![endif]>
<div style="display:none !important;display:none;visibility:hidden;mso-hide:all;font-size:1px;color:#ffffff;line-height:1px;height:0px;max-height:0px;opacity:0;overflow:hidden;">ZjQcmQRYFpfptBannerEnd</div>
<!-- Email Banner : END -->
<!-- BaNnErBlUrFlE-BoDy-end -->
<html><head><!-- BaNnErBlUrFlE-HeAdEr-start -->
<style>
#pfptBannersgw4dxx { all: revert !important; display: block !important;
visibility: visible !important; opacity: 1 !important;
background-color: #D0D8DC !important;
max-width: none !important; max-height: none !important }
.pfptPrimaryButtonsgw4dxx:hover, .pfptPrimaryButtonsgw4dxx:focus {
background-color: #b4c1c7 !important; }
.pfptPrimaryButtonsgw4dxx:active {
background-color: #90a4ae !important; }
</style>
<!-- BaNnErBlUrFlE-HeAdEr-end -->
<meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div><br></div> I have removed an unnecessary PetscMPIIntCast() on MPI rank zero that was causing your test code to fail. See <a href="https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7747__;!!G_uCfscf7eWS!fyVpTZyH7SS1nVHiZBR-6MSWa6uJ0mSExyg1aNmU4PyWfukw1682_dX9rwUKstiGP6Z8i22L4pmElEi9qsfHA6o$">https://gitlab.com/petsc/petsc/-/merge_requests/7747</a><div><br></div><div> Thanks for reporting the problem.</div><div><br></div><div> Barry</div><div><br></div><div> BTW: I don't think we have code to distribute a dense matrix that has values only on one rank to all the ranks. The code would essentially like the combination of </div><div>of MatView_Dense_Binary/MatLoad_Dense_Binary with PetscViewerBinaryWriteReadAll without the saving and reading from disk.</div><div><br></div><div> It is likely relatively easy to fix the dense matrix view/load with native format so that it does not need 64 bit indices to work with your test code.</div><div><br></div><div><br id="lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On Aug 5, 2024, at 9:19 PM, Sreeram R Venkat <srvenkat@utexas.edu> wrote:</div><br class="Apple-interchange-newline"><div><div dir="ltr">Here's an example code that should replicate the error: <a href="https://urldefense.us/v3/__https://github.com/s769/petsc-test/tree/master__;!!G_uCfscf7eWS!fyVpTZyH7SS1nVHiZBR-6MSWa6uJ0mSExyg1aNmU4PyWfukw1682_dX9rwUKstiGP6Z8i22L4pmElEi9y0M6fi0$">https://github.com/s769/petsc-test/tree/master</a>.<div><br></div><div>I tried using the PETSC_FORMAT_NATIVE, but I still get the error. I have a situation where the matrix is created on PETSC_COMM_WORLD but only has entries on the first process due to some layout constraints elsewhere in the program. The nodes I'm running on should have more than enough memory to hold the entire matrix on one process, and the error I get is not an out-of-memory error anyway. </div><div><br></div><div>Let me know if you aren't able to build the example.</div><div><br></div><div>I noticed that if I fully distributed the matrix over all processes, then the save works fine. Is there some way to do that after I create the matrix but before saving it?</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Aug 5, 2024 at 1:19 PM Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><br></div> By default PETSc MatView() to a binary viewer uses the "standard" compressed sparse storage format. This is not efficient (or reasonable) for dense matrices and <div>produces issues with integer overflow.<br><div><br></div><div> To store a dense matrix as dense on disk, use the PetscViewerFormat of PETSC_VIEWER_NATIVE. So for example</div><div><br></div><div> PetscViewerPushFormat(viewer,PETSC_VIEWER_NATIVE);</div><div> MatView(mat, viewer);</div><div> PetscViewerPopFormat(viewer);</div><div><br id="m_3591405006712013394lineBreakAtBeginningOfMessage"><div><br><blockquote type="cite"><div>On Aug 5, 2024, at 1:10 PM, Sreeram R Venkat <<a href="mailto:srvenkat@utexas.edu" target="_blank">srvenkat@utexas.edu</a>> wrote:</div><br><div><div dir="ltr" id="m_3591405006712013394pfptBanners2fszgy" style="writing-mode:revert;color:revert;font-family:revert;font-feature-settings:revert;font-kerning:revert;font-size:revert;font-size-adjust:revert;font-stretch:revert;font-variant-alternates:revert;font-variant-caps:revert;font-variant-east-asian:revert;font-variant-ligatures:revert;font-variant-numeric:revert;font-weight:revert;text-orientation:revert;zoom:revert;letter-spacing:revert;background-blend-mode:revert;background-image:revert;background-position:revert;background-repeat:revert;background-size:revert;border-collapse:revert;box-sizing:revert;break-after:revert;break-before:revert;break-inside:revert;caption-side:revert;clear:revert;columns:revert;column-fill:revert;column-gap:revert;column-rule:revert;column-span:revert;empty-cells:revert;float:revert;image-orientation:revert;isolation:revert;line-break:revert;line-height:revert;list-style:revert;mix-blend-mode:revert;object-fit:revert;object-position:revert;outline:revert;overflow:revert;quotes:revert;table-layout:revert;text-align:left;text-align-last:revert;text-decoration:revert;text-emphasis:revert;text-indent:revert;text-overflow:revert;text-transform:revert;text-underline-position:revert;vertical-align:revert;white-space:revert;word-break:revert;border-spacing:revert;word-spacing:revert;background-clip:revert;background-origin:revert;text-combine-upright:revert;display:block;opacity:revert;background-color:rgb(208,216,220);border-bottom:revert;border-left:revert;border-right:revert;height:revert;max-height:revert;max-width:revert;min-height:revert;width:revert;margin:16px 0px;padding:8px 16px;border-radius:4px;min-width:200px;border-top:4px solid rgb(144,164,174)"><div id="m_3591405006712013394pfptBanners2fszgy" style="writing-mode:unset;color:unset;font-family:unset;font-feature-settings:unset;font-kerning:unset;font-size:unset;font-size-adjust:unset;font-stretch:unset;font-variant-alternates:unset;font-variant-caps:unset;font-variant-east-asian:unset;font-variant-ligatures:unset;font-variant-numeric:unset;font-weight:unset;text-orientation:unset;zoom:unset;letter-spacing:unset;background:unset;background-blend-mode:unset;border-collapse:unset;border:unset;box-sizing:unset;break-after:unset;break-before:unset;break-inside:unset;caption-side:unset;clear:unset;columns:unset;column-fill:unset;column-gap:unset;column-rule:unset;column-span:unset;empty-cells:unset;float:left;image-orientation:unset;isolation:unset;line-break:unset;line-height:unset;list-style:unset;mix-blend-mode:unset;object-fit:unset;object-position:unset;outline:unset;overflow:unset;quotes:unset;table-layout:unset;text-align:unset;text-align-last:unset;text-decoration:unset;text-emphasis:unset;text-indent:unset;text-overflow:unset;text-transform:unset;text-underline-position:unset;vertical-align:unset;white-space:unset;word-break:unset;border-spacing:unset;word-spacing:unset;text-combine-upright:unset;display:block;opacity:unset;border-radius:unset;height:unset;max-height:unset;min-height:unset;min-width:unset;padding:unset;width:unset;margin:0px 0px 1px;max-width:600px"><div id="m_3591405006712013394pfptBanners2fszgy" style="writing-mode:unset;font-family:Arial,sans-serif;font-feature-settings:unset;font-kerning:unset;font-size:14px;font-size-adjust:unset;font-stretch:unset;font-variant-alternates:unset;font-variant-caps:unset;font-variant-east-asian:unset;font-variant-ligatures:unset;font-variant-numeric:unset;font-weight:bold;text-orientation:unset;zoom:unset;letter-spacing:unset;background-blend-mode:unset;background-image:unset;background-position:unset;background-repeat:unset;background-size:unset;border-collapse:unset;border:unset;box-sizing:unset;break-after:unset;break-before:unset;break-inside:unset;caption-side:unset;clear:unset;columns:unset;column-fill:unset;column-gap:unset;column-rule:unset;column-span:unset;empty-cells:unset;float:unset;image-orientation:unset;isolation:unset;line-break:unset;line-height:18px;list-style:unset;mix-blend-mode:unset;object-fit:unset;object-position:unset;outline:unset;overflow:unset;quotes:unset;table-layout:unset;text-align:unset;text-align-last:unset;text-decoration:unset;text-emphasis:unset;text-indent:unset;text-overflow:unset;text-transform:unset;text-underline-position:unset;vertical-align:unset;white-space:unset;word-break:unset;border-spacing:unset;word-spacing:unset;background-clip:unset;background-origin:unset;text-combine-upright:unset;display:block;opacity:unset;background-color:rgb(208,216,220);border-radius:unset;height:unset;margin:unset;max-height:unset;max-width:unset;min-height:unset;min-width:unset;padding:unset;width:unset">This Message Is From an External Sender</div><div id="m_3591405006712013394pfptBanners2fszgy" style="writing-mode:unset;font-family:Arial,sans-serif;font-feature-settings:unset;font-kerning:unset;font-size:12px;font-size-adjust:unset;font-stretch:unset;font-variant-alternates:unset;font-variant-caps:unset;font-variant-east-asian:unset;font-variant-ligatures:unset;font-variant-numeric:unset;font-weight:unset;text-orientation:unset;zoom:unset;letter-spacing:unset;background-blend-mode:unset;background-image:unset;background-position:unset;background-repeat:unset;background-size:unset;border-collapse:unset;border:unset;box-sizing:unset;break-after:unset;break-before:unset;break-inside:unset;caption-side:unset;clear:unset;columns:unset;column-fill:unset;column-gap:unset;column-rule:unset;column-span:unset;empty-cells:unset;float:unset;image-orientation:unset;isolation:unset;line-break:unset;line-height:18px;list-style:unset;mix-blend-mode:unset;object-fit:unset;object-position:unset;outline:unset;overflow:unset;quotes:unset;table-layout:unset;text-align:unset;text-align-last:unset;text-decoration:unset;text-emphasis:unset;text-indent:unset;text-overflow:unset;text-transform:unset;text-underline-position:unset;vertical-align:unset;white-space:unset;word-break:unset;border-spacing:unset;word-spacing:unset;background-clip:unset;background-origin:unset;text-combine-upright:unset;display:block;opacity:unset;background-color:rgb(208,216,220);border-radius:unset;height:unset;margin-bottom:unset;margin-left:unset;margin-right:unset;max-height:unset;max-width:unset;min-height:unset;min-width:unset;padding:unset;width:unset;margin-top:2px">This message came from outside your organization.</div></div><div style="height:0px;clear:both;display:block;line-height:0;font-size:0.01px"></div></div><div dir="ltr" style="font-family:Helvetica;font-size:18px;font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration:none"><div dir="ltr" class="gmail_signature"><div dir="ltr"><div style="color:rgb(34,34,34)">I have a large dense matrix (size ranging from 5e4 to 1e5) that arises as a result of doing MatComputeOperator() on a MatShell. When the total number of nonzeros exceeds the 32 bit integer value, I get an error (MPI buffer size too big) when trying to do MatView() on this to save to binary. Is there a way I can save this matrix to load again for later use? </div><div style="color:rgb(34,34,34)"><br></div><div style="color:rgb(34,34,34)">The other thing I tried was to save each column as a separate dataset in an hdf5 file. Then, I tried to load this in python, combine them to an np array, and then create/save a dense matrix with petsc4py. I was able to create the dense Mat, but the MatView() once again resulted in an error (out of memory). </div><div style="color:rgb(34,34,34)"><br></div><div style="color:rgb(34,34,34)">Thanks,</div><div style="color:rgb(34,34,34)">Sreeram</div></div></div></div></div></blockquote></div><br></div></div></div></blockquote></div>
</div></blockquote></div><br></div></body></html>