<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""></div><div> <a href="https://www.mcs.anl.gov/petsc/documentation/faq.html#efficient-assembly" class="">https://www.mcs.anl.gov/petsc/documentation/faq.html#efficient-assembly</a> Perhaps we should provide more information at this FAQ to help track down such issues.</div><div><br class=""></div><div><br class=""><blockquote type="cite" class=""><div class="">On Jul 2, 2020, at 8:10 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" class="">knepley@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div dir="ltr" class="">On Thu, Jul 2, 2020 at 7:30 PM Karl Lin <<a href="mailto:karl.linkui@gmail.com" class="">karl.linkui@gmail.com</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div dir="ltr" class=""><div class="">Hi, Matt</div><div class=""><br class=""></div><div class="">Thanks for the tip last time. We just encountered another issue with large data sets. This time the behavior is the opposite from last time. The data is 13.5TB, the total number of matrix columns is 2.4 billion. Our program crashed during matrix loading due to memory overflow in one node. As said before, we have a little memory check during loading the matrix to keep track of rss. The printout of rss in the log shows normal increase in many nodes, i.e., if we load in a portion of the matrix that is 1GB, after MatSetValues for that portion, rss will increase roughly about 1GB. On the node that has memory overflow, the rss increased by 2GB after only 1GB of matrix is loaded through MatSetValues. We are very puzzled by this. What could make the memory footprint twice as much as needed? Thanks in advance for any insight.</div></div></div></blockquote><div class=""><br class=""></div><div class="">The only way I can imagine this happening is that you have not preallocated correctly, so that some values are causing additional mallocs.</div><div class=""><br class=""></div><div class=""> Thanks,</div><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div dir="ltr" class=""><div class="">Regards,</div><div class=""><br class=""></div><div class="">Karl <span class=""><span class=""><br class=""></span></span></div></div></div><br class=""><div class="gmail_quote"><div class="gmail_attr" dir="ltr">On Thu, Jun 11, 2020 at 12:00 PM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" class="">knepley@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left:1px solid rgb(204,204,204)"><div dir="ltr" class=""><div dir="ltr" class="">On Thu, Jun 11, 2020 at 12:52 PM Karl Lin <<a href="mailto:karl.linkui@gmail.com" target="_blank" class="">karl.linkui@gmail.com</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left:1px solid rgb(204,204,204)"><div dir="ltr" class=""><div dir="ltr" class=""><div class="">Hi, Matthew</div><div class=""><br class=""></div><div class="">Thanks for the suggestion, just did another run and here are some detailed stack traces, maybe will provide some more insight:</div><div class=""> *** Process received signal ***<br class="">Signal: Aborted (6)<br class="">Signal code: (-6)<br class="">/lib64/libpthread.so.0(+0xf5f0)[0x2b56c46dc5f0]</div><div class=""> [ 1] /lib64/libc.so.6(gsignal+0x37)[0x2b56c5486337]<br class=""> [ 2] /lib64/libc.so.6(abort+0x148)[0x2b56c5487a28]<br class=""> [ 3] /libpetsc.so.3.10(PetscTraceBackErrorHandler+0xc4)[0x2b56c1e6a2d4]<br class=""> [ 4] /libpetsc.so.3.10(PetscError+0x1b5)[0x2b56c1e69f65]<br class=""> [ 5] /libpetsc.so.3.10(PetscCommBuildTwoSidedFReq+0x19f0)[0x2b56c1e03cf0]<br class=""> [ 6] /libpetsc.so.3.10(+0x77db17)[0x2b56c2425b17]<br class=""> [ 7] /libpetsc.so.3.10(+0x77a164)[0x2b56c2422164]<br class=""> [ 8] /libpetsc.so.3.10(MatAssemblyBegin_MPIAIJ+0x36)[0x2b56c23912b6]<br class=""> [ 9] /libpetsc.so.3.10(MatAssemblyBegin+0xca)[0x2b56c1feccda]<br class=""></div><div class=""><br class=""></div><div class="">By reconfiguring, you mean recompiling petsc with that option, correct?</div></div></div></blockquote><div class=""><br class=""></div><div class="">Reconfiguring.</div><div class=""><br class=""></div><div class=""> Thanks,</div><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left:1px solid rgb(204,204,204)"><div dir="ltr" class=""><div dir="ltr" class=""><div class="">Thank you.</div><div class=""><br class=""></div><div class="">Karl<span class=""><span class=""><br class=""></span></span></div></div></div><br class=""><div class="gmail_quote"><div class="gmail_attr" dir="ltr">On Thu, Jun 11, 2020 at 10:56 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank" class="">knepley@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left:1px solid rgb(204,204,204)"><div dir="ltr" class=""><div dir="ltr" class="">On Thu, Jun 11, 2020 at 11:51 AM Karl Lin <<a href="mailto:karl.linkui@gmail.com" target="_blank" class="">karl.linkui@gmail.com</a>> wrote:<br class=""></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left:1px solid rgb(204,204,204)"><div dir="ltr" class=""><div class="">Hi, there</div><div class=""><br class=""></div><div class="">We have written a program using Petsc to solve large sparse matrix system. It has been working fine for a while. Recently we encountered a problem when the size of the sparse matrix is larger than 10TB. We used several hundred nodes and 2200 processes. The program always crashes during MatAssemblyBegin.Upon a closer look, there seems to be something unusual. We have a little memory check during loading the matrix to keep track of rss. The printout of rss in the log shows normal increase up to rank 2160, i.e., if we load in a portion of matrix that is 1GB, after MatSetValues for that portion, rss will increase roughly about that number. From rank 2161 onwards, the rss in every rank doesn't increase after matrix loaded. Then comes MatAssemblyBegin, the program crashed on rank 2160. </div><div class=""><br class=""></div><div class="">Is there a upper limit on the number of processes Petsc can handle? or is there a upper limit in terms of the size of the matrix petsc can handle? Thank you very much for any info.</div></div></blockquote><div class=""><br class=""></div><div class="">It sounds like you overflowed int somewhere. We try and check for this, but catching every place is hard. Try reconfiguring with</div><div class=""><br class=""></div><div class=""> --with-64-bit-indices</div><div class=""><br class=""></div><div class=""> Thanks,</div><div class=""><br class=""></div><div class=""> Matt</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left:1px solid rgb(204,204,204)"><div dir="ltr" class=""><div class="">Regards,</div><div class=""><br class=""></div><div class="">Karl </div></div>
</blockquote></div><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class=""></div></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div dir="ltr" class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class=""></div></div></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div dir="ltr" class="gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br class="">-- Norbert Wiener</div><div class=""><br class=""></div><div class=""><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank" class="">https://www.cse.buffalo.edu/~knepley/</a><br class=""></div></div></div></div></div></div></div></div>
</div></blockquote></div><br class=""></body></html>