<div dir="ltr">Dear Matt, petsc-users:<div><br></div><div>Finally back after the holidays to try to solve this issue, thanks for your patience!</div><div>I compiled the latest petsc (3.12.3) with debugging enabled, the same problem appears: relatively large matrices result in out of memory errors. This is not the case for petsc-3.9.4, all fine there.</div><div>This is a non-hermitian, generalized eigenvalue problem, I generate the A and B matrices myself and then I use example 7 (from the slepc tutorial at $SLEPC_DIR/src/eps/examples/tutorials/ex7.c ) to solve the problem:</div><div><br></div><div>mpiexec -n 24 valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log.%p ./ex7 -malloc off -f1 A.petsc -f2 B.petsc -eps_nev 1 -eps_target -2.5e-4+1.56524i -eps_target_magnitude -eps_tol 1e-14 $opts<br></div><div><br></div><div>where the $opts variable is:</div><div>export opts='-st_type sinvert -st_ksp_type preonly -st_pc_type lu -eps_error_relative ::ascii_info_detail -st_pc_factor_mat_solver_type superlu_dist -mat_superlu_dist_iterrefine 1 -mat_superlu_dist_colperm PARMETIS -mat_superlu_dist_parsymbfact 1 -eps_converged_reason -eps_conv_rel -eps_monitor_conv -eps_true_residual 1'<br></div><div><br></div><div>the output from valgrind (sample from one processor) and from the program are attached.</div><div>If it's of any use the matrices are here (might need at least 180 Gb of ram to solve the problem succesfully under petsc-3.9.4):</div><div><br></div><div><a href="https://www.dropbox.com/s/as9bec9iurjra6r/A.petsc?dl=0">https://www.dropbox.com/s/as9bec9iurjra6r/A.petsc?dl=0</a><br></div><div><a href="https://www.dropbox.com/s/u2bbmng23rp8l91/B.petsc?dl=0">https://www.dropbox.com/s/u2bbmng23rp8l91/B.petsc?dl=0</a></div><div><br></div><div>WIth petsc-3.9.4 and slepc-3.9.2 I can use matrices up to 10Gb (with 240 Gb ram), but only up to 3Gb with the latest petsc/slepc.</div><div>Any suggestions, comments or any other help are very much appreciated!</div><div><br></div><div>Cheers,</div><div>Santiago</div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Dec 23, 2019 at 11:19 PM Matthew Knepley <<a href="mailto:knepley@gmail.com">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Mon, Dec 23, 2019 at 3:14 PM Santiago Andres Triana <<a href="mailto:repepo@gmail.com" target="_blank">repepo@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Dear all,<div><br></div><div>After upgrading to petsc 3.12.2 my solver program crashes consistently. Before the upgrade I was using petsc 3.9.4 with no problems.</div><div><br></div><div>My application deals with a complex-valued, generalized eigenvalue problem. The matrices involved are relatively large, typically 2 to 10 Gb in size, which is no problem for petsc 3.9.4.</div></div></blockquote><div><br></div><div>Are you sure that your indices do not exceed 4B? If so, you need to configure using</div><div><br></div><div> --with-64-bit-indices</div><div><br></div><div>Also, it would be nice if you ran with the debugger so we can get a stack trace for the SEGV.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>However, after the upgrade I can only obtain solutions when the matrices are small, the solver crashes when the matrices' size exceed about 1.5 Gb:</div><div><br></div><div>[0]PETSC ERROR: ------------------------------------------------------------------------<br>[0]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end<br>[0]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>[0]PETSC ERROR: or see <a href="https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" target="_blank">https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a><br>[0]PETSC ERROR: or try <a href="http://valgrind.org" target="_blank">http://valgrind.org</a> on GNU/linux and Apple Mac OS X to find memory corruption errors<br>[0]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run <br>[0]PETSC ERROR: to get more information on the crash.<br></div><div><br></div><div>and so on for each cpu.</div><div><br></div><div><br></div><div>I tried using valgrind and this is the typical output:</div><div><br></div><div>==2874== Conditional jump or move depends on uninitialised value(s)<br>==2874== at 0x4018178: index (in /lib64/<a href="http://ld-2.22.so" target="_blank">ld-2.22.so</a>)<br>==2874== by 0x400752D: expand_dynamic_string_token (in /lib64/<a href="http://ld-2.22.so" target="_blank">ld-2.22.so</a>)<br>==2874== by 0x4008009: _dl_map_object (in /lib64/<a href="http://ld-2.22.so" target="_blank">ld-2.22.so</a>)<br>==2874== by 0x40013E4: map_doit (in /lib64/<a href="http://ld-2.22.so" target="_blank">ld-2.22.so</a>)<br>==2874== by 0x400EA53: _dl_catch_error (in /lib64/<a href="http://ld-2.22.so" target="_blank">ld-2.22.so</a>)<br>==2874== by 0x4000ABE: do_preload (in /lib64/<a href="http://ld-2.22.so" target="_blank">ld-2.22.so</a>)<br>==2874== by 0x4000EC0: handle_ld_preload (in /lib64/<a href="http://ld-2.22.so" target="_blank">ld-2.22.so</a>)<br>==2874== by 0x40034F0: dl_main (in /lib64/<a href="http://ld-2.22.so" target="_blank">ld-2.22.so</a>)<br>==2874== by 0x4016274: _dl_sysdep_start (in /lib64/<a href="http://ld-2.22.so" target="_blank">ld-2.22.so</a>)<br>==2874== by 0x4004A99: _dl_start (in /lib64/<a href="http://ld-2.22.so" target="_blank">ld-2.22.so</a>)<br>==2874== by 0x40011F7: ??? (in /lib64/<a href="http://ld-2.22.so" target="_blank">ld-2.22.so</a>)<br>==2874== by 0x12: ???<br>==2874== <br></div><div><br></div><div><br></div><div>These are my configuration options. Identical for both petsc 3.9.4 and 3.12.2:</div><div><br></div><div>./configure --with-scalar-type=complex --download-mumps --download-parmetis --download-metis --download-scalapack=1 --download-fblaslapack=1 --with-debugging=0 --download-superlu_dist=1 --download-ptscotch=1 CXXOPTFLAGS='-O3 -march=native' FOPTFLAGS='-O3 -march=native' COPTFLAGS='-O3 -march=native'<br></div><div><br></div><div><br></div><div>Thanks in advance for any comments or ideas!</div><div><br></div><div>Cheers,</div><div>Santiago</div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div>