<div dir="ltr"><div>On Wed, Jul 22, 2015 at 8:39 AM, Michael Augspurger <span dir="ltr"><<a href="mailto:michaelaugspurger@gmail.com" target="_blank">michaelaugspurger@gmail.com</a>></span> wrote:<br></div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>Hello:<br><br></div>I'm having a problem that I'm having a rough time diagnosing. My CFD simulation code will run for a long time, sometimes up to 10K steps, and then suddenly I'll get a SEGV error (If I run the same simulation again, I'll get the same error, but always at a different time step, sometimes thousands of steps different). There's nothing obvious going wrong in the simulation at the time. Valgrind points to various internal petsc operations, but the trail doesn't lead back to any part of my code, so I'm not sure where to go next.<br> <br></div><div>Any advice or experience about where I can continue my investigation into this failure? Thanks for any help,<br><br></div><div>Mike Augspurger<br></div><div><br><br><br></div>Here's part of the error code with valgrind:<br><br> Residual norms for pres_redistribute_ solve.<br> 0 KSP Residual norm 2.343992292214e+00<br> 1 KSP Residual norm 3.714369184378e-01<br> 2 KSP Residual norm 5.045817070946e-02<br>[2]PETSC ERROR: ------------------------------------------------------------------------<br>[2]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range<br>[2]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>[2]PETSC ERROR: or see <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" target="_blank">http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a><br>[2]PETSC ERROR: or try <a href="http://valgrind.org" target="_blank">http://valgrind.org</a> on GNU/linux and Apple Mac OS X to find memory corruption errors<br>[2]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run<br>[2]PETSC ERROR: to get more information on the crash.<br>[2]PETSC ERROR: User provided function() line 0 in unknown file<br>==11381==<br>==11381== Process terminating with default action of signal 11 (SIGSEGV)<br>==11381== General Protection Fault<br>==11381== at 0x926047B: __intel_sse2_strcat (in /opt/intel/composer_xe_2013.2.146/compiler/lib/intel64/libintlc.so.5)<br>==11381== by 0x817475E: opal_os_path (os_path.c:99)<br>==11381== by 0x817B9B0: opal_show_help_vstring (show_help.c:153)<br>==11381== by 0x80F7878: orte_show_help (show_help.c:566)<br>==11381== by 0x80A7FFC: warn_fork_cb (ompi_mpi_init.c:139)<br>==11381== by 0x3E4549A285: fork (in /lib64/<a href="http://libc-2.5.so" target="_blank">libc-2.5.so</a>)<br>==11381== by 0x4D6A793: PetscAttachDebugger (in /Users/augspurger/NumericalLibraries/petsc/intel-opt/lib/libpetsc.so.3.6.0)<br>==11381== by 0x4D6B93E: PetscAttachDebuggerErrorHandler (in /Users/augspurger/NumericalLibraries/petsc/intel-opt/lib/libpetsc.so.3.6.0)<br>==11381== by 0x4D6E5BC: PetscError (in /Users/augspurger/NumericalLibraries/petsc/intel-opt/lib/libpetsc.so.3.6.0)<br>==11381== by 0x4D70024: PetscSignalHandlerDefault (in /Users/augspurger/NumericalLibraries/petsc/intel-opt/lib/libpetsc.so.3.6.0)<br>==11381== by 0x4D6F9F3: PetscSignalHandler_Private (in /Users/augspurger/NumericalLibraries/petsc/intel-opt/lib/libpetsc.so.3.6.0)<br></div></blockquote><div><br></div><div>Can you rerun this without -on_error_debugger_attach? And send ALL the output. We need to see what valgrind thinks</div><div>is the real problem.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">==11381== by 0x3E4543002F: ??? (in /lib64/<a href="http://libc-2.5.so" target="_blank">libc-2.5.so</a>)<br>==11381==<br>==11381== HEAP SUMMARY:<br>==11381== in use at exit: 50,117,187 bytes in 90,510 blocks<br>==11381== total heap usage: 136,533,481 allocs, 136,442,971 frees, 85,265,006,726 bytes allocated<br>==11381==<br>==11381== 2 bytes in 1 blocks are definitely lost in loss record 5 of 4,925<br>==11381== at 0x4A0646F: malloc (vg_replace_malloc.c:236)<br>==11381== by 0x926098D: __intel_sse2_strdup (in /opt/intel/composer_xe_2013.2.146/compiler/lib/intel64/libintlc.so.5)<br>==11381== by 0x5F6574726F5F4142: ???<br>==11381== by 0x747365725F6D756D: ???<br>==11381== by 0x4F00303D73747260: ???<br>==11381== by 0x5054554F5F4C414F: ???<br>==11381== by 0x52454454535F5454: ???<br>==11381== by 0x32333D44465F51: ???<br>==11381== by 0x41434D5F49504D4E: ???<br>==11381== by 0x696E69666661705E: ???<br>==11381== by 0x5F657361625F7973: ???<br>==11381== by 0x313D646E756F61: ???<br>==11381==<br>==11381== 9 bytes in 1 blocks are definitely lost in loss record 430 of 4,925<br>==11381== at 0x4A0646F: malloc (vg_replace_malloc.c:236)<br>==11381== by 0x926098D: __intel_sse2_strdup (in /opt/intel/composer_xe_2013.2.146/compiler/lib/intel64/libintlc.so.5)<br>==11381== by 0x2020200A3E746364: ???<br>==11381== by 0x3C2020202020201F: ???<br>==11381== by 0x3E7463656A626F2E: ???<br>==11381== by 0x2020202020202009: ???<br>==11381== by 0xF3: ???<br>==11381== by 0xF3: ???<br>==11381== by 0x3: ???<br>==11381== by 0x3: ???<br>==11381== by 0xE4: ???<br>==11381== by 0xE5: ???<br>==11381==<br>==11381== 11 bytes in 1 blocks are definitely lost in loss record 472 of 4,925<br>==11381== at 0x4A0646F: malloc (vg_replace_malloc.c:236)<br>==11381== by 0x812B188: opal_argv_join (argv.c:269)<br>==11381== by 0xD4F5370: ompi_btl_openib_connect_base_register (btl_openib_connect_base.c:72)<br>==11381== by 0xD4F0CB0: btl_openib_register_mca_params (btl_openib_mca.c:652)<br>==11381== by 0xD4E24B5: btl_openib_component_register (btl_openib_component.c:166)<br>==11381== by 0x815DCC5: mca_base_components_open (mca_base_components_open.c:387)<br>==11381== by 0x80D7140: mca_btl_base_open (btl_base_open.c:115)<br>==11381== by 0xC4612C6: ???<br>==11381== by 0x815DD37: mca_base_components_open (mca_base_components_open.c:427)<br>==11381== by 0x80E4CCA: mca_pml_base_open (pml_base_open.c:126)<br>==11381== by 0x80A7594: ompi_mpi_init (ompi_mpi_init.c:485)<br>==11381== by 0x80BF902: PMPI_Init (pinit.c:84)<br>==11381==<br>==11381== 16 bytes in 1 blocks are definitely lost in loss record 710 of 4,925<br>==11381== at 0x4A0646F: malloc (vg_replace_malloc.c:236)<br>==11381== by 0x813EE92: opal_dss_unpack_byte_object (dss_unpack.c:490)<br>==11381== by 0x813F3AE: opal_dss_unpack_buffer (dss_unpack.c:120)<br>==11381== by 0x813DBD9: opal_dss_unpack (dss_unpack.c:84)<br>==11381== by 0x81021EC: orte_util_nidmap_init (nidmap.c:117)<br>==11381== by 0xAA17573: rte_init (ess_env_module.c:173)<br>==11381== by 0x80E75CA: orte_init (orte_init.c:127)<br>==11381== by 0x80A7005: ompi_mpi_init (ompi_mpi_init.c:357)<br>==11381== by 0x80BF902: PMPI_Init (pinit.c:84)<br>==11381== by 0x71F3956: MPI_INIT (pinit_f.c:75)<br>==11381== by 0x4D0F99F: petscinitialize_ (in /Users/augspurger/NumericalLibraries/petsc/intel-opt/lib/libpetsc.so.3.6.0)<br>==11381== by 0x60B1AB: elafintstartmpi_ (in /nfsscratch/Users/augspurger/PAPER2/PELAFINT3D_EXE)<br>==11381==<br>==11381== 16 bytes in 1 blocks are definitely lost in loss record 711 of 4,925<br>==11381== at 0x4A0646F: malloc (vg_replace_malloc.c:236)<br>==11381== by 0x813EE92: opal_dss_unpack_byte_object (dss_unpack.c:490)<br>==11381== by 0x813F3AE: opal_dss_unpack_buffer (dss_unpack.c:120)<br>==11381== by 0x813DBD9: opal_dss_unpack (dss_unpack.c:84)<br>==11381== by 0x810222C: orte_util_nidmap_init (nidmap.c:130)<br>==11381== by 0xAA17573: rte_init (ess_env_module.c:173)<br>==11381== by 0x80E75CA: orte_init (orte_init.c:127)<br>==11381== by 0x80A7005: ompi_mpi_init (ompi_mpi_init.c:357)<br>==11381== by 0x80BF902: PMPI_Init (pinit.c:84)<br>==11381== by 0x71F3956: MPI_INIT (pinit_f.c:75)<br>==11381== by 0x4D0F99F: petscinitialize_ (in /Users/augspurger/NumericalLibraries/petsc/intel-opt/lib/libpetsc.so.3.6.0)<br>==11381== by 0x60B1AB: elafintstartmpi_ (in /nfsscratch/Users/augspurger/PAPER2/PELAFINT3D_EXE)<br>==11381==<br>==11381== 16 bytes in 16 blocks are definitely lost in loss record 712 of 4,925<br>==11381== at 0x4A0646F: malloc (vg_replace_malloc.c:236)<br>==11381== by 0x810F37B: orte_grpcomm_base_get_proc_attr (grpcomm_base_modex.c:801)<br>==11381== by 0x8098A44: ompi_comm_cid_init (comm_cid.c:139)<br>==11381== by 0x80A7C52: ompi_mpi_init (ompi_mpi_init.c:846)<br>==11381== by 0x80BF902: PMPI_Init (pinit.c:84)<br>==11381== by 0x71F3956: MPI_INIT (pinit_f.c:75)<br>==11381== by 0x4D0F99F: petscinitialize_ (in /Users/augspurger/NumericalLibraries/petsc/intel-opt/lib/libpetsc.so.3.6.0)<br>==11381== by 0x60B1AB: elafintstartmpi_ (in /nfsscratch/Users/augspurger/PAPER2/PELAFINT3D_EXE)<br>==11381== by 0x60A052: MAIN__ (in /nfsscratch/Users/augspurger/PAPER2/PELAFINT3D_EXE)<br>==11381== by 0x42412B: main (in /nfsscratch/Users/augspurger/PAPER2/PELAFINT3D_EXE)<br>==11381==<br><br></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</div></div>