<div dir="ltr">The first thing I would do is reduce to serial until it works there. It makes debugging 100x easier.<div><br></div><div> THanks,</div><div><br></div><div> Matt</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jul 6, 2020 at 2:14 PM Eda Oktay <<a href="mailto:eda.oktay@metu.edu.tr">eda.oktay@metu.edu.tr</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello,<br>
<br>
I am trying to run my codes 100 times and then take the average<br>
elapsed time. Although my program runs correctly for single time, when<br>
I put them into loop, I am getting the following error:<br>
<br>
*** Error in `./avg_coloredge_deneme_new_vecscatter_arastep':<br>
munmap_chunk(): invalid pointer: 0x0000000000b0cbc0 ***<br>
[1]PETSC ERROR:<br>
------------------------------------------------------------------------<br>
[1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,<br>
probably memory access out of range<br>
[1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>
[1]PETSC ERROR: or see<br>
<a href="https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind" rel="noreferrer" target="_blank">https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind</a><br>
[1]PETSC ERROR: or try <a href="http://valgrind.org" rel="noreferrer" target="_blank">http://valgrind.org</a> on GNU/linux and Apple Mac<br>
OS X to find memory corruption errors<br>
[1]PETSC ERROR: likely location of problem given in stack below<br>
[1]PETSC ERROR: --------------------- Stack Frames<br>
------------------------------------<br>
[1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,<br>
[1]PETSC ERROR: INSTEAD the line number of the start of the function<br>
[1]PETSC ERROR: is given.<br>
======= Backtrace: =========<br>
[1]PETSC ERROR: /lib64/libc.so.6(+0x791eb)[0x7f810f22a1eb]<br>
--------------------- Error Message<br>
--------------------------------------------------------------<br>
[1]PETSC ERROR: /lib64/libc.so.6(cfree+0x1f8)[0x7f810f237438]<br>
Signal received<br>
[1]PETSC ERROR: See<br>
<a href="https://www.mcs.anl.gov/petsc/documentation/faq.html" rel="noreferrer" target="_blank">https://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble<br>
shooting.<br>
[1]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020<br>
[1]PETSC ERROR: ./avg_coloredge_deneme_new_vecscatter_arastep on a<br>
arch-linux2-c-debug named localhost.localdomain by edaoktay Mon Jul 6<br>
21:09:07 2020<br>
[1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++<br>
--download-mpich --download-openblas --download-slepc --download-metis<br>
--download-parmetis --download-chaco --with-X=1<br>
[1]PETSC ERROR: #1 User provided function() line 0 in unknown file<br>
./avg_coloredge_deneme_new_vecscatter_arastep[0x402dca]<br>
./avg_coloredge_deneme_new_vecscatter_arastep[0x402f1e]<br>
./avg_coloredge_deneme_new_vecscatter_arastep[0x402953]<br>
./avg_coloredge_deneme_new_vecscatter_arastep[0x406c1c]<br>
application called MPI_Abort(MPI_COMM_WORLD, 50152059) - process 1<br>
/lib64/libc.so.6(__libc_start_main+0xf1)[0x7f810f1d1431]<br>
./avg_coloredge_deneme_new_vecscatter_arastep[0x40260a]<br>
======= Memory map: ========<br>
00400000-00409000 r-xp 00000000 fd:02 29230434<br>
/home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/share/slepc/examples/src/eda/avg_coloredge_deneme_new_vecscatter_arastep<br>
00608000-00609000 r--p 00008000 fd:02 29230434<br>
/home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/share/slepc/examples/src/eda/avg_coloredge_deneme_new_vecscatter_arastep<br>
00609000-0060a000 rw-p 00009000 fd:02 29230434<br>
/home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/share/slepc/examples/src/eda/avg_coloredge_deneme_new_vecscatter_arastep<br>
00956000-01100000 rw-p 00000000 00:00 0 [heap]<br>
7f810c621000-7f810c6ca000 rw-p 00000000 00:00 0<br>
7f810c6ca000-7f810e6ca000 rw-p 00000000 00:00 0<br>
7f810e6ca000-7f810e6d4000 r-xp 00000000 fd:00 1974339<br>
/usr/lib64/<a href="http://libnss_files-2.24.so" rel="noreferrer" target="_blank">libnss_files-2.24.so</a><br>
7f810e6d4000-7f810e8d4000 ---p 0000a000 fd:00 1974339<br>
/usr/lib64/<a href="http://libnss_files-2.24.so" rel="noreferrer" target="_blank">libnss_files-2.24.so</a><br>
7f810e8d4000-7f810e8d5000 r--p 0000a000 fd:00 1974339<br>
/usr/lib64/<a href="http://libnss_files-2.24.so" rel="noreferrer" target="_blank">libnss_files-2.24.so</a><br>
7f810e8d5000-7f810e8d6000 rw-p 0000b000 fd:00 1974339<br>
/usr/lib64/<a href="http://libnss_files-2.24.so" rel="noreferrer" target="_blank">libnss_files-2.24.so</a><br>
7f810e8d6000-7f810eb7d000 rw-p 00000000 00:00 0<br>
7f810eb7d000-7f810eb7f000 r-xp 00000000 fd:00 1974984<br>
/usr/lib64/libXau.so.6.0.0<br>
7f810eb7f000-7f810ed7f000 ---p 00002000 fd:00 1974984<br>
/usr/lib64/libXau.so.6.0.0<br>
7f810ed7f000-7f810ed80000 r--p 00002000 fd:00 1974984<br>
/usr/lib64/libXau.so.6.0.0<br>
7f810ed80000-7f810ed81000 rw-p 00000000 00:00 0<br>
7f810ed81000-7f810ed88000 r-xp 00000000 fd:00 1974440<br>
/usr/lib64/<a href="http://librt-2.24.so" rel="noreferrer" target="_blank">librt-2.24.so</a><br>
7f810ed88000-7f810ef87000 ---p 00007000 fd:00 1974440<br>
/usr/lib64/<a href="http://librt-2.24.so" rel="noreferrer" target="_blank">librt-2.24.so</a><br>
7f810ef87000-7f810ef88000 r--p 00006000 fd:00 1974440<br>
/usr/lib64/<a href="http://librt-2.24.so" rel="noreferrer" target="_blank">librt-2.24.so</a><br>
7f810ef88000-7f810ef89000 rw-p 00007000 fd:00 1974440<br>
/usr/lib64/<a href="http://librt-2.24.so" rel="noreferrer" target="_blank">librt-2.24.so</a><br>
7f810ef89000-7f810efb0000 r-xp 00000000 fd:00 1975953<br>
/usr/lib64/libxcb.so.1.1.0<br>
7f810efb0000-7f810f1af000 ---p 00027000 fd:00 1975953<br>
/usr/lib64/libxcb.so.1.1.0<br>
7f810f1af000-7f810f1b0000 r--p 00026000 fd:00 1975953<br>
/usr/lib64/libxcb.so.1.1.0<br>
7f810f1b0000-7f810f1b1000 rw-p 00027000 fd:00 1975953<br>
/usr/lib64/libxcb.so.1.1.0<br>
7f810f1b1000-7f810f36e000 r-xp 00000000 fd:00 1973798<br>
/usr/lib64/<a href="http://libc-2.24.so" rel="noreferrer" target="_blank">libc-2.24.so</a><br>
7f810f36e000-7f810f56d000 ---p 001bd000 fd:00 1973798<br>
/usr/lib64/<a href="http://libc-2.24.so" rel="noreferrer" target="_blank">libc-2.24.so</a><br>
7f810f56d000-7f810f571000 r--p 001bc000 fd:00 1973798<br>
/usr/lib64/<a href="http://libc-2.24.so" rel="noreferrer" target="_blank">libc-2.24.so</a><br>
7f810f571000-7f810f573000 rw-p 001c0000 fd:00 1973798<br>
/usr/lib64/<a href="http://libc-2.24.so" rel="noreferrer" target="_blank">libc-2.24.so</a><br>
7f810f573000-7f810f577000 rw-p 00000000 00:00 0<br>
7f810f577000-7f810f5b6000 r-xp 00000000 fd:00 1975094<br>
/usr/lib64/libquadmath.so.0.0.0<br>
7f810f5b6000-7f810f7b5000 ---p 0003f000 fd:00 1975094<br>
/usr/lib64/libquadmath.so.0.0.0<br>
7f810f7b5000-7f810f7b6000 r--p 0003e000 fd:00 1975094<br>
/usr/lib64/libquadmath.so.0.0.0<br>
7f810f7b6000-7f810f7b7000 rw-p 0003f000 fd:00 1975094<br>
/usr/lib64/libquadmath.so.0.0.0<br>
7f810f7b7000-7f810f7cd000 r-xp 00000000 fd:00 1973274<br>
/usr/lib64/libgcc_s-6.4.1-20170727.so.1<br>
7f810f7cd000-7f810f9cc000 ---p 00016000 fd:00 1973274<br>
/usr/lib64/libgcc_s-6.4.1-20170727.so.1<br>
7f810f9cc000-7f810f9cd000 r--p 00015000 fd:00 1973274<br>
/usr/lib64/libgcc_s-6.4.1-20170727.so.1<br>
7f810f9cd000-7f810f9ce000 rw-p 00016000 fd:00 1973274<br>
/usr/lib64/libgcc_s-6.4.1-20170727.so.1<br>
7f810f9ce000-7f810fafc000 r-xp 00000000 fd:00 1975116<br>
/usr/lib64/libgfortran.so.3.0.0<br>
7f810fafc000-7f810fcfc000 ---p 0012e000 fd:00 1975116<br>
/usr/lib64/libgfortran.so.3.0.0<br>
7f810fcfc000-7f810fcfd000 r--p 0012e000 fd:00 1975116<br>
/usr/lib64/libgfortran.so.3.0.0<br>
7f810fcfd000-7f810fcfe000 rw-p 0012f000 fd:00 1975116<br>
/usr/lib64/libgfortran.so.3.0.0<br>
7f810fcfe000-7f810fcff000 rw-p 00000000 00:00 0<br>
7f810fcff000-7f8110168000 r-xp 00000000 fd:02 28457339<br>
/home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8<br>
7f8110168000-7f8110368000 ---p 00469000 fd:02 28457339<br>
/home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8<br>
7f8110368000-7f811037a000 r--p 00469000 fd:02 28457339<br>
/home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8<br>
7f811037a000-7f8110385000 rw-p 0047b000 fd:02 28457339<br>
/home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8<br>
7f8110385000-7f81103c4000 rw-p 00000000 00:00 0<br>
7f81103c4000-7f8110435000 r-xp 00000000 fd:02 28457343<br>
/home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8<br>
7f8110435000-7f8110635000 ---p 00071000 fd:02 28457343<br>
/home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8<br>
7f8110635000-7f8110639000 r--p 00071000 fd:02 28457343<br>
/home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8<br>
7f8110639000-7f811063b000 rw-p 00075000 fd:02 28457343<br>
/home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8<br>
7f811063b000-7f811063e000 r-xp 00000000 fd:00 1973895<br>
/usr/lib64/<a href="http://libdl-2.24.so" rel="noreferrer" target="_blank">libdl-2.24.so</a><br>
7f811063e000-7f811083d000 ---p 00003000 fd:00 1973895<br>
/usr/lib64/<a href="http://libdl-2.24.so" rel="noreferrer" target="_blank">libdl-2.24.so</a><br>
7f811083d000-7f811083e000 r--p 00002000 fd:00 1973895<br>
/usr/lib64/<a href="http://libdl-2.24.so" rel="noreferrer" target="_blank">libdl-2.24.so</a><br>
7f811083e000-7f811083f000 rw-p 00003000 fd:00 1973895<br>
/usr/lib64/<a href="http://libdl-2.24.so" rel="noreferrer" target="_blank">libdl-2.24.so</a><br>
7f811083f000-7f81109b7000 r-xp 00000000 fd:00 1974512<br>
/usr/lib64/libstdc++.so.6.0.22<br>
7f81109b7000-7f8110bb7000 ---p 00178000 fd:00 1974512<br>
/usr/lib64/libstdc++.so.6.0.22<br>
7f8110bb7000-7f8110bc1000 r--p 00178000 fd:00 1974512<br>
/usr/lib64/libstdc++.so.6.0.22<br>
7f8110bc1000-7f8110bc3000 rw-p 00182000 fd:00 1974512<br>
/usr/lib64/libstdc++.so.6.0.22<br>
7f8110bc3000-7f8110bc7000 rw-p 00000000 00:00 0<br>
7f8110bc7000-7f8110ccf000 r-xp 00000000 fd:00 1973951<br>
/usr/lib64/<a href="http://libm-2.24.so" rel="noreferrer" target="_blank">libm-2.24.so</a><br>
7f8110ccf000-7f8110ece000 ---p 00108000 fd:00 1973951<br>
/usr/lib64/<a href="http://libm-2.24.so" rel="noreferrer" target="_blank">libm-2.24.so</a><br>
7f8110ece000-7f8110ecf000 r--p 00107000 fd:00 1973951<br>
/usr/lib64/<a href="http://libm-2.24.so" rel="noreferrer" target="_blank">libm-2.24.so</a><br>
7f8110ecf000-7f8110ed0000 rw-p 00108000 fd:00 1973951<br>
/usr/lib64/<a href="http://libm-2.24.so" rel="noreferrer" target="_blank">libm-2.24.so</a><br>
7f8110ed0000-7f8110f60000 r-xp 00000000 fd:02 28457370<br>
/home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so<br>
7f8110f60000-7f811115f000 ---p 00090000 fd:02 28457370<br>
/home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so<br>
7f811115f000-7f8111160000 r--p 0008f000 fd:02 28457370<br>
/home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so<br>
7f8111160000-7f8111161000 rw-p 00090000 fd:02 28457370<br>
/home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so<br>
7f8111161000-7f8111169000 rw-p 00000000 00:00 0<br>
7f8111169000-7f81111bf000 r-xp 00000000 fd:02 28457373<br>
/home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so<br>
7f81111bf000-7f81113bf000 ---p 00056000 fd:02 28457373<br>
/home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so<br>
7f81113bf000-7f81113c0000 r--p 00056000 fd:02 28457373<br>
/home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so<br>
7f81113c0000-7f81113c1000 rw-p 00057000 fd:02 28457373<br>
/home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so<br>
7f81113c1000-7f81114f9000 r-xp 00000000 fd:00 1973763<br>
/usr/lib64/libX11.so.6.3.0<br>
7f81114f9000-7f81116f9000 ---p 00138000 fd:00 1973763<br>
/usr/lib64/libX11.so.6.3.0<br>
7f81116f9000-7f81116fc000 r--p 00138000 fd:00 1973763<br>
/usr/lib64/libX11.so.6.3.0<br>
7f81116fc000-7f8111700000 rw-p 0013b000 fd:00 1973763<br>
/usr/lib64/libX11.so.6.3.0<br>
7f8111700000-7f8111718000 r-xp 00000000 fd:00 1974340<br>
/usr/lib64/<a href="http://libpthread-2.24.so" rel="noreferrer" target="_blank">libpthread-2.24.so</a><br>
7f8111718000-7f8111918000 ---p 00018000 fd:00 1974340<br>
/usr/lib64/<a href="http://libpthread-2.24.so" rel="noreferrer" target="_blank">libpthread-2.24.so</a><br>
7f8111918000-7f8111919000 r--p 00018000 fd:00 1974340<br>
/usr/lib64/<a href="http://libpthread-2.24.so" rel="noreferrer" target="_blank">libpthread-2.24.so</a><br>
7f8111919000-7f811191a000 rw-p 00019000 fd:00 1974340<br>
/usr/lib64/<a href="http://libpthread-2.24.so" rel="noreferrer" target="_blank">libpthread-2.24.so</a><br>
7f811191a000-7f811191e000 rw-p 00000000 00:00 0<br>
7f811191e000-7f81124ff000 r-xp 00000000 fd:02 28457384<br>
/home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/<a href="http://libopenblas_haswell-r0.3.8.dev.so" rel="noreferrer" target="_blank">libopenblas_haswell-r0.3.8.dev.so</a><br>
7f81124ff000-7f81126ff000 ---p 00be1000 fd:02 28457384<br>
/home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/<a href="http://libopenblas_haswell-r0.3.8.dev.so" rel="noreferrer" target="_blank">libopenblas_haswell-r0.3.8.dev.so</a><br>
7f81126ff000-7f8112703000 r--p 00be1000 fd:02 28457384<br>
/home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/<a href="http://libopenblas_haswell-r0.3.8.dev.so" rel="noreferrer" target="_blank">libopenblas_haswell-r0.3.8.dev.so</a><br>
7f8112703000-7f811270d000 rw-p 00be5000 fd:02 28457384<br>
/home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/<a href="http://libopenblas_haswell-r0.3.8.dev.so" rel="noreferrer" target="_blank">libopenblas_haswell-r0.3.8.dev.so</a><br>
7f811270d000-7f811270e000 rw-p 00000000 00:00 0<br>
7f811270e000-7f81147c2000 r-xp 00000000 fd:02 28457400<br>
/home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2<br>
7f81147c2000-7f81149c1000 ---p 020b4000 fd:02 28457400<br>
/home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2<br>
7f81149c1000-7f81149c7000 r--p 020b3000 fd:02 28457400<br>
/home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2<br>
7f81149c7000-7f81149df000 rw-p 020b9000 fd:02 28457400<br>
/home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2<br>
7f81149df000-7f8114bdf000 rw-p 00000000 00:00 0<br>
7f8114bdf000-7f8115220000 r-xp 00000000 fd:02 28457416<br>
/home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2<br>
7f8115220000-7f8115420000 ---p 00641000 fd:02 28457416<br>
/home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2<br>
7f8115420000-7f8115421000 r--p 00641000 fd:02 28457416<br>
/home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2<br>
7f8115421000-7f8115425000 rw-p 00642000 fd:02 28457416<br>
/home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2<br>
7f8115425000-7f811544b000 r-xp 00000000 fd:00 1976838<br>
/usr/lib64/<a href="http://ld-2.24.so" rel="noreferrer" target="_blank">ld-2.24.so</a><br>
7f81154d3000-7f8115621000 rw-p 00000000 00:00 0<br>
7f8115647000-7f811564a000 rw-p 00000000 00:00 0<br>
7f811564a000-7f811564b000 r--p 00025000 fd:00 1976838<br>
/usr/lib64/<a href="http://ld-2.24.so" rel="noreferrer" target="_blank">ld-2.24.so</a><br>
7f811564b000-7f811564c000 rw-p 00026000 fd:00 1976838<br>
/usr/lib64/<a href="http://ld-2.24.so" rel="noreferrer" target="_blank">ld-2.24.so</a><br>
7f811564c000-7f811564d000 rw-p 00000000 00:00 0<br>
7ffc79b33000-7ffc79b55000 rw-p 00000000 00:00 0 [stack]<br>
7ffc79b67000-7ffc79b6a000 r--p 00000000 00:00 0 [vvar]<br>
7ffc79b6a000-7ffc79b6c000 r-xp 00000000 00:00 0 [vdso]<br>
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0<br>
[vsyscall]<br>
<br>
<br>
I created and destroyed every vector, matrix and EPS in the loop to<br>
make sure that there won't be any segmentation fault but still I am<br>
getting some. Why am I getting this type of error?<br>
<br>
Thanks so much!<br>
<br>
Eda<br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div>