[petsc-users] /lib64/libc.so.6(cfree+0x1f8)[0x7f810f237438] Signal received
Matthew Knepley
knepley at gmail.com
Mon Jul 6 15:08:35 CDT 2020
On Mon, Jul 6, 2020 at 4:05 PM Eda Oktay <eda.oktay at metu.edu.tr> wrote:
> I tried for (count=1;count<2;++count), it is working. When I raised it to
> 3, it started giving the same error.
>
> My loop is inside my compiled program, not bash.
>
Okay, give us the full stack trace from the serial run.
Thanks,
Matt
> I started the loop after PetscInitialize and finished it before
> PetscFinalize.
>
> I am not using MPI_Init.
>
> Thanks!
>
> On Mon, Jul 6, 2020, 10:51 PM Barry Smith <bsmith at petsc.dev> wrote:
>
>>
>> Where is your loop? In a bash script or inside your compiled program?
>>
>> If inside the program I assume it is outside of PetscInitialize()?
>>
>> Is it outside of MPI_Init()? It can't be because MPI_Init() can only be
>> called once.
>>
>> If inside the problem you can use -start_in_debugger to get the true
>> location of the crash, if in a bash script you can use
>> -on_error_attach_debugger
>>
>> Barry
>>
>>
>> > On Jul 6, 2020, at 1:14 PM, Eda Oktay <eda.oktay at metu.edu.tr> wrote:
>> >
>> > Hello,
>> >
>> > I am trying to run my codes 100 times and then take the average
>> > elapsed time. Although my program runs correctly for single time, when
>> > I put them into loop, I am getting the following error:
>> >
>> > *** Error in `./avg_coloredge_deneme_new_vecscatter_arastep':
>> > munmap_chunk(): invalid pointer: 0x0000000000b0cbc0 ***
>> > [1]PETSC ERROR:
>> > ------------------------------------------------------------------------
>> > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
>> > probably memory access out of range
>> > [1]PETSC ERROR: Try option -start_in_debugger or
>> -on_error_attach_debugger
>> > [1]PETSC ERROR: or see
>> > https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>> > [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac
>> > OS X to find memory corruption errors
>> > [1]PETSC ERROR: likely location of problem given in stack below
>> > [1]PETSC ERROR: --------------------- Stack Frames
>> > ------------------------------------
>> > [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>> available,
>> > [1]PETSC ERROR: INSTEAD the line number of the start of the
>> function
>> > [1]PETSC ERROR: is given.
>> > ======= Backtrace: =========
>> > [1]PETSC ERROR: /lib64/libc.so.6(+0x791eb)[0x7f810f22a1eb]
>> > --------------------- Error Message
>> > --------------------------------------------------------------
>> > [1]PETSC ERROR: /lib64/libc.so.6(cfree+0x1f8)[0x7f810f237438]
>> > Signal received
>> > [1]PETSC ERROR: See
>> > https://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>> > shooting.
>> > [1]PETSC ERROR: Petsc Release Version 3.13.2, Jun 02, 2020
>> > [1]PETSC ERROR: ./avg_coloredge_deneme_new_vecscatter_arastep on a
>> > arch-linux2-c-debug named localhost.localdomain by edaoktay Mon Jul 6
>> > 21:09:07 2020
>> > [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>> > --download-mpich --download-openblas --download-slepc --download-metis
>> > --download-parmetis --download-chaco --with-X=1
>> > [1]PETSC ERROR: #1 User provided function() line 0 in unknown file
>> > ./avg_coloredge_deneme_new_vecscatter_arastep[0x402dca]
>> > ./avg_coloredge_deneme_new_vecscatter_arastep[0x402f1e]
>> > ./avg_coloredge_deneme_new_vecscatter_arastep[0x402953]
>> > ./avg_coloredge_deneme_new_vecscatter_arastep[0x406c1c]
>> > application called MPI_Abort(MPI_COMM_WORLD, 50152059) - process 1
>> > /lib64/libc.so.6(__libc_start_main+0xf1)[0x7f810f1d1431]
>> > ./avg_coloredge_deneme_new_vecscatter_arastep[0x40260a]
>> > ======= Memory map: ========
>> > 00400000-00409000 r-xp 00000000 fd:02 29230434
>> >
>> /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/share/slepc/examples/src/eda/avg_coloredge_deneme_new_vecscatter_arastep
>> > 00608000-00609000 r--p 00008000 fd:02 29230434
>> >
>> /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/share/slepc/examples/src/eda/avg_coloredge_deneme_new_vecscatter_arastep
>> > 00609000-0060a000 rw-p 00009000 fd:02 29230434
>> >
>> /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/share/slepc/examples/src/eda/avg_coloredge_deneme_new_vecscatter_arastep
>> > 00956000-01100000 rw-p 00000000 00:00 0
>> [heap]
>> > 7f810c621000-7f810c6ca000 rw-p 00000000 00:00 0
>> > 7f810c6ca000-7f810e6ca000 rw-p 00000000 00:00 0
>> > 7f810e6ca000-7f810e6d4000 r-xp 00000000 fd:00 1974339
>> > /usr/lib64/libnss_files-2.24.so
>> > 7f810e6d4000-7f810e8d4000 ---p 0000a000 fd:00 1974339
>> > /usr/lib64/libnss_files-2.24.so
>> > 7f810e8d4000-7f810e8d5000 r--p 0000a000 fd:00 1974339
>> > /usr/lib64/libnss_files-2.24.so
>> > 7f810e8d5000-7f810e8d6000 rw-p 0000b000 fd:00 1974339
>> > /usr/lib64/libnss_files-2.24.so
>> > 7f810e8d6000-7f810eb7d000 rw-p 00000000 00:00 0
>> > 7f810eb7d000-7f810eb7f000 r-xp 00000000 fd:00 1974984
>> > /usr/lib64/libXau.so.6.0.0
>> > 7f810eb7f000-7f810ed7f000 ---p 00002000 fd:00 1974984
>> > /usr/lib64/libXau.so.6.0.0
>> > 7f810ed7f000-7f810ed80000 r--p 00002000 fd:00 1974984
>> > /usr/lib64/libXau.so.6.0.0
>> > 7f810ed80000-7f810ed81000 rw-p 00000000 00:00 0
>> > 7f810ed81000-7f810ed88000 r-xp 00000000 fd:00 1974440
>> > /usr/lib64/librt-2.24.so
>> > 7f810ed88000-7f810ef87000 ---p 00007000 fd:00 1974440
>> > /usr/lib64/librt-2.24.so
>> > 7f810ef87000-7f810ef88000 r--p 00006000 fd:00 1974440
>> > /usr/lib64/librt-2.24.so
>> > 7f810ef88000-7f810ef89000 rw-p 00007000 fd:00 1974440
>> > /usr/lib64/librt-2.24.so
>> > 7f810ef89000-7f810efb0000 r-xp 00000000 fd:00 1975953
>> > /usr/lib64/libxcb.so.1.1.0
>> > 7f810efb0000-7f810f1af000 ---p 00027000 fd:00 1975953
>> > /usr/lib64/libxcb.so.1.1.0
>> > 7f810f1af000-7f810f1b0000 r--p 00026000 fd:00 1975953
>> > /usr/lib64/libxcb.so.1.1.0
>> > 7f810f1b0000-7f810f1b1000 rw-p 00027000 fd:00 1975953
>> > /usr/lib64/libxcb.so.1.1.0
>> > 7f810f1b1000-7f810f36e000 r-xp 00000000 fd:00 1973798
>> > /usr/lib64/libc-2.24.so
>> > 7f810f36e000-7f810f56d000 ---p 001bd000 fd:00 1973798
>> > /usr/lib64/libc-2.24.so
>> > 7f810f56d000-7f810f571000 r--p 001bc000 fd:00 1973798
>> > /usr/lib64/libc-2.24.so
>> > 7f810f571000-7f810f573000 rw-p 001c0000 fd:00 1973798
>> > /usr/lib64/libc-2.24.so
>> > 7f810f573000-7f810f577000 rw-p 00000000 00:00 0
>> > 7f810f577000-7f810f5b6000 r-xp 00000000 fd:00 1975094
>> > /usr/lib64/libquadmath.so.0.0.0
>> > 7f810f5b6000-7f810f7b5000 ---p 0003f000 fd:00 1975094
>> > /usr/lib64/libquadmath.so.0.0.0
>> > 7f810f7b5000-7f810f7b6000 r--p 0003e000 fd:00 1975094
>> > /usr/lib64/libquadmath.so.0.0.0
>> > 7f810f7b6000-7f810f7b7000 rw-p 0003f000 fd:00 1975094
>> > /usr/lib64/libquadmath.so.0.0.0
>> > 7f810f7b7000-7f810f7cd000 r-xp 00000000 fd:00 1973274
>> > /usr/lib64/libgcc_s-6.4.1-20170727.so.1
>> > 7f810f7cd000-7f810f9cc000 ---p 00016000 fd:00 1973274
>> > /usr/lib64/libgcc_s-6.4.1-20170727.so.1
>> > 7f810f9cc000-7f810f9cd000 r--p 00015000 fd:00 1973274
>> > /usr/lib64/libgcc_s-6.4.1-20170727.so.1
>> > 7f810f9cd000-7f810f9ce000 rw-p 00016000 fd:00 1973274
>> > /usr/lib64/libgcc_s-6.4.1-20170727.so.1
>> > 7f810f9ce000-7f810fafc000 r-xp 00000000 fd:00 1975116
>> > /usr/lib64/libgfortran.so.3.0.0
>> > 7f810fafc000-7f810fcfc000 ---p 0012e000 fd:00 1975116
>> > /usr/lib64/libgfortran.so.3.0.0
>> > 7f810fcfc000-7f810fcfd000 r--p 0012e000 fd:00 1975116
>> > /usr/lib64/libgfortran.so.3.0.0
>> > 7f810fcfd000-7f810fcfe000 rw-p 0012f000 fd:00 1975116
>> > /usr/lib64/libgfortran.so.3.0.0
>> > 7f810fcfe000-7f810fcff000 rw-p 00000000 00:00 0
>> > 7f810fcff000-7f8110168000 r-xp 00000000 fd:02 28457339
>> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8
>> > 7f8110168000-7f8110368000 ---p 00469000 fd:02 28457339
>> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8
>> > 7f8110368000-7f811037a000 r--p 00469000 fd:02 28457339
>> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8
>> > 7f811037a000-7f8110385000 rw-p 0047b000 fd:02 28457339
>> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpi.so.12.1.8
>> > 7f8110385000-7f81103c4000 rw-p 00000000 00:00 0
>> > 7f81103c4000-7f8110435000 r-xp 00000000 fd:02 28457343
>> >
>> /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8
>> > 7f8110435000-7f8110635000 ---p 00071000 fd:02 28457343
>> >
>> /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8
>> > 7f8110635000-7f8110639000 r--p 00071000 fd:02 28457343
>> >
>> /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8
>> > 7f8110639000-7f811063b000 rw-p 00075000 fd:02 28457343
>> >
>> /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmpifort.so.12.1.8
>> > 7f811063b000-7f811063e000 r-xp 00000000 fd:00 1973895
>> > /usr/lib64/libdl-2.24.so
>> > 7f811063e000-7f811083d000 ---p 00003000 fd:00 1973895
>> > /usr/lib64/libdl-2.24.so
>> > 7f811083d000-7f811083e000 r--p 00002000 fd:00 1973895
>> > /usr/lib64/libdl-2.24.so
>> > 7f811083e000-7f811083f000 rw-p 00003000 fd:00 1973895
>> > /usr/lib64/libdl-2.24.so
>> > 7f811083f000-7f81109b7000 r-xp 00000000 fd:00 1974512
>> > /usr/lib64/libstdc++.so.6.0.22
>> > 7f81109b7000-7f8110bb7000 ---p 00178000 fd:00 1974512
>> > /usr/lib64/libstdc++.so.6.0.22
>> > 7f8110bb7000-7f8110bc1000 r--p 00178000 fd:00 1974512
>> > /usr/lib64/libstdc++.so.6.0.22
>> > 7f8110bc1000-7f8110bc3000 rw-p 00182000 fd:00 1974512
>> > /usr/lib64/libstdc++.so.6.0.22
>> > 7f8110bc3000-7f8110bc7000 rw-p 00000000 00:00 0
>> > 7f8110bc7000-7f8110ccf000 r-xp 00000000 fd:00 1973951
>> > /usr/lib64/libm-2.24.so
>> > 7f8110ccf000-7f8110ece000 ---p 00108000 fd:00 1973951
>> > /usr/lib64/libm-2.24.so
>> > 7f8110ece000-7f8110ecf000 r--p 00107000 fd:00 1973951
>> > /usr/lib64/libm-2.24.so
>> > 7f8110ecf000-7f8110ed0000 rw-p 00108000 fd:00 1973951
>> > /usr/lib64/libm-2.24.so
>> > 7f8110ed0000-7f8110f60000 r-xp 00000000 fd:02 28457370
>> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so
>> > 7f8110f60000-7f811115f000 ---p 00090000 fd:02 28457370
>> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so
>> > 7f811115f000-7f8111160000 r--p 0008f000 fd:02 28457370
>> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so
>> > 7f8111160000-7f8111161000 rw-p 00090000 fd:02 28457370
>> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libmetis.so
>> > 7f8111161000-7f8111169000 rw-p 00000000 00:00 0
>> > 7f8111169000-7f81111bf000 r-xp 00000000 fd:02 28457373
>> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so
>> > 7f81111bf000-7f81113bf000 ---p 00056000 fd:02 28457373
>> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so
>> > 7f81113bf000-7f81113c0000 r--p 00056000 fd:02 28457373
>> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so
>> > 7f81113c0000-7f81113c1000 rw-p 00057000 fd:02 28457373
>> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libparmetis.so
>> > 7f81113c1000-7f81114f9000 r-xp 00000000 fd:00 1973763
>> > /usr/lib64/libX11.so.6.3.0
>> > 7f81114f9000-7f81116f9000 ---p 00138000 fd:00 1973763
>> > /usr/lib64/libX11.so.6.3.0
>> > 7f81116f9000-7f81116fc000 r--p 00138000 fd:00 1973763
>> > /usr/lib64/libX11.so.6.3.0
>> > 7f81116fc000-7f8111700000 rw-p 0013b000 fd:00 1973763
>> > /usr/lib64/libX11.so.6.3.0
>> > 7f8111700000-7f8111718000 r-xp 00000000 fd:00 1974340
>> > /usr/lib64/libpthread-2.24.so
>> > 7f8111718000-7f8111918000 ---p 00018000 fd:00 1974340
>> > /usr/lib64/libpthread-2.24.so
>> > 7f8111918000-7f8111919000 r--p 00018000 fd:00 1974340
>> > /usr/lib64/libpthread-2.24.so
>> > 7f8111919000-7f811191a000 rw-p 00019000 fd:00 1974340
>> > /usr/lib64/libpthread-2.24.so
>> > 7f811191a000-7f811191e000 rw-p 00000000 00:00 0
>> > 7f811191e000-7f81124ff000 r-xp 00000000 fd:02 28457384
>> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/
>> libopenblas_haswell-r0.3.8.dev.so
>> > 7f81124ff000-7f81126ff000 ---p 00be1000 fd:02 28457384
>> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/
>> libopenblas_haswell-r0.3.8.dev.so
>> > 7f81126ff000-7f8112703000 r--p 00be1000 fd:02 28457384
>> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/
>> libopenblas_haswell-r0.3.8.dev.so
>> > 7f8112703000-7f811270d000 rw-p 00be5000 fd:02 28457384
>> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/
>> libopenblas_haswell-r0.3.8.dev.so
>> > 7f811270d000-7f811270e000 rw-p 00000000 00:00 0
>> > 7f811270e000-7f81147c2000 r-xp 00000000 fd:02 28457400
>> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2
>> > 7f81147c2000-7f81149c1000 ---p 020b4000 fd:02 28457400
>> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2
>> > 7f81149c1000-7f81149c7000 r--p 020b3000 fd:02 28457400
>> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2
>> > 7f81149c7000-7f81149df000 rw-p 020b9000 fd:02 28457400
>> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libpetsc.so.3.13.2
>> > 7f81149df000-7f8114bdf000 rw-p 00000000 00:00 0
>> > 7f8114bdf000-7f8115220000 r-xp 00000000 fd:02 28457416
>> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2
>> > 7f8115220000-7f8115420000 ---p 00641000 fd:02 28457416
>> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2
>> > 7f8115420000-7f8115421000 r--p 00641000 fd:02 28457416
>> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2
>> > 7f8115421000-7f8115425000 rw-p 00642000 fd:02 28457416
>> > /home/edaoktay/petsc-3.13.2/arch-linux2-c-debug/lib/libslepc.so.3.13.2
>> > 7f8115425000-7f811544b000 r-xp 00000000 fd:00 1976838
>> > /usr/lib64/ld-2.24.so
>> > 7f81154d3000-7f8115621000 rw-p 00000000 00:00 0
>> > 7f8115647000-7f811564a000 rw-p 00000000 00:00 0
>> > 7f811564a000-7f811564b000 r--p 00025000 fd:00 1976838
>> > /usr/lib64/ld-2.24.so
>> > 7f811564b000-7f811564c000 rw-p 00026000 fd:00 1976838
>> > /usr/lib64/ld-2.24.so
>> > 7f811564c000-7f811564d000 rw-p 00000000 00:00 0
>> > 7ffc79b33000-7ffc79b55000 rw-p 00000000 00:00 0
>> [stack]
>> > 7ffc79b67000-7ffc79b6a000 r--p 00000000 00:00 0
>> [vvar]
>> > 7ffc79b6a000-7ffc79b6c000 r-xp 00000000 00:00 0
>> [vdso]
>> > ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
>> > [vsyscall]
>> >
>> >
>> > I created and destroyed every vector, matrix and EPS in the loop to
>> > make sure that there won't be any segmentation fault but still I am
>> > getting some. Why am I getting this type of error?
>> >
>> > Thanks so much!
>> >
>> > Eda
>>
>>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20200706/d55d5537/attachment-0001.html>
More information about the petsc-users
mailing list