[petsc-users] segfault after recent scientific linux upgrade

Klaij, Christiaan C.Klaij at marin.nl
Tue Dec 5 09:07:54 CST 2017 

I'm running production software with petsc-3.7.5 and, among
others, superlu_dist 5.1.3 on scientific linux 7.4.

After a recent update of SL7.4, notably of the kernel and glibc,
we found that superlu is somehow broken. Below's a backtrace of a
serial example. Is this a known issue? Could you please advice on
how to proceed (preferably while keeping 3.7.5 for now).

Thanks,
Chris

$ gdb ./refresco ./core.9810
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/cklaij/ReFRESCO/Dev/trunk/Suites/testSuite/FlatPlate_laminar/calcs/Grid64x64/refresco...done.
[New LWP 9810]
Missing separate debuginfo for /home/cklaij/ReFRESCO/Dev/trunk/Libs/install/licensing-1.55.0/sll/lib64/libssl.so.10
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/68/6a25d0a83d002183c835fa5694a8110c78d3bc.debug
Missing separate debuginfo for /home/cklaij/ReFRESCO/Dev/trunk/Libs/install/licensing-1.55.0/sll/lib64/libcrypto.so.10
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/68/d2958189303f421b1082abc33fd87338826c65.debug
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `./refresco'.
Program terminated with signal 11, Segmentation fault.
#0  0x00002ba501c132bc in mc64wd_dist (n=0x5213270, ne=0x2, ip=0x1,
    irn=0x51af520, a=0x51ef260, iperm=0x1000, num=0x7ffc545b2d94,
    jperm=0x51e7260, out=0x51eb260, pr=0x51ef260, q=0x51f3260, l=0x51f7260,
    u=0x51fb270, d__=0x5203270)
    at /home/cklaij/ReFRESCO/Dev/trunk/Libs/install/Linux-x86_64-Intel/SuperLU_DIST_5.1.3/SRC/mc64ad_dist.c:2322
2322    if (iperm[i__] != 0 || iperm[i0] == 0) {
Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 glibc-2.17-196.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-8.el7.x86_64 libcom_err-1.42.9-10.el7.x86_64 libgcc-4.8.5-16.el7.x86_64 libselinux-2.5-11.el7.x86_64 libstdc++-4.8.5-16.el7.x86_64 libxml2-2.9.1-6.el7_2.3.x86_64 numactl-libs-2.0.9-6.el7_2.x86_64 pcre-8.32-17.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) bt
#0  0x00002ba501c132bc in mc64wd_dist (n=0x5213270, ne=0x2, ip=0x1,
    irn=0x51af520, a=0x51ef260, iperm=0x1000, num=0x7ffc545b2d94,
    jperm=0x51e7260, out=0x51eb260, pr=0x51ef260, q=0x51f3260, l=0x51f7260,
    u=0x51fb270, d__=0x5203270)
    at /home/cklaij/ReFRESCO/Dev/trunk/Libs/install/Linux-x86_64-Intel/SuperLU_DIST_5.1.3/SRC/mc64ad_dist.c:2322
#1  0x00002ba501c0ef2b in mc64ad_dist (job=0x5213270, n=0x2, ne=0x1,
    ip=0x51af520, irn=0x51ef260, a=0x1000, num=0x7ffc545b2db0,
    cperm=0x51fb270, liw=0x5187d10, iw=0x51c3130, ldw=0x51af520, dw=0x517b570,
    icntl=0x51e7260, info=0x2ba501c2e556 <dldperm_dist+614>)
    at /home/cklaij/ReFRESCO/Dev/trunk/Libs/install/Linux-x86_64-Intel/SuperLU_DIST_5.1.3/SRC/mc64ad_dist.c:596
#2  0x00002ba501c2e556 in dldperm_dist (job=0, n=0, nnz=0, colptr=0x51af520,
    adjncy=0x51ef260, nzval=0x1000, perm=0x4f00, u=0x1000, v=0x517b001)
    at /home/cklaij/ReFRESCO/Dev/trunk/Libs/install/Linux-x86_64-Intel/SuperLU_DIST_5.1.3/SRC/dldperm_dist.c:141
#3  0x00002ba501c26296 in pdgssvx_ABglobal (options=0x5213270, A=0x2,
    ScalePermstruct=0x1, B=0x51af520, ldb=85914208, nrhs=4096, grid=0x516da30,
    LUstruct=0x517af40, berr=0x1000,
    stat=0x2ba500b36a7d <MatLUFactorNumeric_SuperLU_DIST+2349>, info=0x517af58)
    at /home/cklaij/ReFRESCO/Dev/trunk/Libs/install/Linux-x86_64-Intel/SuperLU_DIST_5.1.3/SRC/pdgssvx_ABglobal.c:716
#4  0x00002ba500b36a7d in MatLUFactorNumeric_SuperLU_DIST (F=0x5213270, A=0x2,
---Type <return> to continue, or q <return> to quit---
    info=0x1)
    at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c:419
#5  0x00002ba500b45a1a in MatLUFactorNumeric (fact=0x5213270, mat=0x2,
    info=0x1)
    at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/mat/interface/matrix.c:2996
#6  0x00002ba500e9e6c7 in PCSetUp_LU (pc=0x5213270)
    at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/ksp/pc/impls/factor/lu/lu.c:172
#7  0x00002ba500ded084 in PCSetUp (pc=0x5213270)
    at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/ksp/pc/interface/precon.c:968
#8  0x00002ba500f2968d in KSPSetUp (ksp=0x5213270)
    at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/ksp/ksp/interface/itfunc.c:390
#9  0x00002ba500f257be in KSPSolve (ksp=0x5213270, b=0x2, x=0x4193510)
    at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/ksp/ksp/interface/itfunc.c:599
#10 0x00002ba500f3e142 in kspsolve_ (ksp=0x5213270, b=0x2, x=0x1,
    __ierr=0x51af520)
    at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/ksp/ksp/interface/ftn-auto/itfuncf.c:261
---Type <return> to continue, or q <return> to quit---
#11 0x0000000000bccf71 in petsc_solvers::petsc_solvers_solve (
    regname='massTransport', rhs_c=..., phi_c=..., tol=0.01, maxiter=500,
    res0=-9.2559631349317831e+61, usediter=0, .tmp.REGNAME.len_V$1790=13)
    at petsc_solvers.F90:580
#12 0x0000000000c2c9c5 in mass_momentum::mass_momentum_pressureprediction ()
    at mass_momentum.F90:989
#13 0x0000000000c0ffc1 in mass_momentum::mass_momentum_core ()
    at mass_momentum.F90:626
#14 0x0000000000c26a2c in mass_momentum::mass_momentum_systempcapply (
    aa_system=76390912, xx_system=68983024, rr_system=68984544, ierr=0)
    at mass_momentum.F90:919
#15 0x00002ba500eaa763 in ourshellapply (pc=0x48da200, x=0x41c98f0,
    y=0x41c9ee0)
    at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/ksp/pc/impls/shell/ftn-custom/zshellpcf.c:41
#16 0x00002ba500ea79be in PCApply_Shell (pc=0x5213270, x=0x2, y=0x1)
    at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/ksp/pc/impls/shell/shellpc.c:124
#17 0x00002ba500df1800 in PCApply (pc=0x5213270, x=0x2, y=0x1)
    at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/ksp/pc/interface/precon.c:482
#18 0x00002ba500f2592a in KSPSolve (ksp=0x5213270, b=0x2, x=0x41c9ee0)
    at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/ksp/ksp/interf---Type <return> to continue, or q <return> to quit---
ace/itfunc.c:631
#19 0x00002ba500f3e142 in kspsolve_ (ksp=0x5213270, b=0x2, x=0x1,
    __ierr=0x51af520)
    at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/ksp/ksp/interface/ftn-auto/itfuncf.c:261
#20 0x0000000000c1b0ea in mass_momentum::mass_momentum_krylov ()
    at mass_momentum.F90:777
#21 0x0000000000c0d242 in mass_momentum::mass_momentum_simple ()
    at mass_momentum.F90:548
#22 0x0000000000c0841f in mass_momentum::mass_momentum_solve ()
    at mass_momentum.F90:465
#23 0x000000000041b5ec in refresco () at refresco.F90:259
#24 0x000000000041999e in main ()
#25 0x00002ba508c98c05 in __libc_start_main () from /lib64/libc.so.6
#26 0x00000000004198a3 in _start ()
(gdb)


dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
MARIN | T +31 317 49 33 44 | mailto:C.Klaij at marin.nl | http://www.marin.nl

MARIN news: http://www.marin.nl/web/News/News-items/Seminar-Blauwe-toekomst-versnellen-van-innovaties-door-samenwerken.htm

More information about the petsc-users mailing list