[petsc-users] segfault after recent scientific linux upgrade

Thu Dec 7 11:07:15 CST 2017

Could you check if your code is valgrind clean?

Satish

On Thu, 7 Dec 2017, Klaij, Christiaan wrote:

> Satish,
> 
> As a first try, I've kept petsc-3.7.5 and only replaced superlu
> by the new xsdk-0.2.0-rc1 version. Unfortunately, this doesn't
> fix the problem, see the backtrace below.
> 
> Fande,
> 
> Perhaps the problem is related to petsc, not superlu?
> 
> What really puzzles me is that everything was working fine with
> petsc-3.7.5 and superlu_dist_5.3.1, it only broke after we
> updated Scientific Linux 7. So this bug (in petsc or in superlu)
> was already there but somehow not triggered before the SL7
> update?
> 
> Chris
> 
> (gdb) bt
> #0  0x00002b38995fa30c in mc64wd_dist (n=0x3da6230, ne=0x2, ip=0x1,
>     irn=0x3d424e0, a=0x3d82220, iperm=0x1000, num=0x7ffc505dd294,
>     jperm=0x3d7a220, out=0x3d7e220, pr=0x3d82220, q=0x3d86220, l=0x3d8a220,
>     u=0x3d8e230, d__=0x3d96230)
>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/install/Linux-x86_64-Intel/superlu_dist-xsdk-0.2.0-rc1/SRC/mc64ad_dist.c:2322
> #1  0x00002b38995f5f7b in mc64ad_dist (job=0x3da6230, n=0x2, ne=0x1,
>     ip=0x3d424e0, irn=0x3d82220, a=0x1000, num=0x7ffc505dd2b0,
>     cperm=0x3d8e230, liw=0x3d1acd0, iw=0x3d560f0, ldw=0x3d424e0, dw=0x3d0e530,
>     icntl=0x3d7a220, info=0x2b3899615546 <dldperm_dist+614>)
>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/install/Linux-x86_64-Intel/superlu_dist-xsdk-0.2.0-rc1/SRC/mc64ad_dist.c:596
> #2  0x00002b3899615546 in dldperm_dist (job=0, n=0, nnz=0, colptr=0x3d424e0,
>     adjncy=0x3d82220, nzval=0x1000, perm=0x4f00, u=0x1000, v=0x3d0e001)
>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/install/Linux-x86_64-Intel/superlu_dist-xsdk-0.2.0-rc1/SRC/dldperm_dist.c:141
> #3  0x00002b389960d286 in pdgssvx_ABglobal (options=0x3da6230, A=0x2,
>     ScalePermstruct=0x1, B=0x3d424e0, ldb=64496160, nrhs=4096, grid=0x3d009f0,
>     LUstruct=0x3d0df00, berr=0x1000,
>     stat=0x2b389851da7d <MatLUFactorNumeric_SuperLU_DIST+2349>, info=0x3d0df18)
>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/install/Linux-x86_64-Intel/superlu_dist-xsdk-0.2.0-rc1/SRC/pdgssvx_ABglobal.c:716
> #4  0x00002b389851da7d in MatLUFactorNumeric_SuperLU_DIST (F=0x3da6230, A=0x2,
> ---Type <return> to continue, or q <return> to quit---
>     info=0x1)
>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c:419
> #5  0x00002b389852ca1a in MatLUFactorNumeric (fact=0x3da6230, mat=0x2,
>     info=0x1)
>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/mat/interface/matrix.c:2996
> #6  0x00002b38988856c7 in PCSetUp_LU (pc=0x3da6230)
>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/ksp/pc/impls/factor/lu/lu.c:172
> #7  0x00002b38987d4084 in PCSetUp (pc=0x3da6230)
>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/ksp/pc/interface/precon.c:968
> #8  0x00002b389891068d in KSPSetUp (ksp=0x3da6230)
>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/ksp/ksp/interface/itfunc.c:390
> #9  0x00002b389890c7be in KSPSolve (ksp=0x3da6230, b=0x2, x=0x2d18d90)
>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/ksp/ksp/interface/itfunc.c:599
> #10 0x00002b3898925142 in kspsolve_ (ksp=0x3da6230, b=0x2, x=0x1,
>     __ierr=0x3d424e0)
>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/ksp/ksp/interface/ftn-auto/itfuncf.c:261
> ---Type <return> to continue, or q <return> to quit---
> #11 0x0000000000bccf71 in petsc_solvers::petsc_solvers_solve (
>     regname='massTransport', rhs_c=..., phi_c=..., tol=0.01, maxiter=500,
>     res0=-9.2559631349317831e+61, usediter=0, .tmp.REGNAME.len_V$1790=13)
>     at petsc_solvers.F90:580
> #12 0x0000000000c2c9c5 in mass_momentum::mass_momentum_pressureprediction ()
>     at mass_momentum.F90:989
> #13 0x0000000000c0ffc1 in mass_momentum::mass_momentum_core ()
>     at mass_momentum.F90:626
> #14 0x0000000000c26a2c in mass_momentum::mass_momentum_systempcapply (
>     aa_system=54952496, xx_system=47570896, rr_system=47572416, ierr=0)
>     at mass_momentum.F90:919
> #15 0x00002b3898891763 in ourshellapply (pc=0x3468230, x=0x2d5dfd0,
>     y=0x2d5e5c0)
>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/ksp/pc/impls/shell/ftn-custom/zshellpcf.c:41
> #16 0x00002b389888e9be in PCApply_Shell (pc=0x3da6230, x=0x2, y=0x1)
>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/ksp/pc/impls/shell/shellpc.c:124
> #17 0x00002b38987d8800 in PCApply (pc=0x3da6230, x=0x2, y=0x1)
>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/ksp/pc/interface/precon.c:482
> #18 0x00002b389890c92a in KSPSolve (ksp=0x3da6230, b=0x2, x=0x2d5e5c0)
>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/ksp/ksp/interf---Type <return> to continue, or q <return> to quit---
> ace/itfunc.c:631
> #19 0x00002b3898925142 in kspsolve_ (ksp=0x3da6230, b=0x2, x=0x1,
>     __ierr=0x3d424e0)
>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/ksp/ksp/interface/ftn-auto/itfuncf.c:261
> #20 0x0000000000c1b0ea in mass_momentum::mass_momentum_krylov ()
>     at mass_momentum.F90:777
> #21 0x0000000000c0d242 in mass_momentum::mass_momentum_simple ()
>     at mass_momentum.F90:548
> #22 0x0000000000c0841f in mass_momentum::mass_momentum_solve ()
>     at mass_momentum.F90:465
> #23 0x000000000041b5ec in refresco () at refresco.F90:259
> #24 0x000000000041999e in main ()
> #25 0x00002b38a067fc05 in __libc_start_main () from /lib64/libc.so.6
> #26 0x00000000004198a3 in _start ()
> (gdb)
> 
> 
> 
> dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
> MARIN | T +31 317 49 33 44 | mailto:C.Klaij at marin.nl | http://www.marin.nl
> 
> MARIN news: http://www.marin.nl/web/News/News-items/Simulator-facility-in-Houston-as-bridge-between-engineering-and-operations.htm
> 
> ________________________________________
> From: Klaij, Christiaan
> Sent: Thursday, December 07, 2017 12:02 PM
> To: petsc-users
> Cc: Fande Kong
> Subject: Re: [petsc-users] segfault after recent scientific linux upgrade
> 
> Thanks Satish, I will give it shot and let you know.
> 
> Chris
> ________________________________________
> From: Satish Balay <balay at mcs.anl.gov>
> Sent: Wednesday, December 06, 2017 6:05 PM
> To: Klaij, Christiaan
> Cc: Fande Kong; petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] segfault after recent scientific linux upgrade
> 
> petsc 3.7 - and 3.8 both default to superlu_dist snapshot:
> 
>     self.gitcommit         = 'xsdk-0.2.0-rc1'
> 
> If using petsc-3.7 - you can use latest maint-3.7 [i.e 3.7.7+]
> [3.7.7 is a latest bugfix update to 3.7 - so there should be no reason to stick to 3.7.5]
> 
> But if you really want to stick to 3.7.5 you can use:
> 
> --download-superlu_dist=1 --download-superlu_dist-commit=xsdk-0.2.0-rc1
> 
> Satish
> 
> On Wed, 6 Dec 2017, Klaij, Christiaan wrote:
> 
> > Fande,
> >
> > Thanks, that's good to know. Upgrading to 3.8.x is definitely my
> > long-term plan, but is there anything I can do short-term to fix
> > the problem while keeping 3.7.5?
> >
> > Chris
> >
> > dr. ir. Christiaan Klaij | Senior Researcher | Research & Development
> > MARIN | T +31 317 49 33 44 | C.Klaij at marin.nl<mailto:C.Klaij at marin.nl> | www.marin.nl<http://www.marin.nl>
> >
> > [LinkedIn]<https://www.linkedin.com/company/marin> [YouTube] <http://www.youtube.com/marinmultimedia>  [Twitter] <https://twitter.com/MARIN_nieuws>  [Facebook] <https://www.facebook.com/marin.wageningen>
> > MARIN news: Seminar ‘Blauwe toekomst: versnellen van innovaties door samenwerken<http://www.marin.nl/web/News/News-items/Seminar-Blauwe-toekomst-versnellen-van-innovaties-door-samenwerken.htm>
> >
> > ________________________________
> > From: Fande Kong <fdkong.jd at gmail.com>
> > Sent: Tuesday, December 05, 2017 4:30 PM
> > To: Klaij, Christiaan
> > Cc: petsc-users at mcs.anl.gov
> > Subject: Re: [petsc-users] segfault after recent scientific linux upgrade
> >
> > I would like to suggest you to use PETSc-3.8.x. Then the bug should go away. It is a known bug related to the reuse of the factorization pattern.
> >
> >
> > Fande,
> >
> > On Tue, Dec 5, 2017 at 8:07 AM, Klaij, Christiaan <C.Klaij at marin.nl<mailto:C.Klaij at marin.nl>> wrote:
> > I'm running production software with petsc-3.7.5 and, among
> > others, superlu_dist 5.1.3 on scientific linux 7.4.
> >
> > After a recent update of SL7.4, notably of the kernel and glibc,
> > we found that superlu is somehow broken. Below's a backtrace of a
> > serial example. Is this a known issue? Could you please advice on
> > how to proceed (preferably while keeping 3.7.5 for now).
> >
> > Thanks,
> > Chris
> >
> > $ gdb ./refresco ./core.9810
> > GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7
> > Copyright (C) 2013 Free Software Foundation, Inc.
> > License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> > This is free software: you are free to change and redistribute it.
> > There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> > and "show warranty" for details.
> > This GDB was configured as "x86_64-redhat-linux-gnu".
> > For bug reporting instructions, please see:
> > <http://www.gnu.org/software/gdb/bugs/>...
> > Reading symbols from /home/cklaij/ReFRESCO/Dev/trunk/Suites/testSuite/FlatPlate_laminar/calcs/Grid64x64/refresco...done.
> > [New LWP 9810]
> > Missing separate debuginfo for /home/cklaij/ReFRESCO/Dev/trunk/Libs/install/licensing-1.55.0/sll/lib64/libssl.so.10
> > Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/68/6a25d0a83d002183c835fa5694a8110c78d3bc.debug
> > Missing separate debuginfo for /home/cklaij/ReFRESCO/Dev/trunk/Libs/install/licensing-1.55.0/sll/lib64/libcrypto.so.10
> > Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/68/d2958189303f421b1082abc33fd87338826c65.debug
> > [Thread debugging using libthread_db enabled]
> > Using host libthread_db library "/lib64/libthread_db.so.1".
> > Core was generated by `./refresco'.
> > Program terminated with signal 11, Segmentation fault.
> > #0  0x00002ba501c132bc in mc64wd_dist (n=0x5213270, ne=0x2, ip=0x1,
> >     irn=0x51af520, a=0x51ef260, iperm=0x1000, num=0x7ffc545b2d94,
> >     jperm=0x51e7260, out=0x51eb260, pr=0x51ef260, q=0x51f3260, l=0x51f7260,
> >     u=0x51fb270, d__=0x5203270)
> >     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/install/Linux-x86_64-Intel/SuperLU_DIST_5.1.3/SRC/mc64ad_dist.c:2322
> > 2322    if (iperm[i__] != 0 || iperm[i0] == 0) {
> > Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 glibc-2.17-196.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-8.el7.x86_64 libcom_err-1.42.9-10.el7.x86_64 libgcc-4.8.5-16.el7.x86_64 libselinux-2.5-11.el7.x86_64 libstdc++-4.8.5-16.el7.x86_64 libxml2-2.9.1-6.el7_2.3.x86_64 numactl-libs-2.0.9-6.el7_2.x86_64 pcre-8.32-17.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64
> > (gdb) bt
> > #0  0x00002ba501c132bc in mc64wd_dist (n=0x5213270, ne=0x2, ip=0x1,
> >     irn=0x51af520, a=0x51ef260, iperm=0x1000, num=0x7ffc545b2d94,
> >     jperm=0x51e7260, out=0x51eb260, pr=0x51ef260, q=0x51f3260, l=0x51f7260,
> >     u=0x51fb270, d__=0x5203270)
> >     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/install/Linux-x86_64-Intel/SuperLU_DIST_5.1.3/SRC/mc64ad_dist.c:2322
> > #1  0x00002ba501c0ef2b in mc64ad_dist (job=0x5213270, n=0x2, ne=0x1,
> >     ip=0x51af520, irn=0x51ef260, a=0x1000, num=0x7ffc545b2db0,
> >     cperm=0x51fb270, liw=0x5187d10, iw=0x51c3130, ldw=0x51af520, dw=0x517b570,
> >     icntl=0x51e7260, info=0x2ba501c2e556 <dldperm_dist+614>)
> >     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/install/Linux-x86_64-Intel/SuperLU_DIST_5.1.3/SRC/mc64ad_dist.c:596
> > #2  0x00002ba501c2e556 in dldperm_dist (job=0, n=0, nnz=0, colptr=0x51af520,
> >     adjncy=0x51ef260, nzval=0x1000, perm=0x4f00, u=0x1000, v=0x517b001)
> >     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/install/Linux-x86_64-Intel/SuperLU_DIST_5.1.3/SRC/dldperm_dist.c:141
> > #3  0x00002ba501c26296 in pdgssvx_ABglobal (options=0x5213270, A=0x2,
> >     ScalePermstruct=0x1, B=0x51af520, ldb=85914208, nrhs=4096, grid=0x516da30,
> >     LUstruct=0x517af40, berr=0x1000,
> >     stat=0x2ba500b36a7d <MatLUFactorNumeric_SuperLU_DIST+2349>, info=0x517af58)
> >     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/install/Linux-x86_64-Intel/SuperLU_DIST_5.1.3/SRC/pdgssvx_ABglobal.c:716
> > #4  0x00002ba500b36a7d in MatLUFactorNumeric_SuperLU_DIST (F=0x5213270, A=0x2,
> > ---Type <return> to continue, or q <return> to quit---
> >     info=0x1)
> >     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/mat/impls/aij/mpi/superlu_dist/superlu_dist.c:419
> > #5  0x00002ba500b45a1a in MatLUFactorNumeric (fact=0x5213270, mat=0x2,
> >     info=0x1)
> >     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/mat/interface/matrix.c:2996
> > #6  0x00002ba500e9e6c7 in PCSetUp_LU (pc=0x5213270)
> >     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/ksp/pc/impls/factor/lu/lu.c:172
> > #7  0x00002ba500ded084 in PCSetUp (pc=0x5213270)
> >     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/ksp/pc/interface/precon.c:968
> > #8  0x00002ba500f2968d in KSPSetUp (ksp=0x5213270)
> >     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/ksp/ksp/interface/itfunc.c:390
> > #9  0x00002ba500f257be in KSPSolve (ksp=0x5213270, b=0x2, x=0x4193510)
> >     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/ksp/ksp/interface/itfunc.c:599
> > #10 0x00002ba500f3e142 in kspsolve_ (ksp=0x5213270, b=0x2, x=0x1,
> >     __ierr=0x51af520)
> >     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/ksp/ksp/interface/ftn-auto/itfuncf.c:261
> > ---Type <return> to continue, or q <return> to quit---
> > #11 0x0000000000bccf71 in petsc_solvers::petsc_solvers_solve (
> >     regname='massTransport', rhs_c=..., phi_c=..., tol=0.01, maxiter=500,
> >     res0=-9.2559631349317831e+61, usediter=0, .tmp.REGNAME.len_V$1790=13)
> >     at petsc_solvers.F90:580
> > #12 0x0000000000c2c9c5 in mass_momentum::mass_momentum_pressureprediction ()
> >     at mass_momentum.F90:989
> > #13 0x0000000000c0ffc1 in mass_momentum::mass_momentum_core ()
> >     at mass_momentum.F90:626
> > #14 0x0000000000c26a2c in mass_momentum::mass_momentum_systempcapply (
> >     aa_system=76390912, xx_system=68983024, rr_system=68984544, ierr=0)
> >     at mass_momentum.F90:919
> > #15 0x00002ba500eaa763 in ourshellapply (pc=0x48da200, x=0x41c98f0,
> >     y=0x41c9ee0)
> >     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/ksp/pc/impls/shell/ftn-custom/zshellpcf.c:41
> > #16 0x00002ba500ea79be in PCApply_Shell (pc=0x5213270, x=0x2, y=0x1)
> >     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/ksp/pc/impls/shell/shellpc.c:124
> > #17 0x00002ba500df1800 in PCApply (pc=0x5213270, x=0x2, y=0x1)
> >     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/ksp/pc/interface/precon.c:482
> > #18 0x00002ba500f2592a in KSPSolve (ksp=0x5213270, b=0x2, x=0x41c9ee0)
> >     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/ksp/ksp/interf---Type <return> to continue, or q <return> to quit---
> > ace/itfunc.c:631
> > #19 0x00002ba500f3e142 in kspsolve_ (ksp=0x5213270, b=0x2, x=0x1,
> >     __ierr=0x51af520)
> >     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/ksp/ksp/interface/ftn-auto/itfuncf.c:261
> > #20 0x0000000000c1b0ea in mass_momentum::mass_momentum_krylov ()
> >     at mass_momentum.F90:777
> > #21 0x0000000000c0d242 in mass_momentum::mass_momentum_simple ()
> >     at mass_momentum.F90:548
> > #22 0x0000000000c0841f in mass_momentum::mass_momentum_solve ()
> >     at mass_momentum.F90:465
> > #23 0x000000000041b5ec in refresco () at refresco.F90:259
> > #24 0x000000000041999e in main ()
> > #25 0x00002ba508c98c05 in __libc_start_main () from /lib64/libc.so.6
> > #26 0x00000000004198a3 in _start ()
> > (gdb)
> >
> >
> > dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
> > MARIN | T +31 317 49 33 44 | mailto:C.Klaij at marin.nl<mailto:C.Klaij at marin.nl> | http://www.marin.nl
> >
> > MARIN news: http://www.marin.nl/web/News/News-items/Seminar-Blauwe-toekomst-versnellen-van-innovaties-door-samenwerken.htm
> >
> >
> >
> >
> >
> 