[petsc-users] segfault after recent scientific linux upgrade

Wed Dec 6 10:58:50 CST 2017

I still think the simplest solution is to upgrade PETSc.  I won't try
anything else.

If you really want to try anything else, you have  the following options

(1)  Not use superlu_dist, and try other preconditioners.

(2) Try  "-mat_superlu_dist_fact" with different values

Fande,

On Wed, Dec 6, 2017 at 12:34 AM, Klaij, Christiaan <C.Klaij at marin.nl> wrote:

> Fande,
>
> Thanks, that's good to know. Upgrading to 3.8.x is definitely my
> long-term plan, but is there anything I can do short-term to fix
> the problem while keeping 3.7.5?
>
> Chris
>
> dr. ir. Christiaan Klaij | Senior Researcher | Research & Development
> MARIN | T +31 317 49 33 44 <+31%20317%20493%20344> | C.Klaij at marin.nl |
> www.marin.nl
>
> [image: LinkedIn] <https://www.linkedin.com/company/marin> [image:
> YouTube] <http://www.youtube.com/marinmultimedia> [image: Twitter]
> <https://twitter.com/MARIN_nieuws> [image: Facebook]
> <https://www.facebook.com/marin.wageningen>
> MARIN news: Seminar ‘Blauwe toekomst: versnellen van innovaties door
> samenwerken
> <http://www.marin.nl/web/News/News-items/Seminar-Blauwe-toekomst-versnellen-van-innovaties-door-samenwerken.htm>
>
> ------------------------------
> *From:* Fande Kong <fdkong.jd at gmail.com>
> *Sent:* Tuesday, December 05, 2017 4:30 PM
> *To:* Klaij, Christiaan
> *Cc:* petsc-users at mcs.anl.gov
> *Subject:* Re: [petsc-users] segfault after recent scientific linux
> upgrade
>
> I would like to suggest you to use PETSc-3.8.x. Then the bug should go
> away. It is a known bug related to the reuse of the factorization pattern.
>
>
> Fande,
>
> On Tue, Dec 5, 2017 at 8:07 AM, Klaij, Christiaan <C.Klaij at marin.nl>
> wrote:
>
>> I'm running production software with petsc-3.7.5 and, among
>> others, superlu_dist 5.1.3 on scientific linux 7.4.
>>
>> After a recent update of SL7.4, notably of the kernel and glibc,
>> we found that superlu is somehow broken. Below's a backtrace of a
>> serial example. Is this a known issue? Could you please advice on
>> how to proceed (preferably while keeping 3.7.5 for now).
>>
>> Thanks,
>> Chris
>>
>> $ gdb ./refresco ./core.9810
>> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7
>> Copyright (C) 2013 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.h
>> tml>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
>> and "show warranty" for details.
>> This GDB was configured as "x86_64-redhat-linux-gnu".
>> For bug reporting instructions, please see:
>> <http://www.gnu.org/software/gdb/bugs/>...
>> Reading symbols from /home/cklaij/ReFRESCO/Dev/trun
>> k/Suites/testSuite/FlatPlate_laminar/calcs/Grid64x64/refresco...done.
>> [New LWP 9810]
>> Missing separate debuginfo for /home/cklaij/ReFRESCO/Dev/trun
>> k/Libs/install/licensing-1.55.0/sll/lib64/libssl.so.10
>> Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/68/6a
>> 25d0a83d002183c835fa5694a8110c78d3bc.debug
>> Missing separate debuginfo for /home/cklaij/ReFRESCO/Dev/trun
>> k/Libs/install/licensing-1.55.0/sll/lib64/libcrypto.so.10
>> Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/68/d2
>> 958189303f421b1082abc33fd87338826c65.debug
>> [Thread debugging using libthread_db enabled]
>> Using host libthread_db library "/lib64/libthread_db.so.1".
>> Core was generated by `./refresco'.
>> Program terminated with signal 11, Segmentation fault.
>> #0  0x00002ba501c132bc in mc64wd_dist (n=0x5213270, ne=0x2, ip=0x1,
>>     irn=0x51af520, a=0x51ef260, iperm=0x1000, num=0x7ffc545b2d94,
>>     jperm=0x51e7260, out=0x51eb260, pr=0x51ef260, q=0x51f3260,
>> l=0x51f7260,
>>     u=0x51fb270, d__=0x5203270)
>>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/install/Linux-x86_64-
>> Intel/SuperLU_DIST_5.1.3/SRC/mc64ad_dist.c:2322
>> 2322    if (iperm[i__] != 0 || iperm[i0] == 0) {
>> Missing separate debuginfos, use: debuginfo-install
>> bzip2-libs-1.0.6-13.el7.x86_64 glibc-2.17-196.el7.x86_64
>> keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-8.el7.x86_64
>> libcom_err-1.42.9-10.el7.x86_64 libgcc-4.8.5-16.el7.x86_64
>> libselinux-2.5-11.el7.x86_64 libstdc++-4.8.5-16.el7.x86_64
>> libxml2-2.9.1-6.el7_2.3.x86_64 numactl-libs-2.0.9-6.el7_2.x86_64
>> pcre-8.32-17.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64
>> (gdb) bt
>> #0  0x00002ba501c132bc in mc64wd_dist (n=0x5213270, ne=0x2, ip=0x1,
>>     irn=0x51af520, a=0x51ef260, iperm=0x1000, num=0x7ffc545b2d94,
>>     jperm=0x51e7260, out=0x51eb260, pr=0x51ef260, q=0x51f3260,
>> l=0x51f7260,
>>     u=0x51fb270, d__=0x5203270)
>>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/install/Linux-x86_64-
>> Intel/SuperLU_DIST_5.1.3/SRC/mc64ad_dist.c:2322
>> #1  0x00002ba501c0ef2b in mc64ad_dist (job=0x5213270, n=0x2, ne=0x1,
>>     ip=0x51af520, irn=0x51ef260, a=0x1000, num=0x7ffc545b2db0,
>>     cperm=0x51fb270, liw=0x5187d10, iw=0x51c3130, ldw=0x51af520,
>> dw=0x517b570,
>>     icntl=0x51e7260, info=0x2ba501c2e556 <dldperm_dist+614>)
>>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/install/Linux-x86_64-
>> Intel/SuperLU_DIST_5.1.3/SRC/mc64ad_dist.c:596
>> #2  0x00002ba501c2e556 in dldperm_dist (job=0, n=0, nnz=0,
>> colptr=0x51af520,
>>     adjncy=0x51ef260, nzval=0x1000, perm=0x4f00, u=0x1000, v=0x517b001)
>>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/install/Linux-x86_64-
>> Intel/SuperLU_DIST_5.1.3/SRC/dldperm_dist.c:141
>> #3  0x00002ba501c26296 in pdgssvx_ABglobal (options=0x5213270, A=0x2,
>>     ScalePermstruct=0x1, B=0x51af520, ldb=85914208, nrhs=4096,
>> grid=0x516da30,
>>     LUstruct=0x517af40, berr=0x1000,
>>     stat=0x2ba500b36a7d <MatLUFactorNumeric_SuperLU_DIST+2349>,
>> info=0x517af58)
>>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/install/Linux-x86_64-
>> Intel/SuperLU_DIST_5.1.3/SRC/pdgssvx_ABglobal.c:716
>> #4  0x00002ba500b36a7d in MatLUFactorNumeric_SuperLU_DIST (F=0x5213270,
>> A=0x2,
>> ---Type <return> to continue, or q <return> to quit---
>>     info=0x1)
>>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/
>> mat/impls/aij/mpi/superlu_dist/superlu_dist.c:419
>> #5  0x00002ba500b45a1a in MatLUFactorNumeric (fact=0x5213270, mat=0x2,
>>     info=0x1)
>>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/
>> mat/interface/matrix.c:2996
>> #6  0x00002ba500e9e6c7 in PCSetUp_LU (pc=0x5213270)
>>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/
>> ksp/pc/impls/factor/lu/lu.c:172
>> #7  0x00002ba500ded084 in PCSetUp (pc=0x5213270)
>>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/
>> ksp/pc/interface/precon.c:968
>> #8  0x00002ba500f2968d in KSPSetUp (ksp=0x5213270)
>>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/
>> ksp/ksp/interface/itfunc.c:390
>> #9  0x00002ba500f257be in KSPSolve (ksp=0x5213270, b=0x2, x=0x4193510)
>>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/
>> ksp/ksp/interface/itfunc.c:599
>> #10 0x00002ba500f3e142 in kspsolve_ (ksp=0x5213270, b=0x2, x=0x1,
>>     __ierr=0x51af520)
>>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/
>> ksp/ksp/interface/ftn-auto/itfuncf.c:261
>> ---Type <return> to continue, or q <return> to quit---
>> #11 0x0000000000bccf71 in petsc_solvers::petsc_solvers_solve (
>>     regname='massTransport', rhs_c=..., phi_c=..., tol=0.01, maxiter=500,
>>     res0=-9.2559631349317831e+61, usediter=0, .tmp.REGNAME.len_V$1790=13)
>>     at petsc_solvers.F90:580
>> #12 0x0000000000c2c9c5 in mass_momentum::mass_momentum_pressureprediction
>> ()
>>     at mass_momentum.F90:989
>> #13 0x0000000000c0ffc1 in mass_momentum::mass_momentum_core ()
>>     at mass_momentum.F90:626
>> #14 0x0000000000c26a2c in mass_momentum::mass_momentum_systempcapply (
>>     aa_system=76390912, xx_system=68983024, rr_system=68984544, ierr=0)
>>     at mass_momentum.F90:919
>> #15 0x00002ba500eaa763 in ourshellapply (pc=0x48da200, x=0x41c98f0,
>>     y=0x41c9ee0)
>>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/
>> ksp/pc/impls/shell/ftn-custom/zshellpcf.c:41
>> #16 0x00002ba500ea79be in PCApply_Shell (pc=0x5213270, x=0x2, y=0x1)
>>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/
>> ksp/pc/impls/shell/shellpc.c:124
>> #17 0x00002ba500df1800 in PCApply (pc=0x5213270, x=0x2, y=0x1)
>>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/
>> ksp/pc/interface/precon.c:482
>> #18 0x00002ba500f2592a in KSPSolve (ksp=0x5213270, b=0x2, x=0x41c9ee0)
>>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/ksp/ksp/interf---Type
>> <return> to continue, or q <return> to quit---
>> ace/itfunc.c:631
>> #19 0x00002ba500f3e142 in kspsolve_ (ksp=0x5213270, b=0x2, x=0x1,
>>     __ierr=0x51af520)
>>     at /home/cklaij/ReFRESCO/Dev/trunk/Libs/build/petsc-3.7.5/src/
>> ksp/ksp/interface/ftn-auto/itfuncf.c:261
>> #20 0x0000000000c1b0ea in mass_momentum::mass_momentum_krylov ()
>>     at mass_momentum.F90:777
>> #21 0x0000000000c0d242 in mass_momentum::mass_momentum_simple ()
>>     at mass_momentum.F90:548
>> #22 0x0000000000c0841f in mass_momentum::mass_momentum_solve ()
>>     at mass_momentum.F90:465
>> #23 0x000000000041b5ec in refresco () at refresco.F90:259
>> #24 0x000000000041999e in main ()
>> #25 0x00002ba508c98c05 in __libc_start_main () from /lib64/libc.so.6
>> #26 0x00000000004198a3 in _start ()
>> (gdb)
>>
>>
>> dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
>> MARIN | T +31 317 49 33 44 <+31%20317%20493%20344> | mailto:
>> C.Klaij at marin.nl | http://www.marin.nl
>>
>> MARIN news: http://www.marin.nl/web/News/News-items/Seminar-Blauwe-toeko
>> mst-versnellen-van-innovaties-door-samenwerken.htm
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20171206/b7e91a3a/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image0cf412.PNG
Type: image/png
Size: 333 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20171206/b7e91a3a/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: imagefdeb13.PNG
Type: image/png
Size: 331 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20171206/b7e91a3a/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image4b2382.PNG
Type: image/png
Size: 253 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20171206/b7e91a3a/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image962398.PNG
Type: image/png
Size: 293 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20171206/b7e91a3a/attachment-0007.png>