<div dir="ltr">I still think the simplest solution is to upgrade PETSc. I won't try anything else.<div><br></div><div>If you really want to try anything else, you have the following options <br><div><br></div><div>(1) Not use superlu_dist, and try other preconditioners. </div></div><div><br></div><div>(2) Try "-mat_superlu_dist_fact" with different values</div><div><br></div><div>Fande,</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Dec 6, 2017 at 12:34 AM, Klaij, Christiaan <span dir="ltr"><<a href="mailto:C.Klaij@marin.nl" target="_blank">C.Klaij@marin.nl</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" style="font-size:12pt;color:#000000;background-color:#ffffff;font-family:Calibri,Arial,Helvetica,sans-serif">
<p></p>
<p>Fande,<br>
<br>
Thanks, that's good to know. Upgrading to 3.8.x is definitely my<br>
long-term plan, but is there anything I can do short-term to fix<br>
the problem while keeping 3.7.5? <br>
<br>
Chris<br>
</p><span class="">
<div style="color:rgb(33,33,33)"></div>
<br>
<font style="font-family:Verdana;font-size:8pt;color:#7d7d7d;font-weight:bold">dr. ir.</font><font style="font-family:Verdana;font-size:8pt;color:#7d7d7d;font-weight:bold">
</font><font style="font-family:Verdana;font-size:8pt;color:#7d7d7d;font-weight:bold">Christiaan</font><font style="font-family:Verdana;font-size:8pt;color:#7d7d7d;font-weight:bold">
</font><font style="font-family:Verdana;font-size:8pt;color:#7d7d7d;font-weight:bold">Klaij</font><font size="3" face="Segoe UI"></font><font style="font-family:Verdana;font-size:8pt;color:#7d7d7d;font-weight:bold"> |
</font><font style="font-family:Verdana;font-size:8pt;color:#7d7d7d;font-weight:bold">Senior Researcher</font><font style="font-family:Verdana;font-size:8pt;color:#7d7d7d;font-weight:bold"> |
</font><font style="font-family:Verdana;font-size:8pt;color:#7d7d7d;font-weight:bold">Research & Development</font><br>
</span><font style="font-family:Verdana;font-size:9pt;color:#7d7d7d;font-weight:bold">MARIN</font><font style="font-family:Verdana;font-size:8pt;color:#7d7d7d"> | T
</font><font style="font-family:Verdana;font-size:8pt;color:#7d7d7d"><a href="tel:+31%20317%20493%20344" value="+31317493344" target="_blank">+31 317 49 33 44</a></font><font color="#7d7d7d" size="2" face="Segoe UI"></font><font style="font-family:Verdana;font-size:8pt"> |
</font><span style="font-family:Verdana;font-size:8pt"><a href="mailto:C.Klaij@marin.nl" title="" style="font-family:Verdana;font-size:8pt" target="_blank"><span style="font-family:Verdana;font-size:8pt">C.Klaij@marin.nl</span></a></span><font color="#7d7d7d" size="2" face="Segoe UI"></font><font style="font-family:Verdana;font-size:8pt"> |
</font><span style="font-family:Verdana;font-size:8pt"><a href="http://www.marin.nl" title="" style="font-family:Verdana;font-size:8pt" target="_blank"><span style="font-family:Verdana;font-size:8pt">www.marin.nl</span></a></span><br>
<br>
<a href="https://www.linkedin.com/company/marin" target="_blank"><img width="16" height="16" style="border:0px Solid" src="cid:image962398.PNG@d332d5ae.45b8508f" alt="LinkedIn"></a> <a href="http://www.youtube.com/marinmultimedia" target="_blank"><img width="16" height="16" style="border:0px Solid" src="cid:imagefdeb13.PNG@1144e9eb.40a46d05" alt="YouTube"></a> <a href="https://twitter.com/MARIN_nieuws" target="_blank"><img width="16" height="16" style="border:0px Solid" src="cid:image0cf412.PNG@16571e23.4bb43d23" alt="Twitter"></a> <a href="https://www.facebook.com/marin.wageningen" target="_blank"><img width="16" height="16" style="border:0px Solid" src="cid:image4b2382.PNG@b97ef7f9.4cb02df0" alt="Facebook"></a><br>
<font style="FONT-FAMILY:Verdana;FONT-SIZE:8pt" color="#7d7d7d" size="2" face="Verdana">MARIN news:
<font style="font-family:Verdana;font-size:8pt"><span style="font-family:Verdana;font-size:8pt"><a href="http://www.marin.nl/web/News/News-items/Seminar-Blauwe-toekomst-versnellen-van-innovaties-door-samenwerken.htm" title="" style="font-family:Verdana;font-size:8pt" target="_blank"><span style="font-family:Verdana;font-size:8pt">Seminar
‘Blauwe toekomst: versnellen van innovaties door samenwerken</span></a></span></font></font>
<p></p>
<p></p>
<div style="color:rgb(33,33,33)">
<hr style="display:inline-block;width:98%">
<div id="m_-3644348030254489672divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Fande Kong <<a href="mailto:fdkong.jd@gmail.com" target="_blank">fdkong.jd@gmail.com</a>><br>
<b>Sent:</b> Tuesday, December 05, 2017 4:30 PM<br>
<b>To:</b> Klaij, Christiaan<br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a><br>
<b>Subject:</b> Re: [petsc-users] segfault after recent scientific linux upgrade</font>
<div> </div>
</div><div><div class="h5">
<div>
<div dir="ltr">I would like to suggest you to use PETSc-3.8.x. Then the bug should go away. It is a known bug related to the reuse of the factorization pattern.
<div><br>
</div>
<div><br>
</div>
<div>Fande,</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Dec 5, 2017 at 8:07 AM, Klaij, Christiaan <span dir="ltr">
<<a href="mailto:C.Klaij@marin.nl" target="_blank">C.Klaij@marin.nl</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I'm running production software with petsc-3.7.5 and, among<br>
others, superlu_dist 5.1.3 on scientific linux 7.4.<br>
<br>
After a recent update of SL7.4, notably of the kernel and glibc,<br>
we found that superlu is somehow broken. Below's a backtrace of a<br>
serial example. Is this a known issue? Could you please advice on<br>
how to proceed (preferably while keeping 3.7.5 for now).<br>
<br>
Thanks,<br>
Chris<br>
<br>
$ gdb ./refresco ./core.9810<br>
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7<br>
Copyright (C) 2013 Free Software Foundation, Inc.<br>
License GPLv3+: GNU GPL version 3 or later <<a href="http://gnu.org/licenses/gpl.html" rel="noreferrer" target="_blank">http://gnu.org/licenses/gpl.h<wbr>tml</a>><br>
This is free software: you are free to change and redistribute it.<br>
There is NO WARRANTY, to the extent permitted by law. Type "show copying"<br>
and "show warranty" for details.<br>
This GDB was configured as "x86_64-redhat-linux-gnu".<br>
For bug reporting instructions, please see:<br>
<<a href="http://www.gnu.org/software/gdb/bugs/" rel="noreferrer" target="_blank">http://www.gnu.org/software/g<wbr>db/bugs/</a>>...<br>
Reading symbols from /home/cklaij/ReFRESCO/Dev/trun<wbr>k/Suites/testSuite/FlatPlate_<wbr>laminar/calcs/Grid64x64/<wbr>refresco...done.<br>
[New LWP 9810]<br>
Missing separate debuginfo for /home/cklaij/ReFRESCO/Dev/trun<wbr>k/Libs/install/licensing-1.55.<wbr>0/sll/lib64/libssl.so.10<br>
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/68/6a<wbr>25d0a83d002183c835fa5694a8110c<wbr>78d3bc.debug<br>
Missing separate debuginfo for /home/cklaij/ReFRESCO/Dev/trun<wbr>k/Libs/install/licensing-1.55.<wbr>0/sll/lib64/libcrypto.so.10<br>
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/68/d2<wbr>958189303f421b1082abc33fd87338<wbr>826c65.debug<br>
[Thread debugging using libthread_db enabled]<br>
Using host libthread_db library "/lib64/libthread_db.so.1".<br>
Core was generated by `./refresco'.<br>
Program terminated with signal 11, Segmentation fault.<br>
#0 0x00002ba501c132bc in mc64wd_dist (n=0x5213270, ne=0x2, ip=0x1,<br>
irn=0x51af520, a=0x51ef260, iperm=0x1000, num=0x7ffc545b2d94,<br>
jperm=0x51e7260, out=0x51eb260, pr=0x51ef260, q=0x51f3260, l=0x51f7260,<br>
u=0x51fb270, d__=0x5203270)<br>
at /home/cklaij/ReFRESCO/Dev/trun<wbr>k/Libs/install/Linux-x86_64-<wbr>Intel/SuperLU_DIST_5.1.3/SRC/<wbr>mc64ad_dist.c:2322<br>
2322 if (iperm[i__] != 0 || iperm[i0] == 0) {<br>
Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 glibc-2.17-196.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_<wbr>64 krb5-libs-1.15.1-8.el7.x86_64 libcom_err-1.42.9-10.el7.x86_6<wbr>4 libgcc-4.8.5-16.el7.x86_64 libselinux-2.5-11.el7.x86_64
libstdc++-4.8.5-16.el7.x86_64 libxml2-2.9.1-6.el7_2.3.x86_64 numactl-libs-2.0.9-6.el7_2.x86<wbr>_64 pcre-8.32-17.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64<br>
(gdb) bt<br>
#0 0x00002ba501c132bc in mc64wd_dist (n=0x5213270, ne=0x2, ip=0x1,<br>
irn=0x51af520, a=0x51ef260, iperm=0x1000, num=0x7ffc545b2d94,<br>
jperm=0x51e7260, out=0x51eb260, pr=0x51ef260, q=0x51f3260, l=0x51f7260,<br>
u=0x51fb270, d__=0x5203270)<br>
at /home/cklaij/ReFRESCO/Dev/trun<wbr>k/Libs/install/Linux-x86_64-<wbr>Intel/SuperLU_DIST_5.1.3/SRC/<wbr>mc64ad_dist.c:2322<br>
#1 0x00002ba501c0ef2b in mc64ad_dist (job=0x5213270, n=0x2, ne=0x1,<br>
ip=0x51af520, irn=0x51ef260, a=0x1000, num=0x7ffc545b2db0,<br>
cperm=0x51fb270, liw=0x5187d10, iw=0x51c3130, ldw=0x51af520, dw=0x517b570,<br>
icntl=0x51e7260, info=0x2ba501c2e556 <dldperm_dist+614>)<br>
at /home/cklaij/ReFRESCO/Dev/trun<wbr>k/Libs/install/Linux-x86_64-<wbr>Intel/SuperLU_DIST_5.1.3/SRC/<wbr>mc64ad_dist.c:596<br>
#2 0x00002ba501c2e556 in dldperm_dist (job=0, n=0, nnz=0, colptr=0x51af520,<br>
adjncy=0x51ef260, nzval=0x1000, perm=0x4f00, u=0x1000, v=0x517b001)<br>
at /home/cklaij/ReFRESCO/Dev/trun<wbr>k/Libs/install/Linux-x86_64-<wbr>Intel/SuperLU_DIST_5.1.3/SRC/<wbr>dldperm_dist.c:141<br>
#3 0x00002ba501c26296 in pdgssvx_ABglobal (options=0x5213270, A=0x2,<br>
ScalePermstruct=0x1, B=0x51af520, ldb=85914208, nrhs=4096, grid=0x516da30,<br>
LUstruct=0x517af40, berr=0x1000,<br>
stat=0x2ba500b36a7d <MatLUFactorNumeric_SuperLU_DI<wbr>ST+2349>, info=0x517af58)<br>
at /home/cklaij/ReFRESCO/Dev/trun<wbr>k/Libs/install/Linux-x86_64-<wbr>Intel/SuperLU_DIST_5.1.3/SRC/<wbr>pdgssvx_ABglobal.c:716<br>
#4 0x00002ba500b36a7d in MatLUFactorNumeric_SuperLU_DIS<wbr>T (F=0x5213270, A=0x2,<br>
---Type <return> to continue, or q <return> to quit---<br>
info=0x1)<br>
at /home/cklaij/ReFRESCO/Dev/trun<wbr>k/Libs/build/petsc-3.7.5/src/<wbr>mat/impls/aij/mpi/superlu_dist<wbr>/superlu_dist.c:419<br>
#5 0x00002ba500b45a1a in MatLUFactorNumeric (fact=0x5213270, mat=0x2,<br>
info=0x1)<br>
at /home/cklaij/ReFRESCO/Dev/trun<wbr>k/Libs/build/petsc-3.7.5/src/<wbr>mat/interface/matrix.c:2996<br>
#6 0x00002ba500e9e6c7 in PCSetUp_LU (pc=0x5213270)<br>
at /home/cklaij/ReFRESCO/Dev/trun<wbr>k/Libs/build/petsc-3.7.5/src/<wbr>ksp/pc/impls/factor/lu/lu.c:<wbr>172<br>
#7 0x00002ba500ded084 in PCSetUp (pc=0x5213270)<br>
at /home/cklaij/ReFRESCO/Dev/trun<wbr>k/Libs/build/petsc-3.7.5/src/<wbr>ksp/pc/interface/precon.c:968<br>
#8 0x00002ba500f2968d in KSPSetUp (ksp=0x5213270)<br>
at /home/cklaij/ReFRESCO/Dev/trun<wbr>k/Libs/build/petsc-3.7.5/src/<wbr>ksp/ksp/interface/itfunc.c:390<br>
#9 0x00002ba500f257be in KSPSolve (ksp=0x5213270, b=0x2, x=0x4193510)<br>
at /home/cklaij/ReFRESCO/Dev/trun<wbr>k/Libs/build/petsc-3.7.5/src/<wbr>ksp/ksp/interface/itfunc.c:599<br>
#10 0x00002ba500f3e142 in kspsolve_ (ksp=0x5213270, b=0x2, x=0x1,<br>
__ierr=0x51af520)<br>
at /home/cklaij/ReFRESCO/Dev/trun<wbr>k/Libs/build/petsc-3.7.5/src/<wbr>ksp/ksp/interface/ftn-auto/<wbr>itfuncf.c:261<br>
---Type <return> to continue, or q <return> to quit---<br>
#11 0x0000000000bccf71 in petsc_solvers::petsc_solvers_s<wbr>olve (<br>
regname='massTransport', rhs_c=..., phi_c=..., tol=0.01, maxiter=500,<br>
res0=-9.2559631349317831e+61, usediter=0, .tmp.REGNAME.len_V$1790=13)<br>
at petsc_solvers.F90:580<br>
#12 0x0000000000c2c9c5 in mass_momentum::mass_momentum_p<wbr>ressureprediction ()<br>
at mass_momentum.F90:989<br>
#13 0x0000000000c0ffc1 in mass_momentum::mass_momentum_c<wbr>ore ()<br>
at mass_momentum.F90:626<br>
#14 0x0000000000c26a2c in mass_momentum::mass_momentum_s<wbr>ystempcapply (<br>
aa_system=76390912, xx_system=68983024, rr_system=68984544, ierr=0)<br>
at mass_momentum.F90:919<br>
#15 0x00002ba500eaa763 in ourshellapply (pc=0x48da200, x=0x41c98f0,<br>
y=0x41c9ee0)<br>
at /home/cklaij/ReFRESCO/Dev/trun<wbr>k/Libs/build/petsc-3.7.5/src/<wbr>ksp/pc/impls/shell/ftn-custom/<wbr>zshellpcf.c:41<br>
#16 0x00002ba500ea79be in PCApply_Shell (pc=0x5213270, x=0x2, y=0x1)<br>
at /home/cklaij/ReFRESCO/Dev/trun<wbr>k/Libs/build/petsc-3.7.5/src/<wbr>ksp/pc/impls/shell/shellpc.c:<wbr>124<br>
#17 0x00002ba500df1800 in PCApply (pc=0x5213270, x=0x2, y=0x1)<br>
at /home/cklaij/ReFRESCO/Dev/trun<wbr>k/Libs/build/petsc-3.7.5/src/<wbr>ksp/pc/interface/precon.c:482<br>
#18 0x00002ba500f2592a in KSPSolve (ksp=0x5213270, b=0x2, x=0x41c9ee0)<br>
at /home/cklaij/ReFRESCO/Dev/trun<wbr>k/Libs/build/petsc-3.7.5/src/<wbr>ksp/ksp/interf---Type <return> to continue, or q <return> to quit---<br>
ace/itfunc.c:631<br>
#19 0x00002ba500f3e142 in kspsolve_ (ksp=0x5213270, b=0x2, x=0x1,<br>
__ierr=0x51af520)<br>
at /home/cklaij/ReFRESCO/Dev/trun<wbr>k/Libs/build/petsc-3.7.5/src/<wbr>ksp/ksp/interface/ftn-auto/<wbr>itfuncf.c:261<br>
#20 0x0000000000c1b0ea in mass_momentum::mass_momentum_k<wbr>rylov ()<br>
at mass_momentum.F90:777<br>
#21 0x0000000000c0d242 in mass_momentum::mass_momentum_s<wbr>imple ()<br>
at mass_momentum.F90:548<br>
#22 0x0000000000c0841f in mass_momentum::mass_momentum_s<wbr>olve ()<br>
at mass_momentum.F90:465<br>
#23 0x000000000041b5ec in refresco () at refresco.F90:259<br>
#24 0x000000000041999e in main ()<br>
#25 0x00002ba508c98c05 in __libc_start_main () from /lib64/libc.so.6<br>
#26 0x00000000004198a3 in _start ()<br>
(gdb)<br>
<br>
<br>
dr. ir. Christiaan Klaij | Senior Researcher | Research & Development<br>
MARIN | T <a href="tel:+31%20317%20493%20344" value="+31317493344" target="_blank">+31 317 49 33 44</a> | mailto:<a href="mailto:C.Klaij@marin.nl" target="_blank">C.Klaij@marin.nl</a> |
<a href="http://www.marin.nl" rel="noreferrer" target="_blank">http://www.marin.nl</a><br>
<br>
MARIN news: <a href="http://www.marin.nl/web/News/News-items/Seminar-Blauwe-toekomst-versnellen-van-innovaties-door-samenwerken.htm" rel="noreferrer" target="_blank">
http://www.marin.nl/web/News/N<wbr>ews-items/Seminar-Blauwe-toeko<wbr>mst-versnellen-van-innovaties-<wbr>door-samenwerken.htm</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div></div></div>
<u></u>
<p></p>
<u></u><u></u><u></u>
</div>
</blockquote></div><br></div>