[petsc-dev] "thread safe"

Tue Feb 17 20:14:00 CST 2015

We have been testing master with a code that calls PETSc serial LU solvers
from threads.  I have seen system messages with OMP (see way below) and
Robert (cc'ed) reported this useful stack trace.

I have not modified my (non-thread) build.  Perhaps I need to or are there
PETSc runtime options?

This is a Cray XC30 with Intel.

Thanks,
Mark

SC[0;39mESC[0;49m[116]PETSC ERROR:* Object is in wrong state*
[116]PETSC ERROR: Logging event had unbalanced begin/end pairs
[116]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[116]PETSC ERROR: Petsc Development GIT revision: v3.5.3-1570-gcaf1481  GIT
Date: 2015-02-07 17:34:17 -0600
[116]PETSC ERROR: ./xgca_petsc36_col on a arch-xc30-opt64-intel named
nid05975 by rhager Tue Feb 17 10:46:32 2015
[116]PETSC ERROR: Configure options --COPTFLAGS="-fast -no-ipo"
--CXXOPTFLAGS="-fast -no-ipoi" --FOPTFLAGS="-fast -no-ipo" --download-hypre
--download-superlu_dist --
download-parmetis --download-metis --with-ssl=0 --with-cc=cc
--with-clib-autodetect=0 --with-cxx=CC --with-cxxlib-autodetect=0
--with-debugging=0 --with-fc=ftn --with
-fortranlib-autodetect=0
--with-hdf5-dir=/opt/cray/hdf5-parallel/1.8.13/intel/140/
--with-shared-libraries=0 --with-x=0 --with-mpiexec=aprun LIBS=-lstdc++
--with-64-b
it-indices PETSC_ARCH=arch-xc30-opt64-intel
PETSC_DIR=/global/u2/m/madams/petsc_master
[116]PETSC ERROR: #1 PetscLogEventEndDefault() line 694 in
/global/u2/m/madams/petsc_master/src/sys/logging/utils/eventlog.c
[116]PETSC ERROR: #2 MatLUFactorSymbolic() line 2894 in
/global/u2/m/madams/petsc_master/src/mat/interface/matrix.c
[116]PETSC ERROR: #3 PCSetUp_LU() line 127 in
/global/u2/m/madams/petsc_master/src/ksp/pc/impls/factor/lu/lu.c
[116]PETSC ERROR: #4 PCSetUp() line 918 in
/global/u2/m/madams/petsc_master/src/ksp/pc/interface/precon.c
[116]PETSC ERROR: #5 KSPSetUp() line 306 in
/global/u2/m/madams/petsc_master/src/ksp/ksp/interface/itfunc.c
[116]PETSC ERROR: #6 KSPSolve() line 503 in
/global/u2/m/madams/petsc_master/src/ksp/ksp/interface/itfunc.c

Other error message:

OMP: Error #13: Assertion failure at kmp_runtime.c(1588).
OMP: Hint: Please submit a bug report with this message, compile and run
commands used, and machine configuration info including native compiler and
operating system versions. Faster response will be obtained by including
all program sources. For information on submitting this issue, please see
http://www.intel.com/software/products/support/.
_pmiu_daemon(SIGCHLD): [NID 05979] [c7-3c0s6n3] [Tue Feb 17 15:14:43 2015]
PE RANK 23 exit signal Killed
_pmiu_daemon(SIGCHLD): [NID 05976] [c7-3c0s6n0] [Tue Feb 17 15:14:43 2015]
PE RANK 10 exit signal Killed
[NID 05979] 2015-02-17 15:14:43 Apid 10147992: initiated application
termination
[NID 05979] 2015-02-17 15:14:43 Apid 10147992: Cray HSN detected critical
error 0x4416[ptag 239]. Please contact admin for details. Killing pid
18637(xgca)
[NID 05976] 2015-02-17 15:14:43 Apid 10147992: Cray HSN detected critical
error 0x4416[ptag 73]. Please contact admin for details. Killing pid
15380(xgca)
[NID 05984] 2015-02-17 15:14:43 Apid 10147992: Cray HSN detected critical
error 0x4416[ptag 0]. Please contact admin for details. Killing pid
34636(xgca)
[NID 05988] 2015-02-17 15:14:43 Apid 10147992: Cray HSN detected critical
error 0x4416[ptag 59]. Please contact admin for details. Killing pid
38496(xgca)
[NID 06019] 2015-02-17 15:14:43 Apid 10147992: Cray HSN detected critical
error 0x4416[ptag 0]. Please contact admin for details. Killing pid
11132(xgca)
[NID 05980] 2015-02-17 15:14:43 Apid 10147992: Cray HSN detected critical
error 0x4416[ptag 0]. Please contact admin for details. Killing pid
8320(xgca)
[NID 05993] 2015-02-17 15:14:43 Apid 10147992: Cray HSN detected critical
error 0x4416[ptag 0]. Please contact admin for details. Killing pid
46182(xgca)
[NID 06020] 2015-02-17 15:14:43 Apid 10147992: Cray HSN detected critical
error 0x4416[ptag 249]. Please contact admin for details. Killing pid
23753(xgca)
[NID 05987] 2015-02-17 15:14:43 Apid 10147992: Cray HSN detected critical
error 0x4416[ptag 87]. Please contact admin for details. Killing pid
11254(xgca)
[NID 05986] 2015-02-17 15:14:43 Apid 10147992: Cray HSN detected critical
error 0x4416[ptag 41]. Please contact admin for details. Killing pid
6630(xgca)
[NID 05981] 2015-02-17 15:14:43 Apid 10147992: Cray HSN detected critical
error 0x4416[ptag 31]. Please contact admin for details. Killing pid
10520(xgca)
[NID 05999] 2015-02-17 15:14:43 Apid 10147992: Cray HSN detected critical
error 0x4416[ptag 7]. Please contact admin for details. Killing pid
1843(xgca)
[NID 05985] 2015-02-17 15:14:43 Apid 10147992: Cray HSN detected critical
error 0x4416[ptag 0]. Please contact admin for details. Killing pid
26498(xgca)
[NID 05998] 2015-02-17 15:14:43 Apid 10147992: Cray HSN detected critical
error 0x4416[ptag 209]. Please contact admin for details. Killing pid
20387(xgca)
[NID 05994] 2015-02-17 15:14:53 Apid 10147992: Cray HSN detected critical
error 0x4416[ptag 0]. Please contact admin for details. Killing pid
39462(xgca)
[NID 05983] 2015-02-17 15:14:53 Apid 10147992: Cray HSN detected critical
error 0x4416[ptag 0]. Please contact admin for details. Killing pid
18598(xgca)
[NID 05995] 2015-02-17 15:14:54 Apid 10147992: Cray HSN detected critical
error 0x4416[ptag 0]. Please contact admin for details. Killing pid
42322(xgca)
[NID 05996] 2015-02-17 15:14:54 Apid 10147992: Cray HSN detected critical
error 0x4416[ptag 0]. Please contact admin for details. Killing pid
34248(xgca)
[NID 05978] 2015-02-17 15:14:55 Apid 10147992: Cray HSN detected critical
error 0x4416[ptag 0]. Please contact admin for details. Killing pid
9483(xgca)
[NID 05975] 2015-02-17 15:14:56 Apid 10147992: Cray HSN detected critical
error 0x4416[ptag 0]. Please contact admin for details. Killing pid
11470(xgca)
Application 10147992 exit codes: 137
Application 10147992 exit signals: Killed
Application 10147992 resources: utime ~2194s, stime ~199s, Rss ~488560,
inblocks ~908164, outblocks ~2571652
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20150217/ff334e70/attachment.html>