<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hei,</p>
<p>I noticed a difference in when the program is running, and when
not. The code works fine if I compile it via a CMake-file and load
PETSc there. If I use the compilation line which is included in
the Makefiles, then the code will fail with the mentioned error.
The cmake-generated compilation line (including armadillo, because
my test sample contained armadillo-code) is<br>
</p>
<p><i>/opt/intel/oneapi/mpi/2021.4.0/bin/mpicxx -D__INSDIR__=""
-I/include -I/opt/petsc/include -I/opt/armadillo/include
-std=c++0x -g -MD -MT CMakeFiles/main.dir/source/main.cpp.o -MF
CMakeFiles/main.dir/source/main.cpp.o.d -o
CMakeFiles/main.dir/source/main.cpp.o -c source/main.cpp</i><i><br>
</i><i>/opt/intel/oneapi/mpi/2021.4.0/bin/mpicxx -rdynamic
CMakeFiles/main.dir/source/main.cpp.o -o main_short
-Wl,-rpath,/opt/petsc/lib:/opt/armadillo/lib64
/opt/petsc/lib/libpetsc.so /opt/armadillo/lib64/libarmadillo.so
</i><br>
</p>
<p>Meanwhile, the original compilation line from PETSc is</p>
<p><i>mpicxx -mavx2 -march=native -O3 -fPIC -fopenmp
-I/opt/petsc/include -I/opt/armadillo/include
-I/opt/intel/oneapi/mkl/latest/include -I/opt/fftw3/include
-I/opt/hdf5/include -I/opt/boost/include source/main.cpp
-Wl,-rpath,/opt/petsc/lib -L/opt/petsc/lib
-Wl,-rpath,/opt/petsc/lib -L/opt/petsc/lib
-L/opt/intel/oneapi/mkl/latest/lib/intel64
-Wl,-rpath,/opt/fftw3/lib64 -L/opt/fftw3/lib64
-Wl,-rpath,/opt/armadillo/lib64 -L/opt/armadillo/lib64
-Wl,-rpath,/opt/intel/oneapi/mkl/latest/lib/intel64
-Wl,-rpath,/opt/hdf5/lib -L/opt/hdf5/lib
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.4.0/lib/release
-L/opt/intel/oneapi/mpi/2021.4.0/lib/release
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.4.0/lib
-L/opt/intel/oneapi/mpi/2021.4.0/lib
-Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/11
-L/usr/lib64/gcc/x86_64-suse-linux/11
-Wl,-rpath,/opt/intel/oneapi/vpl/2021.6.0/lib
-L/opt/intel/oneapi/vpl/2021.6.0/lib
-Wl,-rpath,/opt/intel/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8
-L/opt/intel/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.4.0/libfabric/lib
-L/opt/intel/oneapi/mpi/2021.4.0/libfabric/lib
-Wl,-rpath,/opt/intel/oneapi/mkl/2021.4.0/lib/intel64
-L/opt/intel/oneapi/mkl/2021.4.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/ipp/2021.4.0/lib/intel64
-L/opt/intel/oneapi/ipp/2021.4.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/ippcp/2021.4.0/lib/intel64
-L/opt/intel/oneapi/ippcp/2021.4.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/dnnl/2021.4.0/cpu_dpcpp_gpu_dpcpp/lib
-L/opt/intel/oneapi/dnnl/2021.4.0/cpu_dpcpp_gpu_dpcpp/lib
-Wl,-rpath,/opt/intel/oneapi/dal/2021.4.0/lib/intel64
-L/opt/intel/oneapi/dal/2021.4.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/compiler/2021.4.0/linux/compiler/lib/intel64_lin
-L/opt/intel/oneapi/compiler/2021.4.0/linux/compiler/lib/intel64_lin
-Wl,-rpath,/opt/intel/oneapi/compiler/2021.4.0/linux/lib
-L/opt/intel/oneapi/compiler/2021.4.0/linux/lib
-Wl,-rpath,/opt/intel/oneapi/clck/2021.4.0/lib/intel64
-L/opt/intel/oneapi/clck/2021.4.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp
-L/opt/intel/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp
-Wl,-rpath,/usr/x86_64-suse-linux/lib
-L/usr/x86_64-suse-linux/lib -larmadillo -lpetsc -lHYPRE
-lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord
-lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -lspqr -lumfpack
-lklu -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd
-lsuitesparseconfig -lsuperlu -lsuperlu_dist -lEl
-lElSuiteSparse -lpmrrr -lfftw3_mpi -lfftw3 -lp4est -lsc
-lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -liomp5 -ldl
-lpthread -lptesmumps -lptscotchparmetis -lptscotch
-lptscotcherr -lesmumps -lscotch -lscotcherr -lhdf5_hl -lhdf5
-lparmetis -lmetis -lm -lz -lmuparser -lX11 -lstdc++ -ldl
-lmpifort -lmpi -lrt -lpthread -lgfortran -lm -lgfortran -lm
-lgcc_s -lquadmath -lrt -lquadmath -lstdc++ -ldl -o main_long</i></p>
<p>Both executables have the same libraries linked to them, but in a
different order when comparing them with ldd.<br>
</p>
<p>Does that explain the observed behavior?</p>
<p>Thanks,</p>
<p>regards,</p>
<p>Roland<br>
</p>
<div class="moz-cite-prefix">Am 11.10.21 um 15:13 schrieb Roland
Richter:<br>
</div>
<blockquote type="cite"
cite="mid:b30b3c1f-3ed9-4dfd-fb95-fdfdbbbf7755@ntnu.no">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<p>Hei,</p>
<p>the following code works fine:</p>
<p>#include <iostream><br>
#include <petsc.h><br>
<br>
static char help[] = "Solves 2D Poisson equation using
multigrid.\n\n";<br>
int main(int argc,char **argv) {<br>
PetscInitialize(&argc,&argv,(char*)0,help);<br>
std::cout << "Hello World\n";<br>
PetscFinalize();<br>
return 0;<br>
}</p>
<p>Regards,</p>
<p>Roland<br>
</p>
<div class="moz-cite-prefix">Am 11.10.21 um 14:34 schrieb Stefano
Zampini:<br>
</div>
<blockquote type="cite"
cite="mid:383F2AEB-4CEB-4407-A1B3-E294ACDFD91C@gmail.com"> Can
you try with a simple call that only calls
PetscInitialize/Finalize?
<div class=""><br class="">
<div class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Oct 11, 2021, at 3:30 PM, Roland
Richter <<a href="mailto:roland.richter@ntnu.no"
class="moz-txt-link-freetext" moz-do-not-send="true">roland.richter@ntnu.no</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">
<p class="">At least according to configure.log
mpiexec was defined as</p>
<p class="">Checking for program
/opt/intel/oneapi/mpi/2021.4.0//bin/mpiexec...found<br
class="">
Defined make macro
"MPIEXECEXECUTABLE" to
"/opt/intel/oneapi/mpi/2021.4.0/bin/mpiexec"</p>
<p class="">When running ex19 with this mpiexec it
fails with the usual error, even though all
configuration steps worked fine. I attached the
configuration log.<br class="">
</p>
<p class="">Regards,</p>
<p class="">Roland<br class="">
</p>
<div class="moz-cite-prefix">Am 11.10.21 um 14:24
schrieb Stefano Zampini:<br class="">
</div>
<blockquote type="cite"
cite="mid:1280E270-AAA1-44CE-AE2A-93A2B70462E2@gmail.com"
class=""> You are most probably using a different
mpiexec then the one used to compile petsc.
<div class=""><br class="">
</div>
<div class=""><br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Oct 11, 2021, at 3:23 PM,
Roland Richter <<a
href="mailto:roland.richter@ntnu.no"
class="moz-txt-link-freetext"
moz-do-not-send="true">roland.richter@ntnu.no</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">
<p class="">I tried either <i class="">./ex19</i>
(SNES-example), <i class="">mpirun
./ex19</i> or <i class="">mpirun -n
1 ./ex19</i>, all with the same
result.</p>
<p class="">Regards,</p>
<p class="">Roland<br class="">
</p>
<div class="moz-cite-prefix">Am 11.10.21
um 14:22 schrieb Matthew Knepley:<br
class="">
</div>
<blockquote type="cite"
cite="mid:CAMYG4Gk+KxdcmJ27ZEoS1UFK4bfY5hgD8PLuc7rL4UNe219K3g@mail.gmail.com"
class="">
<div dir="ltr" class="">
<div dir="ltr" class="">On Mon, Oct
11, 2021 at 8:07 AM Roland Richter
<<a
href="mailto:roland.richter@ntnu.no"
moz-do-not-send="true"
class="moz-txt-link-freetext">roland.richter@ntnu.no</a>>
wrote:<br class="">
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div class="">
<p class="">Hei,</p>
<p class="">at least in gdb it
fails with</p>
<p class=""><span
style="font-family:monospace"
class=""><span
style="background-color:
rgb(255, 255, 255);"
class="">Attempting to
use an MPI routine
before initializing
MPICH </span><br
class="">
[Inferior 1 (process 7854)
exited with code 01] <br
class="">
(gdb) backtrace <br
class="">
No stack.<br class="">
</span></p>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">What were you
running? If it never makes it
into PETSc code, I am not sure
what we are</div>
<div class="">doing to cause this.</div>
<div class=""><br class="">
</div>
<div class=""> Thanks,</div>
<div class=""><br class="">
</div>
<div class=""> Matt</div>
<div class=""> </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div class="">
<p class=""><span
style="font-family:monospace"
class=""> </span>Regards,</p>
<p class="">Roland<br class="">
<span
style="font-family:monospace"
class=""></span></p>
<div class="">Am 11.10.21 um
13:57 schrieb Matthew
Knepley:<br class="">
</div>
<blockquote type="cite"
class="">
<div dir="ltr" class="">
<div dir="ltr" class="">On
Mon, Oct 11, 2021 at
5:24 AM Roland Richter
<<a
href="mailto:roland.richter@ntnu.no"
target="_blank"
moz-do-not-send="true"
class="moz-txt-link-freetext">roland.richter@ntnu.no</a>> wrote:<br
class="">
</div>
<div class="gmail_quote">
<blockquote
class="gmail_quote"
style="margin:0px 0px
0px
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex">Hei,<br
class="">
<br class="">
I compiled PETSc with
Intel MPI (MPICH) and
GCC as compiler (i.e.
using<br class="">
Intel OneAPI together
with the supplied
mpicxx-compiler).
Compilation<br
class="">
and installation
worked fine, but
running the tests
resulted in the<br
class="">
error "Attempting to
use an MPI routine
before initializing
MPICH". A<br class="">
simple test program
(attached) worked fine
with the same
combination.<br
class="">
<br class="">
What could be the
reason for that?<br
class="">
</blockquote>
<div class=""><br
class="">
</div>
<div class="">Hi Roland,</div>
<div class=""><br
class="">
</div>
<div class="">Can you
get a stack trace for
this error using the
debugger?</div>
<div class=""><br
class="">
</div>
<div class=""> Thanks,</div>
<div class=""><br
class="">
</div>
<div class=""> Matt</div>
<div class=""> </div>
<blockquote
class="gmail_quote"
style="margin:0px 0px
0px
0.8ex;border-left:1px
solid
rgb(204,204,204);padding-left:1ex">
Thanks!<br class="">
<br class="">
Regards,<br class="">
<br class="">
Roland Richter<br
class="">
</blockquote>
</div>
<br class="" clear="all">
<div class=""><br class="">
</div>
-- <br class="">
<div dir="ltr" class="">
<div dir="ltr" class="">
<div class="">
<div dir="ltr"
class="">
<div class="">
<div dir="ltr"
class="">
<div class="">What
most
experimenters
take for
granted before
they begin
their
experiments is
infinitely
more
interesting
than any
results to
which their
experiments
lead.<br
class="">
-- Norbert
Wiener</div>
<div class=""><br
class="">
</div>
<div class=""><a
href="http://www.cse.buffalo.edu/~knepley/" target="_blank"
moz-do-not-send="true"
class="">https://www.cse.buffalo.edu/~knepley/</a><br
class="">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br class="" clear="all">
<div class=""><br class="">
</div>
-- <br class="">
<div dir="ltr"
class="gmail_signature">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div class="">
<div dir="ltr" class="">
<div class="">What most
experimenters take for
granted before they
begin their
experiments is
infinitely more
interesting than any
results to which their
experiments lead.<br
class="">
-- Norbert Wiener</div>
<div class=""><br
class="">
</div>
<div class=""><a
href="http://www.cse.buffalo.edu/~knepley/"
target="_blank"
moz-do-not-send="true"
class="">https://www.cse.buffalo.edu/~knepley/</a><br
class="">
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</blockquote>
</div>
<span id="cid:3B71A0CF-7121-40E3-AC68-DB609803623C"><configure.log></span></div>
</blockquote>
</div>
<br class="">
</div>
</div>
</blockquote>
</blockquote>
</body>
</html>