<div dir="ltr"><div dir="ltr">On Wed, Oct 13, 2021 at 5:53 AM Roland Richter <<a href="mailto:roland.richter@ntnu.no">roland.richter@ntnu.no</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hei,</p>
<p>I noticed a difference in when the program is running, and when
not. The code works fine if I compile it via a CMake-file and load
PETSc there. If I use the compilation line which is included in
the Makefiles, then the code will fail with the mentioned error.
The cmake-generated compilation line (including armadillo, because
my test sample contained armadillo-code) is<br>
</p>
<p><i></i></p></div></blockquote><div>One of these is a compile command and the other is a link command.</div><div><br></div><div> Matt</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><p><i>/opt/intel/oneapi/mpi/2021.4.0/bin/mpicxx -D__INSDIR__=""
-I/include -I/opt/petsc/include -I/opt/armadillo/include
-std=c++0x -g -MD -MT CMakeFiles/main.dir/source/main.cpp.o -MF
CMakeFiles/main.dir/source/main.cpp.o.d -o
CMakeFiles/main.dir/source/main.cpp.o -c source/main.cpp</i><i><br>
</i><i>/opt/intel/oneapi/mpi/2021.4.0/bin/mpicxx -rdynamic
CMakeFiles/main.dir/source/main.cpp.o -o main_short
-Wl,-rpath,/opt/petsc/lib:/opt/armadillo/lib64
/opt/petsc/lib/libpetsc.so /opt/armadillo/lib64/libarmadillo.so
</i><br>
</p>
<p>Meanwhile, the original compilation line from PETSc is</p>
<p><i>mpicxx -mavx2 -march=native -O3 -fPIC -fopenmp
-I/opt/petsc/include -I/opt/armadillo/include
-I/opt/intel/oneapi/mkl/latest/include -I/opt/fftw3/include
-I/opt/hdf5/include -I/opt/boost/include source/main.cpp
-Wl,-rpath,/opt/petsc/lib -L/opt/petsc/lib
-Wl,-rpath,/opt/petsc/lib -L/opt/petsc/lib
-L/opt/intel/oneapi/mkl/latest/lib/intel64
-Wl,-rpath,/opt/fftw3/lib64 -L/opt/fftw3/lib64
-Wl,-rpath,/opt/armadillo/lib64 -L/opt/armadillo/lib64
-Wl,-rpath,/opt/intel/oneapi/mkl/latest/lib/intel64
-Wl,-rpath,/opt/hdf5/lib -L/opt/hdf5/lib
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.4.0/lib/release
-L/opt/intel/oneapi/mpi/2021.4.0/lib/release
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.4.0/lib
-L/opt/intel/oneapi/mpi/2021.4.0/lib
-Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/11
-L/usr/lib64/gcc/x86_64-suse-linux/11
-Wl,-rpath,/opt/intel/oneapi/vpl/2021.6.0/lib
-L/opt/intel/oneapi/vpl/2021.6.0/lib
-Wl,-rpath,/opt/intel/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8
-L/opt/intel/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.4.0/libfabric/lib
-L/opt/intel/oneapi/mpi/2021.4.0/libfabric/lib
-Wl,-rpath,/opt/intel/oneapi/mkl/2021.4.0/lib/intel64
-L/opt/intel/oneapi/mkl/2021.4.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/ipp/2021.4.0/lib/intel64
-L/opt/intel/oneapi/ipp/2021.4.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/ippcp/2021.4.0/lib/intel64
-L/opt/intel/oneapi/ippcp/2021.4.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/dnnl/2021.4.0/cpu_dpcpp_gpu_dpcpp/lib
-L/opt/intel/oneapi/dnnl/2021.4.0/cpu_dpcpp_gpu_dpcpp/lib
-Wl,-rpath,/opt/intel/oneapi/dal/2021.4.0/lib/intel64
-L/opt/intel/oneapi/dal/2021.4.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/compiler/2021.4.0/linux/compiler/lib/intel64_lin
-L/opt/intel/oneapi/compiler/2021.4.0/linux/compiler/lib/intel64_lin
-Wl,-rpath,/opt/intel/oneapi/compiler/2021.4.0/linux/lib
-L/opt/intel/oneapi/compiler/2021.4.0/linux/lib
-Wl,-rpath,/opt/intel/oneapi/clck/2021.4.0/lib/intel64
-L/opt/intel/oneapi/clck/2021.4.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp
-L/opt/intel/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp
-Wl,-rpath,/usr/x86_64-suse-linux/lib
-L/usr/x86_64-suse-linux/lib -larmadillo -lpetsc -lHYPRE
-lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord
-lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -lspqr -lumfpack
-lklu -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd
-lsuitesparseconfig -lsuperlu -lsuperlu_dist -lEl
-lElSuiteSparse -lpmrrr -lfftw3_mpi -lfftw3 -lp4est -lsc
-lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -liomp5 -ldl
-lpthread -lptesmumps -lptscotchparmetis -lptscotch
-lptscotcherr -lesmumps -lscotch -lscotcherr -lhdf5_hl -lhdf5
-lparmetis -lmetis -lm -lz -lmuparser -lX11 -lstdc++ -ldl
-lmpifort -lmpi -lrt -lpthread -lgfortran -lm -lgfortran -lm
-lgcc_s -lquadmath -lrt -lquadmath -lstdc++ -ldl -o main_long</i></p>
<p>Both executables have the same libraries linked to them, but in a
different order when comparing them with ldd.<br>
</p>
<p>Does that explain the observed behavior?</p>
<p>Thanks,</p>
<p>regards,</p>
<p>Roland<br>
</p>
<div>Am 11.10.21 um 15:13 schrieb Roland
Richter:<br>
</div>
<blockquote type="cite">
<p>Hei,</p>
<p>the following code works fine:</p>
<p>#include <iostream><br>
#include <petsc.h><br>
<br>
static char help[] = "Solves 2D Poisson equation using
multigrid.\n\n";<br>
int main(int argc,char **argv) {<br>
PetscInitialize(&argc,&argv,(char*)0,help);<br>
std::cout << "Hello World\n";<br>
PetscFinalize();<br>
return 0;<br>
}</p>
<p>Regards,</p>
<p>Roland<br>
</p>
<div>Am 11.10.21 um 14:34 schrieb Stefano
Zampini:<br>
</div>
<blockquote type="cite"> Can
you try with a simple call that only calls
PetscInitialize/Finalize?
<div><br>
<div>
<div><br>
<blockquote type="cite">
<div>On Oct 11, 2021, at 3:30 PM, Roland
Richter <<a href="mailto:roland.richter@ntnu.no" target="_blank">roland.richter@ntnu.no</a>>
wrote:</div>
<br>
<div>
<div>
<p>At least according to configure.log
mpiexec was defined as</p>
<p>Checking for program
/opt/intel/oneapi/mpi/2021.4.0//bin/mpiexec...found<br>
Defined make macro
"MPIEXECEXECUTABLE" to
"/opt/intel/oneapi/mpi/2021.4.0/bin/mpiexec"</p>
<p>When running ex19 with this mpiexec it
fails with the usual error, even though all
configuration steps worked fine. I attached the
configuration log.<br>
</p>
<p>Regards,</p>
<p>Roland<br>
</p>
<div>Am 11.10.21 um 14:24
schrieb Stefano Zampini:<br>
</div>
<blockquote type="cite"> You are most probably using a different
mpiexec then the one used to compile petsc.
<div><br>
</div>
<div><br>
<div><br>
<blockquote type="cite">
<div>On Oct 11, 2021, at 3:23 PM,
Roland Richter <<a href="mailto:roland.richter@ntnu.no" target="_blank">roland.richter@ntnu.no</a>>
wrote:</div>
<br>
<div>
<div>
<p>I tried either <i>./ex19</i>
(SNES-example), <i>mpirun
./ex19</i> or <i>mpirun -n
1 ./ex19</i>, all with the same
result.</p>
<p>Regards,</p>
<p>Roland<br>
</p>
<div>Am 11.10.21
um 14:22 schrieb Matthew Knepley:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">On Mon, Oct
11, 2021 at 8:07 AM Roland Richter
<<a href="mailto:roland.richter@ntnu.no" target="_blank">roland.richter@ntnu.no</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hei,</p>
<p>at least in gdb it
fails with</p>
<p><span style="font-family:monospace"><span style="background-color:rgb(255,255,255)">Attempting to
use an MPI routine
before initializing
MPICH </span><br>
[Inferior 1 (process 7854)
exited with code 01] <br>
(gdb) backtrace <br>
No stack.<br>
</span></p>
</div>
</blockquote>
<div><br>
</div>
<div>What were you
running? If it never makes it
into PETSc code, I am not sure
what we are</div>
<div>doing to cause this.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p><span style="font-family:monospace"> </span>Regards,</p>
<p>Roland<br>
<span style="font-family:monospace"></span></p>
<div>Am 11.10.21 um
13:57 schrieb Matthew
Knepley:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">On
Mon, Oct 11, 2021 at
5:24 AM Roland Richter
<<a href="mailto:roland.richter@ntnu.no" target="_blank">roland.richter@ntnu.no</a>> wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hei,<br>
<br>
I compiled PETSc with
Intel MPI (MPICH) and
GCC as compiler (i.e.
using<br>
Intel OneAPI together
with the supplied
mpicxx-compiler).
Compilation<br>
and installation
worked fine, but
running the tests
resulted in the<br>
error "Attempting to
use an MPI routine
before initializing
MPICH". A<br>
simple test program
(attached) worked fine
with the same
combination.<br>
<br>
What could be the
reason for that?<br>
</blockquote>
<div><br>
</div>
<div>Hi Roland,</div>
<div><br>
</div>
<div>Can you
get a stack trace for
this error using the
debugger?</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Thanks!<br>
<br>
Regards,<br>
<br>
Roland Richter<br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What
most
experimenters
take for
granted before
they begin
their
experiments is
infinitely
more
interesting
than any
results to
which their
experiments
lead.<br>
-- Norbert
Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most
experimenters take for
granted before they
begin their
experiments is
infinitely more
interesting than any
results to which their
experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
<span id="gmail-m_5718787638988412959cid:3B71A0CF-7121-40E3-AC68-DB609803623C"><configure.log></span></div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</blockquote>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>