<div dir="ltr"><div dir="ltr">On Wed, Oct 13, 2021 at 6:32 AM Roland Richter <<a href="mailto:roland.richter@ntnu.no">roland.richter@ntnu.no</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Yes, the first part (which works) consists out of a compilation
line and a linking line, while the second command is a combination
of compile- and linking line.<br>
</p>
<div></div></div></blockquote><div>The link line in the first does not tell us anything because MPI is not even present. It is being pulled in I presume from libarmadillo, which we cannot see. It still</div><div>seems most likely, as Stefano said, that you are mixing versions of MPI.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div>Am 13.10.21 um 12:26 schrieb Matthew
Knepley:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">On Wed, Oct 13, 2021 at 5:53 AM Roland Richter
<<a href="mailto:roland.richter@ntnu.no" target="_blank">roland.richter@ntnu.no</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hei,</p>
<p>I noticed a difference in when the program is running,
and when not. The code works fine if I compile it via a
CMake-file and load PETSc there. If I use the
compilation line which is included in the Makefiles,
then the code will fail with the mentioned error. The
cmake-generated compilation line (including armadillo,
because my test sample contained armadillo-code) is<br>
</p>
</div>
</blockquote>
<div>One of these is a compile command and the other is a link
command.</div>
<div><br>
</div>
<div> Matt</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p><i>/opt/intel/oneapi/mpi/2021.4.0/bin/mpicxx
-D__INSDIR__="" -I/include -I/opt/petsc/include
-I/opt/armadillo/include -std=c++0x -g -MD -MT
CMakeFiles/main.dir/source/main.cpp.o -MF
CMakeFiles/main.dir/source/main.cpp.o.d -o
CMakeFiles/main.dir/source/main.cpp.o -c
source/main.cpp</i><i><br>
</i><i>/opt/intel/oneapi/mpi/2021.4.0/bin/mpicxx
-rdynamic CMakeFiles/main.dir/source/main.cpp.o -o
main_short
-Wl,-rpath,/opt/petsc/lib:/opt/armadillo/lib64
/opt/petsc/lib/libpetsc.so
/opt/armadillo/lib64/libarmadillo.so </i><br>
</p>
<p>Meanwhile, the original compilation line from PETSc is</p>
<p><i>mpicxx -mavx2 -march=native -O3 -fPIC -fopenmp
-I/opt/petsc/include -I/opt/armadillo/include
-I/opt/intel/oneapi/mkl/latest/include
-I/opt/fftw3/include -I/opt/hdf5/include
-I/opt/boost/include source/main.cpp
-Wl,-rpath,/opt/petsc/lib -L/opt/petsc/lib
-Wl,-rpath,/opt/petsc/lib -L/opt/petsc/lib
-L/opt/intel/oneapi/mkl/latest/lib/intel64
-Wl,-rpath,/opt/fftw3/lib64 -L/opt/fftw3/lib64
-Wl,-rpath,/opt/armadillo/lib64 -L/opt/armadillo/lib64
-Wl,-rpath,/opt/intel/oneapi/mkl/latest/lib/intel64
-Wl,-rpath,/opt/hdf5/lib -L/opt/hdf5/lib
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.4.0/lib/release
-L/opt/intel/oneapi/mpi/2021.4.0/lib/release
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.4.0/lib
-L/opt/intel/oneapi/mpi/2021.4.0/lib
-Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/11
-L/usr/lib64/gcc/x86_64-suse-linux/11
-Wl,-rpath,/opt/intel/oneapi/vpl/2021.6.0/lib
-L/opt/intel/oneapi/vpl/2021.6.0/lib
-Wl,-rpath,/opt/intel/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8
-L/opt/intel/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.4.0/libfabric/lib
-L/opt/intel/oneapi/mpi/2021.4.0/libfabric/lib
-Wl,-rpath,/opt/intel/oneapi/mkl/2021.4.0/lib/intel64
-L/opt/intel/oneapi/mkl/2021.4.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/ipp/2021.4.0/lib/intel64
-L/opt/intel/oneapi/ipp/2021.4.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/ippcp/2021.4.0/lib/intel64
-L/opt/intel/oneapi/ippcp/2021.4.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/dnnl/2021.4.0/cpu_dpcpp_gpu_dpcpp/lib
-L/opt/intel/oneapi/dnnl/2021.4.0/cpu_dpcpp_gpu_dpcpp/lib
-Wl,-rpath,/opt/intel/oneapi/dal/2021.4.0/lib/intel64
-L/opt/intel/oneapi/dal/2021.4.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/compiler/2021.4.0/linux/compiler/lib/intel64_lin
-L/opt/intel/oneapi/compiler/2021.4.0/linux/compiler/lib/intel64_lin
-Wl,-rpath,/opt/intel/oneapi/compiler/2021.4.0/linux/lib
-L/opt/intel/oneapi/compiler/2021.4.0/linux/lib
-Wl,-rpath,/opt/intel/oneapi/clck/2021.4.0/lib/intel64
-L/opt/intel/oneapi/clck/2021.4.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp
-L/opt/intel/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp
-Wl,-rpath,/usr/x86_64-suse-linux/lib
-L/usr/x86_64-suse-linux/lib -larmadillo -lpetsc
-lHYPRE -lcmumps -ldmumps -lsmumps -lzmumps
-lmumps_common -lpord -lmkl_scalapack_lp64
-lmkl_blacs_intelmpi_lp64 -lspqr -lumfpack -lklu
-lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd
-lsuitesparseconfig -lsuperlu -lsuperlu_dist -lEl
-lElSuiteSparse -lpmrrr -lfftw3_mpi -lfftw3 -lp4est
-lsc -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread
-liomp5 -ldl -lpthread -lptesmumps -lptscotchparmetis
-lptscotch -lptscotcherr -lesmumps -lscotch
-lscotcherr -lhdf5_hl -lhdf5 -lparmetis -lmetis -lm
-lz -lmuparser -lX11 -lstdc++ -ldl -lmpifort -lmpi
-lrt -lpthread -lgfortran -lm -lgfortran -lm -lgcc_s
-lquadmath -lrt -lquadmath -lstdc++ -ldl -o main_long</i></p>
<p>Both executables have the same libraries linked to
them, but in a different order when comparing them with
ldd.<br>
</p>
<p>Does that explain the observed behavior?</p>
<p>Thanks,</p>
<p>regards,</p>
<p>Roland<br>
</p>
<div>Am 11.10.21 um 15:13 schrieb Roland Richter:<br>
</div>
<blockquote type="cite">
<p>Hei,</p>
<p>the following code works fine:</p>
<p>#include <iostream><br>
#include <petsc.h><br>
<br>
static char help[] = "Solves 2D Poisson equation using
multigrid.\n\n";<br>
int main(int argc,char **argv) {<br>
PetscInitialize(&argc,&argv,(char*)0,help);<br>
std::cout << "Hello World\n";<br>
PetscFinalize();<br>
return 0;<br>
}</p>
<p>Regards,</p>
<p>Roland<br>
</p>
<div>Am 11.10.21 um 14:34 schrieb Stefano Zampini:<br>
</div>
<blockquote type="cite"> Can you try with a simple call
that only calls PetscInitialize/Finalize?
<div><br>
<div>
<div><br>
<blockquote type="cite">
<div>On Oct 11, 2021, at 3:30 PM, Roland
Richter <<a href="mailto:roland.richter@ntnu.no" target="_blank">roland.richter@ntnu.no</a>>
wrote:</div>
<br>
<div>
<div>
<p>At least according to configure.log
mpiexec was defined as</p>
<p>Checking for program
/opt/intel/oneapi/mpi/2021.4.0//bin/mpiexec...found<br>
Defined make macro
"MPIEXECEXECUTABLE" to
"/opt/intel/oneapi/mpi/2021.4.0/bin/mpiexec"</p>
<p>When running ex19 with this mpiexec it
fails with the usual error, even though
all configuration steps worked fine. I
attached the configuration log.<br>
</p>
<p>Regards,</p>
<p>Roland<br>
</p>
<div>Am 11.10.21 um 14:24 schrieb Stefano
Zampini:<br>
</div>
<blockquote type="cite"> You are most
probably using a different mpiexec then
the one used to compile petsc.
<div><br>
</div>
<div><br>
<div><br>
<blockquote type="cite">
<div>On Oct 11, 2021, at 3:23 PM,
Roland Richter <<a href="mailto:roland.richter@ntnu.no" target="_blank">roland.richter@ntnu.no</a>>
wrote:</div>
<br>
<div>
<div>
<p>I tried either <i>./ex19</i>
(SNES-example), <i>mpirun
./ex19</i> or <i>mpirun
-n 1 ./ex19</i>, all with
the same result.</p>
<p>Regards,</p>
<p>Roland<br>
</p>
<div>Am 11.10.21 um 14:22
schrieb Matthew Knepley:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">On Mon, Oct
11, 2021 at 8:07 AM
Roland Richter <<a href="mailto:roland.richter@ntnu.no" target="_blank">roland.richter@ntnu.no</a>> wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hei,</p>
<p>at least in gdb
it fails with</p>
<p><span style="font-family:monospace"><span style="background-color:rgb(255,255,255)">Attempting to use an MPI
routine before
initializing
MPICH </span><br>
[Inferior 1
(process 7854)
exited with code
01] <br>
(gdb) backtrace
<br>
No stack.<br>
</span></p>
</div>
</blockquote>
<div><br>
</div>
<div>What were you
running? If it never
makes it into PETSc
code, I am not sure
what we are</div>
<div>doing to cause
this.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p><span style="font-family:monospace">
</span>Regards,</p>
<p>Roland<br>
<span style="font-family:monospace"></span></p>
<div>Am 11.10.21 um
13:57 schrieb
Matthew Knepley:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">On
Mon, Oct 11,
2021 at 5:24
AM Roland
Richter <<a href="mailto:roland.richter@ntnu.no" target="_blank">roland.richter@ntnu.no</a>> wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hei,<br>
<br>
I compiled
PETSc with
Intel MPI
(MPICH) and
GCC as
compiler (i.e.
using<br>
Intel OneAPI
together with
the supplied
mpicxx-compiler).
Compilation<br>
and
installation
worked fine,
but running
the tests
resulted in
the<br>
error
"Attempting to
use an MPI
routine before
initializing
MPICH". A<br>
simple test
program
(attached)
worked fine
with the same
combination.<br>
<br>
What could be
the reason for
that?<br>
</blockquote>
<div><br>
</div>
<div>Hi
Roland,</div>
<div><br>
</div>
<div>Can you
get a stack
trace for this
error using
the debugger?</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Thanks!<br>
<br>
Regards,<br>
<br>
Roland Richter<br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most
experimenters
take for
granted before
they begin
their
experiments is
infinitely
more
interesting
than any
results to
which their
experiments
lead.<br>
-- Norbert
Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most
experimenters
take for
granted before
they begin
their
experiments is
infinitely
more
interesting
than any
results to
which their
experiments
lead.<br>
-- Norbert
Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
<span id="gmail-m_-3594755976688188372gmail-m_5718787638988412959cid:3B71A0CF-7121-40E3-AC68-DB609803623C"><configure.log></span></div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before
they begin their experiments is infinitely more
interesting than any results to which their
experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>