<div dir="ltr"><div dir="ltr">On Wed, Oct 13, 2021 at 6:43 AM Roland Richter <<a href="mailto:roland.richter@ntnu.no">roland.richter@ntnu.no</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Shouldn't I see a mixture of MPI-libraries when calling ldd if I
mix versions of MPI? <br>
</p>
<p></p></div></blockquote><div>Not if the compilation uses one and the link another. </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><p>I also removed all calls to armadillo, and accordingly all
references to it in the compilation, but the behavior is still
unchanged.</p></div></blockquote><div>I do not understand what you mean here. If you remove the armadiillo library, how are you linking MPI?</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<p>Regards,</p>
<p>Roland<br>
</p>
<div>Am 13.10.21 um 12:36 schrieb Matthew
Knepley:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">On Wed, Oct 13, 2021 at 6:32 AM Roland Richter
<<a href="mailto:roland.richter@ntnu.no" target="_blank">roland.richter@ntnu.no</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Yes, the first part (which works) consists out of a
compilation line and a linking line, while the second
command is a combination of compile- and linking line.<br>
</p>
</div>
</blockquote>
<div>The link line in the first does not tell us anything
because MPI is not even present. It is being pulled in I
presume from libarmadillo, which we cannot see. It still</div>
<div>seems most likely, as Stefano said, that you are mixing
versions of MPI.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Am 13.10.21 um 12:26 schrieb Matthew Knepley:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">On Wed, Oct 13, 2021 at 5:53 AM Roland
Richter <<a href="mailto:roland.richter@ntnu.no" target="_blank">roland.richter@ntnu.no</a>>
wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hei,</p>
<p>I noticed a difference in when the program is
running, and when not. The code works fine if
I compile it via a CMake-file and load PETSc
there. If I use the compilation line which is
included in the Makefiles, then the code will
fail with the mentioned error. The
cmake-generated compilation line (including
armadillo, because my test sample contained
armadillo-code) is<br>
</p>
</div>
</blockquote>
<div>One of these is a compile command and the other
is a link command.</div>
<div><br>
</div>
<div> Matt</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p><i>/opt/intel/oneapi/mpi/2021.4.0/bin/mpicxx
-D__INSDIR__="" -I/include
-I/opt/petsc/include
-I/opt/armadillo/include -std=c++0x -g -MD
-MT CMakeFiles/main.dir/source/main.cpp.o
-MF CMakeFiles/main.dir/source/main.cpp.o.d
-o CMakeFiles/main.dir/source/main.cpp.o -c
source/main.cpp</i><i><br>
</i><i>/opt/intel/oneapi/mpi/2021.4.0/bin/mpicxx
-rdynamic
CMakeFiles/main.dir/source/main.cpp.o -o
main_short
-Wl,-rpath,/opt/petsc/lib:/opt/armadillo/lib64
/opt/petsc/lib/libpetsc.so
/opt/armadillo/lib64/libarmadillo.so </i><br>
</p>
<p>Meanwhile, the original compilation line from
PETSc is</p>
<p><i>mpicxx -mavx2 -march=native -O3 -fPIC
-fopenmp -I/opt/petsc/include
-I/opt/armadillo/include
-I/opt/intel/oneapi/mkl/latest/include
-I/opt/fftw3/include -I/opt/hdf5/include
-I/opt/boost/include source/main.cpp
-Wl,-rpath,/opt/petsc/lib -L/opt/petsc/lib
-Wl,-rpath,/opt/petsc/lib -L/opt/petsc/lib
-L/opt/intel/oneapi/mkl/latest/lib/intel64
-Wl,-rpath,/opt/fftw3/lib64
-L/opt/fftw3/lib64
-Wl,-rpath,/opt/armadillo/lib64
-L/opt/armadillo/lib64
-Wl,-rpath,/opt/intel/oneapi/mkl/latest/lib/intel64
-Wl,-rpath,/opt/hdf5/lib -L/opt/hdf5/lib
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.4.0/lib/release
-L/opt/intel/oneapi/mpi/2021.4.0/lib/release
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.4.0/lib
-L/opt/intel/oneapi/mpi/2021.4.0/lib
-Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/11
-L/usr/lib64/gcc/x86_64-suse-linux/11
-Wl,-rpath,/opt/intel/oneapi/vpl/2021.6.0/lib
-L/opt/intel/oneapi/vpl/2021.6.0/lib
-Wl,-rpath,/opt/intel/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8
-L/opt/intel/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.4.0/libfabric/lib
-L/opt/intel/oneapi/mpi/2021.4.0/libfabric/lib
-Wl,-rpath,/opt/intel/oneapi/mkl/2021.4.0/lib/intel64
-L/opt/intel/oneapi/mkl/2021.4.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/ipp/2021.4.0/lib/intel64
-L/opt/intel/oneapi/ipp/2021.4.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/ippcp/2021.4.0/lib/intel64
-L/opt/intel/oneapi/ippcp/2021.4.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/dnnl/2021.4.0/cpu_dpcpp_gpu_dpcpp/lib
-L/opt/intel/oneapi/dnnl/2021.4.0/cpu_dpcpp_gpu_dpcpp/lib
-Wl,-rpath,/opt/intel/oneapi/dal/2021.4.0/lib/intel64
-L/opt/intel/oneapi/dal/2021.4.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/compiler/2021.4.0/linux/compiler/lib/intel64_lin
-L/opt/intel/oneapi/compiler/2021.4.0/linux/compiler/lib/intel64_lin
-Wl,-rpath,/opt/intel/oneapi/compiler/2021.4.0/linux/lib
-L/opt/intel/oneapi/compiler/2021.4.0/linux/lib
-Wl,-rpath,/opt/intel/oneapi/clck/2021.4.0/lib/intel64
-L/opt/intel/oneapi/clck/2021.4.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp
-L/opt/intel/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp
-Wl,-rpath,/usr/x86_64-suse-linux/lib
-L/usr/x86_64-suse-linux/lib -larmadillo
-lpetsc -lHYPRE -lcmumps -ldmumps -lsmumps
-lzmumps -lmumps_common -lpord
-lmkl_scalapack_lp64
-lmkl_blacs_intelmpi_lp64 -lspqr -lumfpack
-lklu -lcholmod -lbtf -lccolamd -lcolamd
-lcamd -lamd -lsuitesparseconfig -lsuperlu
-lsuperlu_dist -lEl -lElSuiteSparse -lpmrrr
-lfftw3_mpi -lfftw3 -lp4est -lsc
-lmkl_intel_lp64 -lmkl_core
-lmkl_intel_thread -liomp5 -ldl -lpthread
-lptesmumps -lptscotchparmetis -lptscotch
-lptscotcherr -lesmumps -lscotch -lscotcherr
-lhdf5_hl -lhdf5 -lparmetis -lmetis -lm -lz
-lmuparser -lX11 -lstdc++ -ldl -lmpifort
-lmpi -lrt -lpthread -lgfortran -lm
-lgfortran -lm -lgcc_s -lquadmath -lrt
-lquadmath -lstdc++ -ldl -o main_long</i></p>
<p>Both executables have the same libraries
linked to them, but in a different order when
comparing them with ldd.<br>
</p>
<p>Does that explain the observed behavior?</p>
<p>Thanks,</p>
<p>regards,</p>
<p>Roland<br>
</p>
<div>Am 11.10.21 um 15:13 schrieb Roland
Richter:<br>
</div>
<blockquote type="cite">
<p>Hei,</p>
<p>the following code works fine:</p>
<p>#include <iostream><br>
#include <petsc.h><br>
<br>
static char help[] = "Solves 2D Poisson
equation using multigrid.\n\n";<br>
int main(int argc,char **argv) {<br>
PetscInitialize(&argc,&argv,(char*)0,help);<br>
std::cout << "Hello World\n";<br>
PetscFinalize();<br>
return 0;<br>
}</p>
<p>Regards,</p>
<p>Roland<br>
</p>
<div>Am 11.10.21 um 14:34 schrieb Stefano
Zampini:<br>
</div>
<blockquote type="cite"> Can you try with a
simple call that only calls
PetscInitialize/Finalize?
<div><br>
<div>
<div><br>
<blockquote type="cite">
<div>On Oct 11, 2021, at 3:30 PM,
Roland Richter <<a href="mailto:roland.richter@ntnu.no" target="_blank">roland.richter@ntnu.no</a>>
wrote:</div>
<br>
<div>
<div>
<p>At least according to
configure.log mpiexec was
defined as</p>
<p>Checking for program
/opt/intel/oneapi/mpi/2021.4.0//bin/mpiexec...found<br>
Defined make
macro "MPIEXECEXECUTABLE" to
"/opt/intel/oneapi/mpi/2021.4.0/bin/mpiexec"</p>
<p>When running ex19 with this
mpiexec it fails with the
usual error, even though all
configuration steps worked
fine. I attached the
configuration log.<br>
</p>
<p>Regards,</p>
<p>Roland<br>
</p>
<div>Am 11.10.21 um 14:24
schrieb Stefano Zampini:<br>
</div>
<blockquote type="cite"> You are
most probably using a
different mpiexec then the one
used to compile petsc.
<div><br>
</div>
<div><br>
<div><br>
<blockquote type="cite">
<div>On Oct 11, 2021, at
3:23 PM, Roland
Richter <<a href="mailto:roland.richter@ntnu.no" target="_blank">roland.richter@ntnu.no</a>> wrote:</div>
<br>
<div>
<div>
<p>I tried either <i>./ex19</i>
(SNES-example), <i>mpirun
./ex19</i> or <i>mpirun
-n 1 ./ex19</i>,
all with the same
result.</p>
<p>Regards,</p>
<p>Roland<br>
</p>
<div>Am 11.10.21 um
14:22 schrieb
Matthew Knepley:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">On
Mon, Oct 11,
2021 at 8:07
AM Roland
Richter <<a href="mailto:roland.richter@ntnu.no" target="_blank">roland.richter@ntnu.no</a>> wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hei,</p>
<p>at least in
gdb it fails
with</p>
<p><span style="font-family:monospace"><span style="background-color:rgb(255,255,255)">Attempting to use an MPI
routine before
initializing
MPICH </span><br>
[Inferior 1
(process 7854)
exited with
code 01] <br>
(gdb)
backtrace <br>
No stack.<br>
</span></p>
</div>
</blockquote>
<div><br>
</div>
<div>What were
you running?
If it never
makes it into
PETSc code, I
am not sure
what we are</div>
<div>doing to
cause this.</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p><span style="font-family:monospace">
</span>Regards,</p>
<p>Roland<br>
<span style="font-family:monospace"></span></p>
<div>Am
11.10.21 um
13:57 schrieb
Matthew
Knepley:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">On
Mon, Oct 11,
2021 at 5:24
AM Roland
Richter <<a href="mailto:roland.richter@ntnu.no" target="_blank">roland.richter@ntnu.no</a>> wrote:<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hei,<br>
<br>
I compiled
PETSc with
Intel MPI
(MPICH) and
GCC as
compiler (i.e.
using<br>
Intel OneAPI
together with
the supplied
mpicxx-compiler).
Compilation<br>
and
installation
worked fine,
but running
the tests
resulted in
the<br>
error
"Attempting to
use an MPI
routine before
initializing
MPICH". A<br>
simple test
program
(attached)
worked fine
with the same
combination.<br>
<br>
What could be
the reason for
that?<br>
</blockquote>
<div><br>
</div>
<div>Hi
Roland,</div>
<div><br>
</div>
<div>Can you
get a stack
trace for this
error using
the debugger?</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Thanks!<br>
<br>
Regards,<br>
<br>
Roland Richter<br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most
experimenters
take for
granted before
they begin
their
experiments is
infinitely
more
interesting
than any
results to
which their
experiments
lead.<br>
-- Norbert
Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most
experimenters
take for
granted before
they begin
their
experiments is
infinitely
more
interesting
than any
results to
which their
experiments
lead.<br>
-- Norbert
Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
<span id="gmail-m_6803292070734712534gmail-m_-3594755976688188372gmail-m_5718787638988412959cid:3B71A0CF-7121-40E3-AC68-DB609803623C"><configure.log></span></div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for
granted before they begin their
experiments is infinitely more
interesting than any results to which
their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before
they begin their experiments is infinitely more
interesting than any results to which their
experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>