[petsc-users] Error "Attempting to use an MPI routine before initializing MPICH" after compiling PETSc with Intel MPI and GCC
Roland Richter
roland.richter at ntnu.no
Wed Oct 13 04:53:24 CDT 2021
Hei,
I noticed a difference in when the program is running, and when not. The
code works fine if I compile it via a CMake-file and load PETSc there.
If I use the compilation line which is included in the Makefiles, then
the code will fail with the mentioned error. The cmake-generated
compilation line (including armadillo, because my test sample contained
armadillo-code) is
//opt/intel/oneapi/mpi/2021.4.0/bin/mpicxx -D__INSDIR__="" -I/include
-I/opt/petsc/include -I/opt/armadillo/include -std=c++0x -g -MD -MT
CMakeFiles/main.dir/source/main.cpp.o -MF
CMakeFiles/main.dir/source/main.cpp.o.d -o
CMakeFiles/main.dir/source/main.cpp.o -c source/main.cpp//
///opt/intel/oneapi/mpi/2021.4.0/bin/mpicxx -rdynamic
CMakeFiles/main.dir/source/main.cpp.o -o main_short
-Wl,-rpath,/opt/petsc/lib:/opt/armadillo/lib64
/opt/petsc/lib/libpetsc.so /opt/armadillo/lib64/libarmadillo.so /
Meanwhile, the original compilation line from PETSc is
/mpicxx -mavx2 -march=native -O3 -fPIC -fopenmp -I/opt/petsc/include
-I/opt/armadillo/include -I/opt/intel/oneapi/mkl/latest/include
-I/opt/fftw3/include -I/opt/hdf5/include -I/opt/boost/include
source/main.cpp -Wl,-rpath,/opt/petsc/lib -L/opt/petsc/lib
-Wl,-rpath,/opt/petsc/lib -L/opt/petsc/lib
-L/opt/intel/oneapi/mkl/latest/lib/intel64 -Wl,-rpath,/opt/fftw3/lib64
-L/opt/fftw3/lib64 -Wl,-rpath,/opt/armadillo/lib64
-L/opt/armadillo/lib64
-Wl,-rpath,/opt/intel/oneapi/mkl/latest/lib/intel64
-Wl,-rpath,/opt/hdf5/lib -L/opt/hdf5/lib
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.4.0/lib/release
-L/opt/intel/oneapi/mpi/2021.4.0/lib/release
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.4.0/lib
-L/opt/intel/oneapi/mpi/2021.4.0/lib
-Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/11
-L/usr/lib64/gcc/x86_64-suse-linux/11
-Wl,-rpath,/opt/intel/oneapi/vpl/2021.6.0/lib
-L/opt/intel/oneapi/vpl/2021.6.0/lib
-Wl,-rpath,/opt/intel/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8
-L/opt/intel/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8
-Wl,-rpath,/opt/intel/oneapi/mpi/2021.4.0/libfabric/lib
-L/opt/intel/oneapi/mpi/2021.4.0/libfabric/lib
-Wl,-rpath,/opt/intel/oneapi/mkl/2021.4.0/lib/intel64
-L/opt/intel/oneapi/mkl/2021.4.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/ipp/2021.4.0/lib/intel64
-L/opt/intel/oneapi/ipp/2021.4.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/ippcp/2021.4.0/lib/intel64
-L/opt/intel/oneapi/ippcp/2021.4.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/dnnl/2021.4.0/cpu_dpcpp_gpu_dpcpp/lib
-L/opt/intel/oneapi/dnnl/2021.4.0/cpu_dpcpp_gpu_dpcpp/lib
-Wl,-rpath,/opt/intel/oneapi/dal/2021.4.0/lib/intel64
-L/opt/intel/oneapi/dal/2021.4.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/compiler/2021.4.0/linux/compiler/lib/intel64_lin
-L/opt/intel/oneapi/compiler/2021.4.0/linux/compiler/lib/intel64_lin
-Wl,-rpath,/opt/intel/oneapi/compiler/2021.4.0/linux/lib
-L/opt/intel/oneapi/compiler/2021.4.0/linux/lib
-Wl,-rpath,/opt/intel/oneapi/clck/2021.4.0/lib/intel64
-L/opt/intel/oneapi/clck/2021.4.0/lib/intel64
-Wl,-rpath,/opt/intel/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp
-L/opt/intel/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp
-Wl,-rpath,/usr/x86_64-suse-linux/lib -L/usr/x86_64-suse-linux/lib
-larmadillo -lpetsc -lHYPRE -lcmumps -ldmumps -lsmumps -lzmumps
-lmumps_common -lpord -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64
-lspqr -lumfpack -lklu -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd
-lsuitesparseconfig -lsuperlu -lsuperlu_dist -lEl -lElSuiteSparse
-lpmrrr -lfftw3_mpi -lfftw3 -lp4est -lsc -lmkl_intel_lp64 -lmkl_core
-lmkl_intel_thread -liomp5 -ldl -lpthread -lptesmumps -lptscotchparmetis
-lptscotch -lptscotcherr -lesmumps -lscotch -lscotcherr -lhdf5_hl -lhdf5
-lparmetis -lmetis -lm -lz -lmuparser -lX11 -lstdc++ -ldl -lmpifort
-lmpi -lrt -lpthread -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath
-lrt -lquadmath -lstdc++ -ldl -o main_long/
Both executables have the same libraries linked to them, but in a
different order when comparing them with ldd.
Does that explain the observed behavior?
Thanks,
regards,
Roland
Am 11.10.21 um 15:13 schrieb Roland Richter:
>
> Hei,
>
> the following code works fine:
>
> #include <iostream>
> #include <petsc.h>
>
> static char help[] = "Solves 2D Poisson equation using multigrid.\n\n";
> int main(int argc,char **argv) {
> PetscInitialize(&argc,&argv,(char*)0,help);
> std::cout << "Hello World\n";
> PetscFinalize();
> return 0;
> }
>
> Regards,
>
> Roland
>
> Am 11.10.21 um 14:34 schrieb Stefano Zampini:
>> Can you try with a simple call that only calls PetscInitialize/Finalize?
>>
>>
>>> On Oct 11, 2021, at 3:30 PM, Roland Richter <roland.richter at ntnu.no>
>>> wrote:
>>>
>>> At least according to configure.log mpiexec was defined as
>>>
>>> Checking for program /opt/intel/oneapi/mpi/2021.4.0//bin/mpiexec...found
>>> Defined make macro "MPIEXECEXECUTABLE" to
>>> "/opt/intel/oneapi/mpi/2021.4.0/bin/mpiexec"
>>>
>>> When running ex19 with this mpiexec it fails with the usual error,
>>> even though all configuration steps worked fine. I attached the
>>> configuration log.
>>>
>>> Regards,
>>>
>>> Roland
>>>
>>> Am 11.10.21 um 14:24 schrieb Stefano Zampini:
>>>> You are most probably using a different mpiexec then the one used
>>>> to compile petsc.
>>>>
>>>>
>>>>
>>>>> On Oct 11, 2021, at 3:23 PM, Roland Richter
>>>>> <roland.richter at ntnu.no> wrote:
>>>>>
>>>>> I tried either /./ex19/ (SNES-example), /mpirun ./ex19/ or /mpirun
>>>>> -n 1 ./ex19/, all with the same result.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Roland
>>>>>
>>>>> Am 11.10.21 um 14:22 schrieb Matthew Knepley:
>>>>>> On Mon, Oct 11, 2021 at 8:07 AM Roland Richter
>>>>>> <roland.richter at ntnu.no> wrote:
>>>>>>
>>>>>> Hei,
>>>>>>
>>>>>> at least in gdb it fails with
>>>>>>
>>>>>> Attempting to use an MPI routine before initializing MPICH
>>>>>> [Inferior 1 (process 7854) exited with code 01]
>>>>>> (gdb) backtrace
>>>>>> No stack.
>>>>>>
>>>>>>
>>>>>> What were you running? If it never makes it into PETSc code, I am
>>>>>> not sure what we are
>>>>>> doing to cause this.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Matt
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Roland
>>>>>>
>>>>>> Am 11.10.21 um 13:57 schrieb Matthew Knepley:
>>>>>>> On Mon, Oct 11, 2021 at 5:24 AM Roland Richter
>>>>>>> <roland.richter at ntnu.no> wrote:
>>>>>>>
>>>>>>> Hei,
>>>>>>>
>>>>>>> I compiled PETSc with Intel MPI (MPICH) and GCC as
>>>>>>> compiler (i.e. using
>>>>>>> Intel OneAPI together with the supplied
>>>>>>> mpicxx-compiler). Compilation
>>>>>>> and installation worked fine, but running the tests
>>>>>>> resulted in the
>>>>>>> error "Attempting to use an MPI routine before
>>>>>>> initializing MPICH". A
>>>>>>> simple test program (attached) worked fine with the same
>>>>>>> combination.
>>>>>>>
>>>>>>> What could be the reason for that?
>>>>>>>
>>>>>>>
>>>>>>> Hi Roland,
>>>>>>>
>>>>>>> Can you get a stack trace for this error using the debugger?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Matt
>>>>>>>
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Roland Richter
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin
>>>>>>> their experiments is infinitely more interesting than any
>>>>>>> results to which their experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to
>>>>>> which their experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>> <configure.log>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20211013/a5a9d168/attachment.html>
More information about the petsc-users
mailing list