[petsc-users] Error "Attempting to use an MPI routine before initializing MPICH" after compiling PETSc with Intel MPI and GCC
Roland Richter
roland.richter at ntnu.no
Wed Oct 13 05:32:44 CDT 2021
Yes, the first part (which works) consists out of a compilation line and
a linking line, while the second command is a combination of compile-
and linking line.
Am 13.10.21 um 12:26 schrieb Matthew Knepley:
> On Wed, Oct 13, 2021 at 5:53 AM Roland Richter
> <roland.richter at ntnu.no> wrote:
>
> Hei,
>
> I noticed a difference in when the program is running, and when
> not. The code works fine if I compile it via a CMake-file and load
> PETSc there. If I use the compilation line which is included in
> the Makefiles, then the code will fail with the mentioned error.
> The cmake-generated compilation line (including armadillo, because
> my test sample contained armadillo-code) is
>
> One of these is a compile command and the other is a link command.
>
> Matt
>
> //opt/intel/oneapi/mpi/2021.4.0/bin/mpicxx -D__INSDIR__=""
> -I/include -I/opt/petsc/include -I/opt/armadillo/include
> -std=c++0x -g -MD -MT CMakeFiles/main.dir/source/main.cpp.o -MF
> CMakeFiles/main.dir/source/main.cpp.o.d -o
> CMakeFiles/main.dir/source/main.cpp.o -c source/main.cpp//
> ///opt/intel/oneapi/mpi/2021.4.0/bin/mpicxx -rdynamic
> CMakeFiles/main.dir/source/main.cpp.o -o main_short
> -Wl,-rpath,/opt/petsc/lib:/opt/armadillo/lib64
> /opt/petsc/lib/libpetsc.so /opt/armadillo/lib64/libarmadillo.so /
>
> Meanwhile, the original compilation line from PETSc is
>
> /mpicxx -mavx2 -march=native -O3 -fPIC -fopenmp
> -I/opt/petsc/include -I/opt/armadillo/include
> -I/opt/intel/oneapi/mkl/latest/include -I/opt/fftw3/include
> -I/opt/hdf5/include -I/opt/boost/include source/main.cpp
> -Wl,-rpath,/opt/petsc/lib -L/opt/petsc/lib
> -Wl,-rpath,/opt/petsc/lib -L/opt/petsc/lib
> -L/opt/intel/oneapi/mkl/latest/lib/intel64
> -Wl,-rpath,/opt/fftw3/lib64 -L/opt/fftw3/lib64
> -Wl,-rpath,/opt/armadillo/lib64 -L/opt/armadillo/lib64
> -Wl,-rpath,/opt/intel/oneapi/mkl/latest/lib/intel64
> -Wl,-rpath,/opt/hdf5/lib -L/opt/hdf5/lib
> -Wl,-rpath,/opt/intel/oneapi/mpi/2021.4.0/lib/release
> -L/opt/intel/oneapi/mpi/2021.4.0/lib/release
> -Wl,-rpath,/opt/intel/oneapi/mpi/2021.4.0/lib
> -L/opt/intel/oneapi/mpi/2021.4.0/lib
> -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/11
> -L/usr/lib64/gcc/x86_64-suse-linux/11
> -Wl,-rpath,/opt/intel/oneapi/vpl/2021.6.0/lib
> -L/opt/intel/oneapi/vpl/2021.6.0/lib
> -Wl,-rpath,/opt/intel/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8
> -L/opt/intel/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8
> -Wl,-rpath,/opt/intel/oneapi/mpi/2021.4.0/libfabric/lib
> -L/opt/intel/oneapi/mpi/2021.4.0/libfabric/lib
> -Wl,-rpath,/opt/intel/oneapi/mkl/2021.4.0/lib/intel64
> -L/opt/intel/oneapi/mkl/2021.4.0/lib/intel64
> -Wl,-rpath,/opt/intel/oneapi/ipp/2021.4.0/lib/intel64
> -L/opt/intel/oneapi/ipp/2021.4.0/lib/intel64
> -Wl,-rpath,/opt/intel/oneapi/ippcp/2021.4.0/lib/intel64
> -L/opt/intel/oneapi/ippcp/2021.4.0/lib/intel64
> -Wl,-rpath,/opt/intel/oneapi/dnnl/2021.4.0/cpu_dpcpp_gpu_dpcpp/lib
> -L/opt/intel/oneapi/dnnl/2021.4.0/cpu_dpcpp_gpu_dpcpp/lib
> -Wl,-rpath,/opt/intel/oneapi/dal/2021.4.0/lib/intel64
> -L/opt/intel/oneapi/dal/2021.4.0/lib/intel64
> -Wl,-rpath,/opt/intel/oneapi/compiler/2021.4.0/linux/compiler/lib/intel64_lin
> -L/opt/intel/oneapi/compiler/2021.4.0/linux/compiler/lib/intel64_lin
> -Wl,-rpath,/opt/intel/oneapi/compiler/2021.4.0/linux/lib
> -L/opt/intel/oneapi/compiler/2021.4.0/linux/lib
> -Wl,-rpath,/opt/intel/oneapi/clck/2021.4.0/lib/intel64
> -L/opt/intel/oneapi/clck/2021.4.0/lib/intel64
> -Wl,-rpath,/opt/intel/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp
> -L/opt/intel/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp
> -Wl,-rpath,/usr/x86_64-suse-linux/lib -L/usr/x86_64-suse-linux/lib
> -larmadillo -lpetsc -lHYPRE -lcmumps -ldmumps -lsmumps -lzmumps
> -lmumps_common -lpord -lmkl_scalapack_lp64
> -lmkl_blacs_intelmpi_lp64 -lspqr -lumfpack -lklu -lcholmod -lbtf
> -lccolamd -lcolamd -lcamd -lamd -lsuitesparseconfig -lsuperlu
> -lsuperlu_dist -lEl -lElSuiteSparse -lpmrrr -lfftw3_mpi -lfftw3
> -lp4est -lsc -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread
> -liomp5 -ldl -lpthread -lptesmumps -lptscotchparmetis -lptscotch
> -lptscotcherr -lesmumps -lscotch -lscotcherr -lhdf5_hl -lhdf5
> -lparmetis -lmetis -lm -lz -lmuparser -lX11 -lstdc++ -ldl
> -lmpifort -lmpi -lrt -lpthread -lgfortran -lm -lgfortran -lm
> -lgcc_s -lquadmath -lrt -lquadmath -lstdc++ -ldl -o main_long/
>
> Both executables have the same libraries linked to them, but in a
> different order when comparing them with ldd.
>
> Does that explain the observed behavior?
>
> Thanks,
>
> regards,
>
> Roland
>
> Am 11.10.21 um 15:13 schrieb Roland Richter:
>>
>> Hei,
>>
>> the following code works fine:
>>
>> #include <iostream>
>> #include <petsc.h>
>>
>> static char help[] = "Solves 2D Poisson equation using
>> multigrid.\n\n";
>> int main(int argc,char **argv) {
>> PetscInitialize(&argc,&argv,(char*)0,help);
>> std::cout << "Hello World\n";
>> PetscFinalize();
>> return 0;
>> }
>>
>> Regards,
>>
>> Roland
>>
>> Am 11.10.21 um 14:34 schrieb Stefano Zampini:
>>> Can you try with a simple call that only calls
>>> PetscInitialize/Finalize?
>>>
>>>
>>>> On Oct 11, 2021, at 3:30 PM, Roland Richter
>>>> <roland.richter at ntnu.no> wrote:
>>>>
>>>> At least according to configure.log mpiexec was defined as
>>>>
>>>> Checking for program
>>>> /opt/intel/oneapi/mpi/2021.4.0//bin/mpiexec...found
>>>> Defined make macro "MPIEXECEXECUTABLE" to
>>>> "/opt/intel/oneapi/mpi/2021.4.0/bin/mpiexec"
>>>>
>>>> When running ex19 with this mpiexec it fails with the usual
>>>> error, even though all configuration steps worked fine. I
>>>> attached the configuration log.
>>>>
>>>> Regards,
>>>>
>>>> Roland
>>>>
>>>> Am 11.10.21 um 14:24 schrieb Stefano Zampini:
>>>>> You are most probably using a different mpiexec then the one
>>>>> used to compile petsc.
>>>>>
>>>>>
>>>>>
>>>>>> On Oct 11, 2021, at 3:23 PM, Roland Richter
>>>>>> <roland.richter at ntnu.no> wrote:
>>>>>>
>>>>>> I tried either /./ex19/ (SNES-example), /mpirun ./ex19/ or
>>>>>> /mpirun -n 1 ./ex19/, all with the same result.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Roland
>>>>>>
>>>>>> Am 11.10.21 um 14:22 schrieb Matthew Knepley:
>>>>>>> On Mon, Oct 11, 2021 at 8:07 AM Roland Richter
>>>>>>> <roland.richter at ntnu.no> wrote:
>>>>>>>
>>>>>>> Hei,
>>>>>>>
>>>>>>> at least in gdb it fails with
>>>>>>>
>>>>>>> Attempting to use an MPI routine before initializing MPICH
>>>>>>> [Inferior 1 (process 7854) exited with code 01]
>>>>>>> (gdb) backtrace
>>>>>>> No stack.
>>>>>>>
>>>>>>>
>>>>>>> What were you running? If it never makes it into PETSc code,
>>>>>>> I am not sure what we are
>>>>>>> doing to cause this.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Matt
>>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Roland
>>>>>>>
>>>>>>> Am 11.10.21 um 13:57 schrieb Matthew Knepley:
>>>>>>>> On Mon, Oct 11, 2021 at 5:24 AM Roland Richter
>>>>>>>> <roland.richter at ntnu.no> wrote:
>>>>>>>>
>>>>>>>> Hei,
>>>>>>>>
>>>>>>>> I compiled PETSc with Intel MPI (MPICH) and GCC as
>>>>>>>> compiler (i.e. using
>>>>>>>> Intel OneAPI together with the supplied
>>>>>>>> mpicxx-compiler). Compilation
>>>>>>>> and installation worked fine, but running the tests
>>>>>>>> resulted in the
>>>>>>>> error "Attempting to use an MPI routine before
>>>>>>>> initializing MPICH". A
>>>>>>>> simple test program (attached) worked fine with the
>>>>>>>> same combination.
>>>>>>>>
>>>>>>>> What could be the reason for that?
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi Roland,
>>>>>>>>
>>>>>>>> Can you get a stack trace for this error using the
>>>>>>>> debugger?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Matt
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Roland Richter
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> What most experimenters take for granted before they
>>>>>>>> begin their experiments is infinitely more interesting
>>>>>>>> than any results to which their experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>>>
>>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin
>>>>>>> their experiments is infinitely more interesting than any
>>>>>>> results to which their experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>
>>>> <configure.log>
>>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which
> their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20211013/6485438a/attachment-0001.html>
More information about the petsc-users
mailing list