[petsc-users] Error "Attempting to use an MPI routine before initializing MPICH" after compiling PETSc with Intel MPI and GCC
Roland Richter
roland.richter at ntnu.no
Wed Oct 13 05:43:48 CDT 2021
Shouldn't I see a mixture of MPI-libraries when calling ldd if I mix
versions of MPI?
I also removed all calls to armadillo, and accordingly all references to
it in the compilation, but the behavior is still unchanged.
Regards,
Roland
Am 13.10.21 um 12:36 schrieb Matthew Knepley:
> On Wed, Oct 13, 2021 at 6:32 AM Roland Richter
> <roland.richter at ntnu.no> wrote:
>
> Yes, the first part (which works) consists out of a compilation
> line and a linking line, while the second command is a combination
> of compile- and linking line.
>
> The link line in the first does not tell us anything because MPI is
> not even present. It is being pulled in I presume from libarmadillo,
> which we cannot see. It still
> seems most likely, as Stefano said, that you are mixing versions of MPI.
>
> Thanks,
>
> Matt
>
>
> Am 13.10.21 um 12:26 schrieb Matthew Knepley:
>> On Wed, Oct 13, 2021 at 5:53 AM Roland Richter
>> <roland.richter at ntnu.no> wrote:
>>
>> Hei,
>>
>> I noticed a difference in when the program is running, and
>> when not. The code works fine if I compile it via a
>> CMake-file and load PETSc there. If I use the compilation
>> line which is included in the Makefiles, then the code will
>> fail with the mentioned error. The cmake-generated
>> compilation line (including armadillo, because my test sample
>> contained armadillo-code) is
>>
>> One of these is a compile command and the other is a link command.
>>
>> Matt
>>
>> //opt/intel/oneapi/mpi/2021.4.0/bin/mpicxx -D__INSDIR__=""
>> -I/include -I/opt/petsc/include -I/opt/armadillo/include
>> -std=c++0x -g -MD -MT CMakeFiles/main.dir/source/main.cpp.o
>> -MF CMakeFiles/main.dir/source/main.cpp.o.d -o
>> CMakeFiles/main.dir/source/main.cpp.o -c source/main.cpp//
>> ///opt/intel/oneapi/mpi/2021.4.0/bin/mpicxx -rdynamic
>> CMakeFiles/main.dir/source/main.cpp.o -o main_short
>> -Wl,-rpath,/opt/petsc/lib:/opt/armadillo/lib64
>> /opt/petsc/lib/libpetsc.so /opt/armadillo/lib64/libarmadillo.so /
>>
>> Meanwhile, the original compilation line from PETSc is
>>
>> /mpicxx -mavx2 -march=native -O3 -fPIC -fopenmp
>> -I/opt/petsc/include -I/opt/armadillo/include
>> -I/opt/intel/oneapi/mkl/latest/include -I/opt/fftw3/include
>> -I/opt/hdf5/include -I/opt/boost/include source/main.cpp
>> -Wl,-rpath,/opt/petsc/lib -L/opt/petsc/lib
>> -Wl,-rpath,/opt/petsc/lib -L/opt/petsc/lib
>> -L/opt/intel/oneapi/mkl/latest/lib/intel64
>> -Wl,-rpath,/opt/fftw3/lib64 -L/opt/fftw3/lib64
>> -Wl,-rpath,/opt/armadillo/lib64 -L/opt/armadillo/lib64
>> -Wl,-rpath,/opt/intel/oneapi/mkl/latest/lib/intel64
>> -Wl,-rpath,/opt/hdf5/lib -L/opt/hdf5/lib
>> -Wl,-rpath,/opt/intel/oneapi/mpi/2021.4.0/lib/release
>> -L/opt/intel/oneapi/mpi/2021.4.0/lib/release
>> -Wl,-rpath,/opt/intel/oneapi/mpi/2021.4.0/lib
>> -L/opt/intel/oneapi/mpi/2021.4.0/lib
>> -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/11
>> -L/usr/lib64/gcc/x86_64-suse-linux/11
>> -Wl,-rpath,/opt/intel/oneapi/vpl/2021.6.0/lib
>> -L/opt/intel/oneapi/vpl/2021.6.0/lib
>> -Wl,-rpath,/opt/intel/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8
>> -L/opt/intel/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8
>> -Wl,-rpath,/opt/intel/oneapi/mpi/2021.4.0/libfabric/lib
>> -L/opt/intel/oneapi/mpi/2021.4.0/libfabric/lib
>> -Wl,-rpath,/opt/intel/oneapi/mkl/2021.4.0/lib/intel64
>> -L/opt/intel/oneapi/mkl/2021.4.0/lib/intel64
>> -Wl,-rpath,/opt/intel/oneapi/ipp/2021.4.0/lib/intel64
>> -L/opt/intel/oneapi/ipp/2021.4.0/lib/intel64
>> -Wl,-rpath,/opt/intel/oneapi/ippcp/2021.4.0/lib/intel64
>> -L/opt/intel/oneapi/ippcp/2021.4.0/lib/intel64
>> -Wl,-rpath,/opt/intel/oneapi/dnnl/2021.4.0/cpu_dpcpp_gpu_dpcpp/lib
>> -L/opt/intel/oneapi/dnnl/2021.4.0/cpu_dpcpp_gpu_dpcpp/lib
>> -Wl,-rpath,/opt/intel/oneapi/dal/2021.4.0/lib/intel64
>> -L/opt/intel/oneapi/dal/2021.4.0/lib/intel64
>> -Wl,-rpath,/opt/intel/oneapi/compiler/2021.4.0/linux/compiler/lib/intel64_lin
>> -L/opt/intel/oneapi/compiler/2021.4.0/linux/compiler/lib/intel64_lin
>> -Wl,-rpath,/opt/intel/oneapi/compiler/2021.4.0/linux/lib
>> -L/opt/intel/oneapi/compiler/2021.4.0/linux/lib
>> -Wl,-rpath,/opt/intel/oneapi/clck/2021.4.0/lib/intel64
>> -L/opt/intel/oneapi/clck/2021.4.0/lib/intel64
>> -Wl,-rpath,/opt/intel/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp
>> -L/opt/intel/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp
>> -Wl,-rpath,/usr/x86_64-suse-linux/lib
>> -L/usr/x86_64-suse-linux/lib -larmadillo -lpetsc -lHYPRE
>> -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord
>> -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -lspqr
>> -lumfpack -lklu -lcholmod -lbtf -lccolamd -lcolamd -lcamd
>> -lamd -lsuitesparseconfig -lsuperlu -lsuperlu_dist -lEl
>> -lElSuiteSparse -lpmrrr -lfftw3_mpi -lfftw3 -lp4est -lsc
>> -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -liomp5 -ldl
>> -lpthread -lptesmumps -lptscotchparmetis -lptscotch
>> -lptscotcherr -lesmumps -lscotch -lscotcherr -lhdf5_hl -lhdf5
>> -lparmetis -lmetis -lm -lz -lmuparser -lX11 -lstdc++ -ldl
>> -lmpifort -lmpi -lrt -lpthread -lgfortran -lm -lgfortran -lm
>> -lgcc_s -lquadmath -lrt -lquadmath -lstdc++ -ldl -o main_long/
>>
>> Both executables have the same libraries linked to them, but
>> in a different order when comparing them with ldd.
>>
>> Does that explain the observed behavior?
>>
>> Thanks,
>>
>> regards,
>>
>> Roland
>>
>> Am 11.10.21 um 15:13 schrieb Roland Richter:
>>>
>>> Hei,
>>>
>>> the following code works fine:
>>>
>>> #include <iostream>
>>> #include <petsc.h>
>>>
>>> static char help[] = "Solves 2D Poisson equation using
>>> multigrid.\n\n";
>>> int main(int argc,char **argv) {
>>> PetscInitialize(&argc,&argv,(char*)0,help);
>>> std::cout << "Hello World\n";
>>> PetscFinalize();
>>> return 0;
>>> }
>>>
>>> Regards,
>>>
>>> Roland
>>>
>>> Am 11.10.21 um 14:34 schrieb Stefano Zampini:
>>>> Can you try with a simple call that only calls
>>>> PetscInitialize/Finalize?
>>>>
>>>>
>>>>> On Oct 11, 2021, at 3:30 PM, Roland Richter
>>>>> <roland.richter at ntnu.no> wrote:
>>>>>
>>>>> At least according to configure.log mpiexec was defined as
>>>>>
>>>>> Checking for program
>>>>> /opt/intel/oneapi/mpi/2021.4.0//bin/mpiexec...found
>>>>> Defined make macro "MPIEXECEXECUTABLE"
>>>>> to "/opt/intel/oneapi/mpi/2021.4.0/bin/mpiexec"
>>>>>
>>>>> When running ex19 with this mpiexec it fails with the
>>>>> usual error, even though all configuration steps worked
>>>>> fine. I attached the configuration log.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Roland
>>>>>
>>>>> Am 11.10.21 um 14:24 schrieb Stefano Zampini:
>>>>>> You are most probably using a different mpiexec then the
>>>>>> one used to compile petsc.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On Oct 11, 2021, at 3:23 PM, Roland Richter
>>>>>>> <roland.richter at ntnu.no> wrote:
>>>>>>>
>>>>>>> I tried either /./ex19/ (SNES-example), /mpirun ./ex19/
>>>>>>> or /mpirun -n 1 ./ex19/, all with the same result.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Roland
>>>>>>>
>>>>>>> Am 11.10.21 um 14:22 schrieb Matthew Knepley:
>>>>>>>> On Mon, Oct 11, 2021 at 8:07 AM Roland Richter
>>>>>>>> <roland.richter at ntnu.no> wrote:
>>>>>>>>
>>>>>>>> Hei,
>>>>>>>>
>>>>>>>> at least in gdb it fails with
>>>>>>>>
>>>>>>>> Attempting to use an MPI routine before
>>>>>>>> initializing MPICH
>>>>>>>> [Inferior 1 (process 7854) exited with code 01]
>>>>>>>> (gdb) backtrace
>>>>>>>> No stack.
>>>>>>>>
>>>>>>>>
>>>>>>>> What were you running? If it never makes it into PETSc
>>>>>>>> code, I am not sure what we are
>>>>>>>> doing to cause this.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Matt
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Roland
>>>>>>>>
>>>>>>>> Am 11.10.21 um 13:57 schrieb Matthew Knepley:
>>>>>>>>> On Mon, Oct 11, 2021 at 5:24 AM Roland Richter
>>>>>>>>> <roland.richter at ntnu.no> wrote:
>>>>>>>>>
>>>>>>>>> Hei,
>>>>>>>>>
>>>>>>>>> I compiled PETSc with Intel MPI (MPICH) and
>>>>>>>>> GCC as compiler (i.e. using
>>>>>>>>> Intel OneAPI together with the supplied
>>>>>>>>> mpicxx-compiler). Compilation
>>>>>>>>> and installation worked fine, but running the
>>>>>>>>> tests resulted in the
>>>>>>>>> error "Attempting to use an MPI routine before
>>>>>>>>> initializing MPICH". A
>>>>>>>>> simple test program (attached) worked fine
>>>>>>>>> with the same combination.
>>>>>>>>>
>>>>>>>>> What could be the reason for that?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Roland,
>>>>>>>>>
>>>>>>>>> Can you get a stack trace for this error using the
>>>>>>>>> debugger?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Matt
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Roland Richter
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> What most experimenters take for granted before
>>>>>>>>> they begin their experiments is infinitely more
>>>>>>>>> interesting than any results to which their
>>>>>>>>> experiments lead.
>>>>>>>>> -- Norbert Wiener
>>>>>>>>>
>>>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> What most experimenters take for granted before they
>>>>>>>> begin their experiments is infinitely more interesting
>>>>>>>> than any results to which their experiments lead.
>>>>>>>> -- Norbert Wiener
>>>>>>>>
>>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>
>>>>> <configure.log>
>>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to
>> which their experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which
> their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20211013/d7ff54a1/attachment-0001.html>
More information about the petsc-users
mailing list