[petsc-users] Error "Attempting to use an MPI routine before initializing MPICH" after compiling PETSc with Intel MPI and GCC
Matthew Knepley
knepley at gmail.com
Wed Oct 13 06:01:12 CDT 2021
On Wed, Oct 13, 2021 at 6:43 AM Roland Richter <roland.richter at ntnu.no>
wrote:
> Shouldn't I see a mixture of MPI-libraries when calling ldd if I mix
> versions of MPI?
>
> Not if the compilation uses one and the link another.
> I also removed all calls to armadillo, and accordingly all references to
> it in the compilation, but the behavior is still unchanged.
>
I do not understand what you mean here. If you remove the
armadiillo library, how are you linking MPI?
Thanks,
Matt
> Regards,
>
> Roland
> Am 13.10.21 um 12:36 schrieb Matthew Knepley:
>
> On Wed, Oct 13, 2021 at 6:32 AM Roland Richter <roland.richter at ntnu.no>
> wrote:
>
>> Yes, the first part (which works) consists out of a compilation line and
>> a linking line, while the second command is a combination of compile- and
>> linking line.
>>
> The link line in the first does not tell us anything because MPI is not
> even present. It is being pulled in I presume from libarmadillo, which we
> cannot see. It still
> seems most likely, as Stefano said, that you are mixing versions of MPI.
>
> Thanks,
>
> Matt
>
>
>> Am 13.10.21 um 12:26 schrieb Matthew Knepley:
>>
>> On Wed, Oct 13, 2021 at 5:53 AM Roland Richter <roland.richter at ntnu.no>
>> wrote:
>>
>>> Hei,
>>>
>>> I noticed a difference in when the program is running, and when not. The
>>> code works fine if I compile it via a CMake-file and load PETSc there. If I
>>> use the compilation line which is included in the Makefiles, then the code
>>> will fail with the mentioned error. The cmake-generated compilation line
>>> (including armadillo, because my test sample contained armadillo-code) is
>>>
>> One of these is a compile command and the other is a link command.
>>
>> Matt
>>
>>> */opt/intel/oneapi/mpi/2021.4.0/bin/mpicxx -D__INSDIR__="" -I/include
>>> -I/opt/petsc/include -I/opt/armadillo/include -std=c++0x -g -MD -MT
>>> CMakeFiles/main.dir/source/main.cpp.o -MF
>>> CMakeFiles/main.dir/source/main.cpp.o.d -o
>>> CMakeFiles/main.dir/source/main.cpp.o -c source/main.cpp*
>>> */opt/intel/oneapi/mpi/2021.4.0/bin/mpicxx -rdynamic
>>> CMakeFiles/main.dir/source/main.cpp.o -o main_short
>>> -Wl,-rpath,/opt/petsc/lib:/opt/armadillo/lib64 /opt/petsc/lib/libpetsc.so
>>> /opt/armadillo/lib64/libarmadillo.so *
>>>
>>> Meanwhile, the original compilation line from PETSc is
>>>
>>> *mpicxx -mavx2 -march=native -O3 -fPIC -fopenmp -I/opt/petsc/include
>>> -I/opt/armadillo/include -I/opt/intel/oneapi/mkl/latest/include
>>> -I/opt/fftw3/include -I/opt/hdf5/include -I/opt/boost/include
>>> source/main.cpp -Wl,-rpath,/opt/petsc/lib -L/opt/petsc/lib
>>> -Wl,-rpath,/opt/petsc/lib -L/opt/petsc/lib
>>> -L/opt/intel/oneapi/mkl/latest/lib/intel64 -Wl,-rpath,/opt/fftw3/lib64
>>> -L/opt/fftw3/lib64 -Wl,-rpath,/opt/armadillo/lib64 -L/opt/armadillo/lib64
>>> -Wl,-rpath,/opt/intel/oneapi/mkl/latest/lib/intel64
>>> -Wl,-rpath,/opt/hdf5/lib -L/opt/hdf5/lib
>>> -Wl,-rpath,/opt/intel/oneapi/mpi/2021.4.0/lib/release
>>> -L/opt/intel/oneapi/mpi/2021.4.0/lib/release
>>> -Wl,-rpath,/opt/intel/oneapi/mpi/2021.4.0/lib
>>> -L/opt/intel/oneapi/mpi/2021.4.0/lib
>>> -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/11
>>> -L/usr/lib64/gcc/x86_64-suse-linux/11
>>> -Wl,-rpath,/opt/intel/oneapi/vpl/2021.6.0/lib
>>> -L/opt/intel/oneapi/vpl/2021.6.0/lib
>>> -Wl,-rpath,/opt/intel/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8
>>> -L/opt/intel/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8
>>> -Wl,-rpath,/opt/intel/oneapi/mpi/2021.4.0/libfabric/lib
>>> -L/opt/intel/oneapi/mpi/2021.4.0/libfabric/lib
>>> -Wl,-rpath,/opt/intel/oneapi/mkl/2021.4.0/lib/intel64
>>> -L/opt/intel/oneapi/mkl/2021.4.0/lib/intel64
>>> -Wl,-rpath,/opt/intel/oneapi/ipp/2021.4.0/lib/intel64
>>> -L/opt/intel/oneapi/ipp/2021.4.0/lib/intel64
>>> -Wl,-rpath,/opt/intel/oneapi/ippcp/2021.4.0/lib/intel64
>>> -L/opt/intel/oneapi/ippcp/2021.4.0/lib/intel64
>>> -Wl,-rpath,/opt/intel/oneapi/dnnl/2021.4.0/cpu_dpcpp_gpu_dpcpp/lib
>>> -L/opt/intel/oneapi/dnnl/2021.4.0/cpu_dpcpp_gpu_dpcpp/lib
>>> -Wl,-rpath,/opt/intel/oneapi/dal/2021.4.0/lib/intel64
>>> -L/opt/intel/oneapi/dal/2021.4.0/lib/intel64
>>> -Wl,-rpath,/opt/intel/oneapi/compiler/2021.4.0/linux/compiler/lib/intel64_lin
>>> -L/opt/intel/oneapi/compiler/2021.4.0/linux/compiler/lib/intel64_lin
>>> -Wl,-rpath,/opt/intel/oneapi/compiler/2021.4.0/linux/lib
>>> -L/opt/intel/oneapi/compiler/2021.4.0/linux/lib
>>> -Wl,-rpath,/opt/intel/oneapi/clck/2021.4.0/lib/intel64
>>> -L/opt/intel/oneapi/clck/2021.4.0/lib/intel64
>>> -Wl,-rpath,/opt/intel/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp
>>> -L/opt/intel/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp
>>> -Wl,-rpath,/usr/x86_64-suse-linux/lib -L/usr/x86_64-suse-linux/lib
>>> -larmadillo -lpetsc -lHYPRE -lcmumps -ldmumps -lsmumps -lzmumps
>>> -lmumps_common -lpord -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -lspqr
>>> -lumfpack -lklu -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd
>>> -lsuitesparseconfig -lsuperlu -lsuperlu_dist -lEl -lElSuiteSparse -lpmrrr
>>> -lfftw3_mpi -lfftw3 -lp4est -lsc -lmkl_intel_lp64 -lmkl_core
>>> -lmkl_intel_thread -liomp5 -ldl -lpthread -lptesmumps -lptscotchparmetis
>>> -lptscotch -lptscotcherr -lesmumps -lscotch -lscotcherr -lhdf5_hl -lhdf5
>>> -lparmetis -lmetis -lm -lz -lmuparser -lX11 -lstdc++ -ldl -lmpifort -lmpi
>>> -lrt -lpthread -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lrt
>>> -lquadmath -lstdc++ -ldl -o main_long*
>>>
>>> Both executables have the same libraries linked to them, but in a
>>> different order when comparing them with ldd.
>>>
>>> Does that explain the observed behavior?
>>>
>>> Thanks,
>>>
>>> regards,
>>>
>>> Roland
>>> Am 11.10.21 um 15:13 schrieb Roland Richter:
>>>
>>> Hei,
>>>
>>> the following code works fine:
>>>
>>> #include <iostream>
>>> #include <petsc.h>
>>>
>>> static char help[] = "Solves 2D Poisson equation using multigrid.\n\n";
>>> int main(int argc,char **argv) {
>>> PetscInitialize(&argc,&argv,(char*)0,help);
>>> std::cout << "Hello World\n";
>>> PetscFinalize();
>>> return 0;
>>> }
>>>
>>> Regards,
>>>
>>> Roland
>>> Am 11.10.21 um 14:34 schrieb Stefano Zampini:
>>>
>>> Can you try with a simple call that only calls PetscInitialize/Finalize?
>>>
>>>
>>> On Oct 11, 2021, at 3:30 PM, Roland Richter <roland.richter at ntnu.no>
>>> wrote:
>>>
>>> At least according to configure.log mpiexec was defined as
>>>
>>> Checking for program /opt/intel/oneapi/mpi/2021.4.0//bin/mpiexec...found
>>> Defined make macro "MPIEXECEXECUTABLE" to
>>> "/opt/intel/oneapi/mpi/2021.4.0/bin/mpiexec"
>>>
>>> When running ex19 with this mpiexec it fails with the usual error, even
>>> though all configuration steps worked fine. I attached the configuration
>>> log.
>>>
>>> Regards,
>>>
>>> Roland
>>> Am 11.10.21 um 14:24 schrieb Stefano Zampini:
>>>
>>> You are most probably using a different mpiexec then the one used to
>>> compile petsc.
>>>
>>>
>>>
>>> On Oct 11, 2021, at 3:23 PM, Roland Richter <roland.richter at ntnu.no>
>>> wrote:
>>>
>>> I tried either *./ex19* (SNES-example), *mpirun ./ex19* or *mpirun -n 1
>>> ./ex19*, all with the same result.
>>>
>>> Regards,
>>>
>>> Roland
>>> Am 11.10.21 um 14:22 schrieb Matthew Knepley:
>>>
>>> On Mon, Oct 11, 2021 at 8:07 AM Roland Richter <roland.richter at ntnu.no>
>>> wrote:
>>>
>>>> Hei,
>>>>
>>>> at least in gdb it fails with
>>>>
>>>> Attempting to use an MPI routine before initializing MPICH
>>>> [Inferior 1 (process 7854) exited with code 01]
>>>> (gdb) backtrace
>>>> No stack.
>>>>
>>>
>>> What were you running? If it never makes it into PETSc code, I am not
>>> sure what we are
>>> doing to cause this.
>>>
>>> Thanks,
>>>
>>> Matt
>>>
>>>
>>>> Regards,
>>>>
>>>> Roland
>>>> Am 11.10.21 um 13:57 schrieb Matthew Knepley:
>>>>
>>>> On Mon, Oct 11, 2021 at 5:24 AM Roland Richter <roland.richter at ntnu.no>
>>>> wrote:
>>>>
>>>>> Hei,
>>>>>
>>>>> I compiled PETSc with Intel MPI (MPICH) and GCC as compiler (i.e. using
>>>>> Intel OneAPI together with the supplied mpicxx-compiler). Compilation
>>>>> and installation worked fine, but running the tests resulted in the
>>>>> error "Attempting to use an MPI routine before initializing MPICH". A
>>>>> simple test program (attached) worked fine with the same combination.
>>>>>
>>>>> What could be the reason for that?
>>>>>
>>>>
>>>> Hi Roland,
>>>>
>>>> Can you get a stack trace for this error using the debugger?
>>>>
>>>> Thanks,
>>>>
>>>> Matt
>>>>
>>>>
>>>>> Thanks!
>>>>>
>>>>> Regards,
>>>>>
>>>>> Roland Richter
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>>
>>> <configure.log>
>>>
>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20211013/f54159b3/attachment-0001.html>
More information about the petsc-users
mailing list