[petsc-users] Error "Attempting to use an MPI routine before initializing MPICH" after compiling PETSc with Intel MPI and GCC

Wed Oct 13 05:26:31 CDT 2021

On Wed, Oct 13, 2021 at 5:53 AM Roland Richter <roland.richter at ntnu.no>
wrote:

> Hei,
>
> I noticed a difference in when the program is running, and when not. The
> code works fine if I compile it via a CMake-file and load PETSc there. If I
> use the compilation line which is included in the Makefiles, then the code
> will fail with the mentioned error. The cmake-generated compilation line
> (including armadillo, because my test sample contained armadillo-code) is
>
> One of these is a compile command and the other is a link command.

   Matt

> */opt/intel/oneapi/mpi/2021.4.0/bin/mpicxx -D__INSDIR__="" -I/include
> -I/opt/petsc/include -I/opt/armadillo/include -std=c++0x -g -MD -MT
> CMakeFiles/main.dir/source/main.cpp.o -MF
> CMakeFiles/main.dir/source/main.cpp.o.d -o
> CMakeFiles/main.dir/source/main.cpp.o -c source/main.cpp*
> */opt/intel/oneapi/mpi/2021.4.0/bin/mpicxx -rdynamic
> CMakeFiles/main.dir/source/main.cpp.o -o main_short
> -Wl,-rpath,/opt/petsc/lib:/opt/armadillo/lib64 /opt/petsc/lib/libpetsc.so
> /opt/armadillo/lib64/libarmadillo.so *
>
> Meanwhile, the original compilation line from PETSc is
>
> *mpicxx -mavx2 -march=native -O3 -fPIC -fopenmp    -I/opt/petsc/include
> -I/opt/armadillo/include -I/opt/intel/oneapi/mkl/latest/include
> -I/opt/fftw3/include -I/opt/hdf5/include -I/opt/boost/include
> source/main.cpp -Wl,-rpath,/opt/petsc/lib -L/opt/petsc/lib
> -Wl,-rpath,/opt/petsc/lib -L/opt/petsc/lib
> -L/opt/intel/oneapi/mkl/latest/lib/intel64 -Wl,-rpath,/opt/fftw3/lib64
> -L/opt/fftw3/lib64 -Wl,-rpath,/opt/armadillo/lib64 -L/opt/armadillo/lib64
> -Wl,-rpath,/opt/intel/oneapi/mkl/latest/lib/intel64
> -Wl,-rpath,/opt/hdf5/lib -L/opt/hdf5/lib
> -Wl,-rpath,/opt/intel/oneapi/mpi/2021.4.0/lib/release
> -L/opt/intel/oneapi/mpi/2021.4.0/lib/release
> -Wl,-rpath,/opt/intel/oneapi/mpi/2021.4.0/lib
> -L/opt/intel/oneapi/mpi/2021.4.0/lib
> -Wl,-rpath,/usr/lib64/gcc/x86_64-suse-linux/11
> -L/usr/lib64/gcc/x86_64-suse-linux/11
> -Wl,-rpath,/opt/intel/oneapi/vpl/2021.6.0/lib
> -L/opt/intel/oneapi/vpl/2021.6.0/lib
> -Wl,-rpath,/opt/intel/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8
> -L/opt/intel/oneapi/tbb/2021.4.0/lib/intel64/gcc4.8
> -Wl,-rpath,/opt/intel/oneapi/mpi/2021.4.0/libfabric/lib
> -L/opt/intel/oneapi/mpi/2021.4.0/libfabric/lib
> -Wl,-rpath,/opt/intel/oneapi/mkl/2021.4.0/lib/intel64
> -L/opt/intel/oneapi/mkl/2021.4.0/lib/intel64
> -Wl,-rpath,/opt/intel/oneapi/ipp/2021.4.0/lib/intel64
> -L/opt/intel/oneapi/ipp/2021.4.0/lib/intel64
> -Wl,-rpath,/opt/intel/oneapi/ippcp/2021.4.0/lib/intel64
> -L/opt/intel/oneapi/ippcp/2021.4.0/lib/intel64
> -Wl,-rpath,/opt/intel/oneapi/dnnl/2021.4.0/cpu_dpcpp_gpu_dpcpp/lib
> -L/opt/intel/oneapi/dnnl/2021.4.0/cpu_dpcpp_gpu_dpcpp/lib
> -Wl,-rpath,/opt/intel/oneapi/dal/2021.4.0/lib/intel64
> -L/opt/intel/oneapi/dal/2021.4.0/lib/intel64
> -Wl,-rpath,/opt/intel/oneapi/compiler/2021.4.0/linux/compiler/lib/intel64_lin
> -L/opt/intel/oneapi/compiler/2021.4.0/linux/compiler/lib/intel64_lin
> -Wl,-rpath,/opt/intel/oneapi/compiler/2021.4.0/linux/lib
> -L/opt/intel/oneapi/compiler/2021.4.0/linux/lib
> -Wl,-rpath,/opt/intel/oneapi/clck/2021.4.0/lib/intel64
> -L/opt/intel/oneapi/clck/2021.4.0/lib/intel64
> -Wl,-rpath,/opt/intel/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp
> -L/opt/intel/oneapi/ccl/2021.4.0/lib/cpu_gpu_dpcpp
> -Wl,-rpath,/usr/x86_64-suse-linux/lib -L/usr/x86_64-suse-linux/lib
> -larmadillo -lpetsc -lHYPRE -lcmumps -ldmumps -lsmumps -lzmumps
> -lmumps_common -lpord -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -lspqr
> -lumfpack -lklu -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd
> -lsuitesparseconfig -lsuperlu -lsuperlu_dist -lEl -lElSuiteSparse -lpmrrr
> -lfftw3_mpi -lfftw3 -lp4est -lsc -lmkl_intel_lp64 -lmkl_core
> -lmkl_intel_thread -liomp5 -ldl -lpthread -lptesmumps -lptscotchparmetis
> -lptscotch -lptscotcherr -lesmumps -lscotch -lscotcherr -lhdf5_hl -lhdf5
> -lparmetis -lmetis -lm -lz -lmuparser -lX11 -lstdc++ -ldl -lmpifort -lmpi
> -lrt -lpthread -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lrt
> -lquadmath -lstdc++ -ldl -o main_long*
>
> Both executables have the same libraries linked to them, but in a
> different order when comparing them with ldd.
>
> Does that explain the observed behavior?
>
> Thanks,
>
> regards,
>
> Roland
> Am 11.10.21 um 15:13 schrieb Roland Richter:
>
> Hei,
>
> the following code works fine:
>
> #include <iostream>
> #include <petsc.h>
>
> static char help[] = "Solves 2D Poisson equation using multigrid.\n\n";
> int main(int argc,char **argv) {
>     PetscInitialize(&argc,&argv,(char*)0,help);
>     std::cout << "Hello World\n";
>     PetscFinalize();
>     return 0;
> }
>
> Regards,
>
> Roland
> Am 11.10.21 um 14:34 schrieb Stefano Zampini:
>
> Can you try with a simple call that only calls PetscInitialize/Finalize?
>
>
> On Oct 11, 2021, at 3:30 PM, Roland Richter <roland.richter at ntnu.no>
> wrote:
>
> At least according to configure.log mpiexec was defined as
>
> Checking for program /opt/intel/oneapi/mpi/2021.4.0//bin/mpiexec...found
>                   Defined make macro "MPIEXECEXECUTABLE" to
> "/opt/intel/oneapi/mpi/2021.4.0/bin/mpiexec"
>
> When running ex19 with this mpiexec it fails with the usual error, even
> though all configuration steps worked fine. I attached the configuration
> log.
>
> Regards,
>
> Roland
> Am 11.10.21 um 14:24 schrieb Stefano Zampini:
>
> You are most probably using a different mpiexec then the one used to
> compile petsc.
>
>
>
> On Oct 11, 2021, at 3:23 PM, Roland Richter <roland.richter at ntnu.no>
> wrote:
>
> I tried either *./ex19* (SNES-example), *mpirun ./ex19* or *mpirun -n 1
> ./ex19*, all with the same result.
>
> Regards,
>
> Roland
> Am 11.10.21 um 14:22 schrieb Matthew Knepley:
>
> On Mon, Oct 11, 2021 at 8:07 AM Roland Richter <roland.richter at ntnu.no>
> wrote:
>
>> Hei,
>>
>> at least in gdb it fails with
>>
>> Attempting to use an MPI routine before initializing MPICH
>> [Inferior 1 (process 7854) exited with code 01]
>> (gdb) backtrace
>> No stack.
>>
>
> What were you running? If it never makes it into PETSc code, I am not sure
> what we are
> doing to cause this.
>
>   Thanks,
>
>      Matt
>
>
>> Regards,
>>
>> Roland
>> Am 11.10.21 um 13:57 schrieb Matthew Knepley:
>>
>> On Mon, Oct 11, 2021 at 5:24 AM Roland Richter <roland.richter at ntnu.no>
>> wrote:
>>
>>> Hei,
>>>
>>> I compiled PETSc with Intel MPI (MPICH) and GCC as compiler (i.e. using
>>> Intel OneAPI together with the supplied mpicxx-compiler). Compilation
>>> and installation worked fine, but running the tests resulted in the
>>> error "Attempting to use an MPI routine before initializing MPICH". A
>>> simple test program (attached) worked fine with the same combination.
>>>
>>> What could be the reason for that?
>>>
>>
>> Hi Roland,
>>
>> Can you get a stack trace for this error using the debugger?
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Thanks!
>>>
>>> Regards,
>>>
>>> Roland Richter
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
> <configure.log>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20211013/602d6d1a/attachment-0001.html>