[petsc-users] Cross-compilation cluster

[Balay, Satish]

As these warnings indicate - 64bit compiler ignores 32bit libraries [so
they don't get used].

i.e mixing 32bit and 64bit libraries ins not the cause of your
problems on this machine..

Satish


On Wed, 13 Mar 2019, Amneet Bhalla via petsc-users wrote:

> Hi Folks,
> 
> I am on a cluster that has -L/lib dir with 32-bit libraries and -L/lib64
> with 64-bit libraries. During compilation of some of libraries required for
> my code (such as SAMRAI and libMesh) both paths
> get picked  -L/lib and -L/lib64.
> 
> I am seeing some sporadic behavior in runtime when at some timesteps PETSc
> does not converge. The same code with the same number of processors run
> just fine on my workstation that has just 64-bit version of libraries.
> 
> Even during the final linking stage of the executable, the linker gives
> warnings like
> 
> ld: skipping incompatible //lib/libm.so when searching for -lm
> 
> ld: skipping incompatible /lib/libm.so when searching for -lm
> 
> ld: skipping incompatible /lib/libm.so when searching for -lm
> 
> ld: skipping incompatible //lib/libpthread.so when searching for -lpthread
> 
> ld: skipping incompatible /lib/libpthread.so when searching for -lpthread
> 
> ld: skipping incompatible /lib/libpthread.so when searching for -lpthread
> 
> ld: skipping incompatible //lib/libdl.so when searching for -ldl
> 
> ld: skipping incompatible //lib/libc.so when searching for -lc
> 
> ld: skipping incompatible /lib/libc.so when searching for -lc
> 
> ld: skipping incompatible /lib/libc.so when searching for -lc
> but the executable runs.
> 
> 
> This is during config of SAMRAI when it picks both -L/lib and -L/lib64:
> 
> checking whether we are using the GNU Fortran 77 compiler... no
> 
> checking whether ifort accepts -g... yes
> 
> checking how to get verbose linking output from ifort... -v
> 
> checking for Fortran 77 libraries of ifort...
> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4
> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64
> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64
> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64
> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin
> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/
> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64
> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/ -L/lib/../lib64
> -L/lib/../lib64/ -L/usr/lib/../lib64 -L/usr/lib/../lib64/
> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4/
> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64/
> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64/
> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64/
> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../ -L/lib64 -L/lib/
> -L/usr/lib64 -L/usr/lib -lifport -lifcoremt -limf -lsvml -lm -lipgo -lirc
> -lpthread -lgcc -lgcc_s -lirc_s -ldl
> 
> libMesh is also picking that path
> 
> libmesh_optional_LIBS............ : -lhdf5 -lhdf5_cpp -lz
> -L/home/asbhalla/softwares/PETSc-BitBucket/PETSc/linux-opt/lib
> -Wl,-rpath,/home/asbhalla/softwares/PETSc-BitBucket/PETSc/linux-opt/lib
> -Wl,-rpath,/opt/intel/mkl/lib/intel64 -L/opt/intel/mkl/lib/intel64
> -Wl,-rpath,/opt/mellanox/hcoll/lib -L/opt/mellanox/hcoll/lib
> -Wl,-rpath,/opt/mellanox/mxm/lib -L/opt/mellanox/mxm/lib
> -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4
> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4
> -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64
> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64
> -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64
> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64
> -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64
> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64
> -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin
> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin
> -lpetsc -lHYPRE -lmkl_intel_lp64 -lmkl_sequential -lmkl_core
> -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lifport
> -lifcoremt_pic -limf -lsvml -lm -lipgo -lirc -lpthread -lgcc_s -lirc_s
> -lstdc++ -ldl -L/lib -Wl,-rpath,/lib
> -Wl,-rpath,/usr/local/mpi/intel/openmpi-4.0.0/lib64
> -L/usr/local/mpi/intel/openmpi-4.0.0/lib64
> -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.8.5
> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5
> 
> Perhaps PETSc also picks up both versions (and there is a way to query it
> from PETSc?), but I can't confirm this. Is there a way to instruct make to
> select only -L/lib64? I want to rule out that 32-bit dynamic library is not
> a culprit for the random non-convergence of PETSc solvers and the eventual
> crash of the simulations. I have tried both gcc-7.3.0 and intel-18
> compilers -- but the same thing is happening.
> 
> 
>