[petsc-users] Cross-compilation cluster

[Amneet Bhalla]

Hi Folks,

I am on a cluster that has -L/lib dir with 32-bit libraries and -L/lib64
with 64-bit libraries. During compilation of some of libraries required for
my code (such as SAMRAI and libMesh) both paths
get picked  -L/lib and -L/lib64.

I am seeing some sporadic behavior in runtime when at some timesteps PETSc
does not converge. The same code with the same number of processors run
just fine on my workstation that has just 64-bit version of libraries.

Even during the final linking stage of the executable, the linker gives
warnings like

ld: skipping incompatible //lib/libm.so when searching for -lm

ld: skipping incompatible /lib/libm.so when searching for -lm

ld: skipping incompatible /lib/libm.so when searching for -lm

ld: skipping incompatible //lib/libpthread.so when searching for -lpthread

ld: skipping incompatible /lib/libpthread.so when searching for -lpthread

ld: skipping incompatible /lib/libpthread.so when searching for -lpthread

ld: skipping incompatible //lib/libdl.so when searching for -ldl

ld: skipping incompatible //lib/libc.so when searching for -lc

ld: skipping incompatible /lib/libc.so when searching for -lc

ld: skipping incompatible /lib/libc.so when searching for -lc
but the executable runs.


This is during config of SAMRAI when it picks both -L/lib and -L/lib64:

checking whether we are using the GNU Fortran 77 compiler... no

checking whether ifort accepts -g... yes

checking how to get verbose linking output from ifort... -v

checking for Fortran 77 libraries of ifort...
-L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4
-L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64
-L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64
-L/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64
-L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin
-L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/
-L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64
-L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/ -L/lib/../lib64
-L/lib/../lib64/ -L/usr/lib/../lib64 -L/usr/lib/../lib64/
-L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4/
-L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64/
-L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64/
-L/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64/
-L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../ -L/lib64 -L/lib/
-L/usr/lib64 -L/usr/lib -lifport -lifcoremt -limf -lsvml -lm -lipgo -lirc
-lpthread -lgcc -lgcc_s -lirc_s -ldl

libMesh is also picking that path

libmesh_optional_LIBS............ : -lhdf5 -lhdf5_cpp -lz
-L/home/asbhalla/softwares/PETSc-BitBucket/PETSc/linux-opt/lib
-Wl,-rpath,/home/asbhalla/softwares/PETSc-BitBucket/PETSc/linux-opt/lib
-Wl,-rpath,/opt/intel/mkl/lib/intel64 -L/opt/intel/mkl/lib/intel64
-Wl,-rpath,/opt/mellanox/hcoll/lib -L/opt/mellanox/hcoll/lib
-Wl,-rpath,/opt/mellanox/mxm/lib -L/opt/mellanox/mxm/lib
-Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4
-L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4
-Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64
-L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64
-Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64
-L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64
-Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64
-L/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64
-Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin
-L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin
-lpetsc -lHYPRE -lmkl_intel_lp64 -lmkl_sequential -lmkl_core
-lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lifport
-lifcoremt_pic -limf -lsvml -lm -lipgo -lirc -lpthread -lgcc_s -lirc_s
-lstdc++ -ldl -L/lib -Wl,-rpath,/lib
-Wl,-rpath,/usr/local/mpi/intel/openmpi-4.0.0/lib64
-L/usr/local/mpi/intel/openmpi-4.0.0/lib64
-Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.8.5
-L/usr/lib/gcc/x86_64-redhat-linux/4.8.5

Perhaps PETSc also picks up both versions (and there is a way to query it
from PETSc?), but I can't confirm this. Is there a way to instruct make to
select only -L/lib64? I want to rule out that 32-bit dynamic library is not
a culprit for the random non-convergence of PETSc solvers and the eventual
crash of the simulations. I have tried both gcc-7.3.0 and intel-18
compilers -- but the same thing is happening.


-- 
--Amneet
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190313/0ff11080/attachment.html>