<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi Folks, <div><br></div><div>I am on a cluster that has -L/lib dir with 32-bit libraries and -L/lib64 with 64-bit libraries. During compilation of some of libraries required for my code (such as SAMRAI and libMesh) both paths</div><div>get picked  -L/lib and -L/lib64. </div><div><br></div><div>I am seeing some sporadic behavior in runtime when at some timesteps PETSc does not converge. The same code with the same number of processors run just fine on my workstation that has just 64-bit version of libraries. </div><div><br></div><div>Even during the final linking stage of the executable, the linker gives warnings like </div><div><br></div><div><p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85098)"><span style="font-variant-ligatures:no-common-ligatures">ld: skipping incompatible //lib/libm.so when searching for -lm</span></p>
<p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85098)"><span style="font-variant-ligatures:no-common-ligatures">ld: skipping incompatible /lib/libm.so when searching for -lm</span></p>
<p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85098)"><span style="font-variant-ligatures:no-common-ligatures">ld: skipping incompatible /lib/libm.so when searching for -lm</span></p>
<p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85098)"><span style="font-variant-ligatures:no-common-ligatures">ld: skipping incompatible //lib/libpthread.so when searching for -lpthread</span></p>
<p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85098)"><span style="font-variant-ligatures:no-common-ligatures">ld: skipping incompatible /lib/libpthread.so when searching for -lpthread</span></p>
<p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85098)"><span style="font-variant-ligatures:no-common-ligatures">ld: skipping incompatible /lib/libpthread.so when searching for -lpthread</span></p>
<p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85098)"><span style="font-variant-ligatures:no-common-ligatures">ld: skipping incompatible //lib/libdl.so when searching for -ldl</span></p><p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85098)"><span style="font-variant-ligatures:no-common-ligatures">ld: skipping incompatible //lib/libc.so when searching for -lc</span></p><p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85098)"><span style="font-variant-ligatures:no-common-ligatures">ld: skipping incompatible /lib/libc.so when searching for -lc</span></p><p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85098)"><span style="font-variant-ligatures:no-common-ligatures"> </span></p><p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85098)"><span style="font-variant-ligatures:no-common-ligatures">ld: skipping incompatible /lib/libc.so when searching for -lc</span></p></div><div><span style="font-variant-ligatures:no-common-ligatures">but the executable runs.  </span></div><div><br></div><div><br></div><div>This is during config of SAMRAI when it picks both -L/lib and -L/lib64:</div><div><br></div><div><p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85098)"><span style="font-variant-ligatures:no-common-ligatures">checking whether we are using the GNU Fortran 77 compiler... no</span></p><p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85098)"><span style="font-variant-ligatures:no-common-ligatures">checking whether ifort accepts -g... yes</span></p><p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85098)"><span style="font-variant-ligatures:no-common-ligatures">checking how to get verbose linking output from ifort... -v</span></p><p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85098)"><span style="font-variant-ligatures:no-common-ligatures">checking for Fortran 77 libraries of ifort...  -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/ -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/ -L/lib/../lib64 -L/lib/../lib64/ -L/usr/lib/../lib64 -L/usr/lib/../lib64/ -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4/ -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64/ -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64/ -L/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64/ -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../ -L/lib64 -L/lib/ -L/usr/lib64 -L/usr/lib -lifport -lifcoremt -limf -lsvml -lm -lipgo -lirc -lpthread -lgcc -lgcc_s -lirc_s -ldl</span></p></div><div><br></div><div>libMesh is also picking that path <div><br></div><div><p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85098)"><span style="font-variant-ligatures:no-common-ligatures">libmesh_optional_LIBS............ : -lhdf5 -lhdf5_cpp -lz -L/home/asbhalla/softwares/PETSc-BitBucket/PETSc/linux-opt/lib -Wl,-rpath,/home/asbhalla/softwares/PETSc-BitBucket/PETSc/linux-opt/lib -Wl,-rpath,/opt/intel/mkl/lib/intel64 -L/opt/intel/mkl/lib/intel64 -Wl,-rpath,/opt/mellanox/hcoll/lib -L/opt/mellanox/hcoll/lib -Wl,-rpath,/opt/mellanox/mxm/lib -L/opt/mellanox/mxm/lib -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4 -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64 -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64 -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64 -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin -lpetsc -lHYPRE -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lifport -lifcoremt_pic -limf -lsvml -lm -lipgo -lirc -lpthread -lgcc_s -lirc_s -lstdc++ -ldl -L/lib -Wl,-rpath,/lib -Wl,-rpath,/usr/local/mpi/intel/openmpi-4.0.0/lib64 -L/usr/local/mpi/intel/openmpi-4.0.0/lib64 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5</span></p></div></div><div><br></div><div>Perhaps PETSc also picks up both versions (and there is a way to query it from PETSc?), but I can't confirm this. Is there a way to instruct make to select only -L/lib64? I want to rule out that 32-bit dynamic library is not a culprit for the random non-convergence of PETSc solvers and the eventual crash of the simulations. I have tried both gcc-7.3.0 and intel-18 compilers -- but the same thing is happening. </div><div><br></div><div> </div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div>--Amneet <br><br></div><div><br></div><div><br></div></div></div></div></div></div></div></div></div>