[petsc-users] Cross-compilation cluster

Thu Mar 14 07:28:15 CDT 2019

Matt -- SAMRAI, PETSc, and libMesh configure logs are attached with this
email. Also including some other log files in case they are useful.

Thanks,
--Amneet

On Thu, Mar 14, 2019 at 4:21 AM Matthew Knepley <knepley at gmail.com> wrote:

> In order to see why each flag was included, we need to see configure.log.
>
>   Thanks,
>
>      Matt
>
> On Thu, Mar 14, 2019 at 2:40 AM Amneet Bhalla via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Hi Folks,
>>
>> I am on a cluster that has -L/lib dir with 32-bit libraries and -L/lib64
>> with 64-bit libraries. During compilation of some of libraries required for
>> my code (such as SAMRAI and libMesh) both paths
>> get picked  -L/lib and -L/lib64.
>>
>> I am seeing some sporadic behavior in runtime when at some timesteps
>> PETSc does not converge. The same code with the same number of processors
>> run just fine on my workstation that has just 64-bit version of libraries.
>>
>> Even during the final linking stage of the executable, the linker gives
>> warnings like
>>
>> ld: skipping incompatible //lib/libm.so when searching for -lm
>>
>> ld: skipping incompatible /lib/libm.so when searching for -lm
>>
>> ld: skipping incompatible /lib/libm.so when searching for -lm
>>
>> ld: skipping incompatible //lib/libpthread.so when searching for -lpthread
>>
>> ld: skipping incompatible /lib/libpthread.so when searching for -lpthread
>>
>> ld: skipping incompatible /lib/libpthread.so when searching for -lpthread
>>
>> ld: skipping incompatible //lib/libdl.so when searching for -ldl
>>
>> ld: skipping incompatible //lib/libc.so when searching for -lc
>>
>> ld: skipping incompatible /lib/libc.so when searching for -lc
>>
>> ld: skipping incompatible /lib/libc.so when searching for -lc
>> but the executable runs.
>>
>>
>> This is during config of SAMRAI when it picks both -L/lib and -L/lib64:
>>
>> checking whether we are using the GNU Fortran 77 compiler... no
>>
>> checking whether ifort accepts -g... yes
>>
>> checking how to get verbose linking output from ifort... -v
>>
>> checking for Fortran 77 libraries of ifort...
>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4
>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64
>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64
>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64
>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin
>> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/
>> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64
>> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/ -L/lib/../lib64
>> -L/lib/../lib64/ -L/usr/lib/../lib64 -L/usr/lib/../lib64/
>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4/
>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64/
>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64/
>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64/
>> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../ -L/lib64 -L/lib/
>> -L/usr/lib64 -L/usr/lib -lifport -lifcoremt -limf -lsvml -lm -lipgo -lirc
>> -lpthread -lgcc -lgcc_s -lirc_s -ldl
>>
>> libMesh is also picking that path
>>
>> libmesh_optional_LIBS............ : -lhdf5 -lhdf5_cpp -lz
>> -L/home/asbhalla/softwares/PETSc-BitBucket/PETSc/linux-opt/lib
>> -Wl,-rpath,/home/asbhalla/softwares/PETSc-BitBucket/PETSc/linux-opt/lib
>> -Wl,-rpath,/opt/intel/mkl/lib/intel64 -L/opt/intel/mkl/lib/intel64
>> -Wl,-rpath,/opt/mellanox/hcoll/lib -L/opt/mellanox/hcoll/lib
>> -Wl,-rpath,/opt/mellanox/mxm/lib -L/opt/mellanox/mxm/lib
>> -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4
>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4
>> -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64
>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64
>> -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64
>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64
>> -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64
>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64
>> -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin
>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin
>> -lpetsc -lHYPRE -lmkl_intel_lp64 -lmkl_sequential -lmkl_core
>> -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lifport
>> -lifcoremt_pic -limf -lsvml -lm -lipgo -lirc -lpthread -lgcc_s -lirc_s
>> -lstdc++ -ldl -L/lib -Wl,-rpath,/lib
>> -Wl,-rpath,/usr/local/mpi/intel/openmpi-4.0.0/lib64
>> -L/usr/local/mpi/intel/openmpi-4.0.0/lib64
>> -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.8.5
>> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5
>>
>> Perhaps PETSc also picks up both versions (and there is a way to query it
>> from PETSc?), but I can't confirm this. Is there a way to instruct make to
>> select only -L/lib64? I want to rule out that 32-bit dynamic library is not
>> a culprit for the random non-convergence of PETSc solvers and the eventual
>> crash of the simulations. I have tried both gcc-7.3.0 and intel-18
>> compilers -- but the same thing is happening.
>>
>>
>> --
>> --Amneet
>>
>>
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>

-- 
--Amneet
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190314/e8ca2c4e/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: libMesh_config.log
Type: application/octet-stream
Size: 248696 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190314/e8ca2c4e/attachment-0006.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: SAMRAI_config.log
Type: application/octet-stream
Size: 202853 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190314/e8ca2c4e/attachment-0007.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PETSc_config.log
Type: application/octet-stream
Size: 10074128 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190314/e8ca2c4e/attachment-0008.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: HDF5_config.log
Type: application/octet-stream
Size: 178092 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190314/e8ca2c4e/attachment-0009.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Silo_config.log
Type: application/octet-stream
Size: 83751 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190314/e8ca2c4e/attachment-0010.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: IBAMR_config.log
Type: application/octet-stream
Size: 398301 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190314/e8ca2c4e/attachment-0011.obj>