[petsc-users] Cross-compilation cluster

Matthew Knepley knepley at gmail.com
Thu Mar 14 08:44:40 CDT 2019 

It is very dangerous to use different compilers. I would make sure that all
the compilers are the MPI compilers.

  Thanks,

    Matt

On Thu, Mar 14, 2019 at 8:46 AM Amneet Bhalla <mail2amneet at gmail.com> wrote:

> Ah, Ok. Do serial compilers look OK to you?
>
> Can lib-32 and lib-64 (say -lm) operate simulataneously during runtime, or
> this is my imagination?
>
>
>
> On Thu, Mar 14, 2019 at 5:36 AM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Thu, Mar 14, 2019 at 8:28 AM Amneet Bhalla <mail2amneet at gmail.com>
>> wrote:
>>
>>> Matt -- SAMRAI, PETSc, and libMesh configure logs are attached with this
>>> email. Also including some other log files in case they are useful.
>>>
>>
>> Okay, PETSc is not sticking in a /usr/lib (or /usr/lib64). However, I can
>> see that you mpif90 (and perhaps other of the wrappers) are
>> reporting both /usr/lib64 AND /usr/lib flags, and I am guessing that is
>> where the other configures are picking it up.
>>
>>   Thanks,
>>
>>     Matt
>>
>>
>>> Thanks,
>>> --Amneet
>>>
>>> On Thu, Mar 14, 2019 at 4:21 AM Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> In order to see why each flag was included, we need to see
>>>> configure.log.
>>>>
>>>>   Thanks,
>>>>
>>>>      Matt
>>>>
>>>> On Thu, Mar 14, 2019 at 2:40 AM Amneet Bhalla via petsc-users <
>>>> petsc-users at mcs.anl.gov> wrote:
>>>>
>>>>> Hi Folks,
>>>>>
>>>>> I am on a cluster that has -L/lib dir with 32-bit libraries and
>>>>> -L/lib64 with 64-bit libraries. During compilation of some of
>>>>> libraries required for my code (such as SAMRAI and libMesh) both paths
>>>>> get picked  -L/lib and -L/lib64.
>>>>>
>>>>> I am seeing some sporadic behavior in runtime when at some timesteps
>>>>> PETSc does not converge. The same code with the same number of processors
>>>>> run just fine on my workstation that has just 64-bit version of libraries.
>>>>>
>>>>> Even during the final linking stage of the executable, the linker
>>>>> gives warnings like
>>>>>
>>>>> ld: skipping incompatible //lib/libm.so when searching for -lm
>>>>>
>>>>> ld: skipping incompatible /lib/libm.so when searching for -lm
>>>>>
>>>>> ld: skipping incompatible /lib/libm.so when searching for -lm
>>>>>
>>>>> ld: skipping incompatible //lib/libpthread.so when searching for
>>>>> -lpthread
>>>>>
>>>>> ld: skipping incompatible /lib/libpthread.so when searching for
>>>>> -lpthread
>>>>>
>>>>> ld: skipping incompatible /lib/libpthread.so when searching for
>>>>> -lpthread
>>>>>
>>>>> ld: skipping incompatible //lib/libdl.so when searching for -ldl
>>>>>
>>>>> ld: skipping incompatible //lib/libc.so when searching for -lc
>>>>>
>>>>> ld: skipping incompatible /lib/libc.so when searching for -lc
>>>>>
>>>>> ld: skipping incompatible /lib/libc.so when searching for -lc
>>>>> but the executable runs.
>>>>>
>>>>>
>>>>> This is during config of SAMRAI when it picks both -L/lib and -L/lib64:
>>>>>
>>>>> checking whether we are using the GNU Fortran 77 compiler... no
>>>>>
>>>>> checking whether ifort accepts -g... yes
>>>>>
>>>>> checking how to get verbose linking output from ifort... -v
>>>>>
>>>>> checking for Fortran 77 libraries of ifort...
>>>>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4
>>>>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64
>>>>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64
>>>>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64
>>>>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin
>>>>> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/
>>>>> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64
>>>>> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/ -L/lib/../lib64
>>>>> -L/lib/../lib64/ -L/usr/lib/../lib64 -L/usr/lib/../lib64/
>>>>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4/
>>>>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64/
>>>>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64/
>>>>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64/
>>>>> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../ -L/lib64 -L/lib/
>>>>> -L/usr/lib64 -L/usr/lib -lifport -lifcoremt -limf -lsvml -lm -lipgo -lirc
>>>>> -lpthread -lgcc -lgcc_s -lirc_s -ldl
>>>>>
>>>>> libMesh is also picking that path
>>>>>
>>>>> libmesh_optional_LIBS............ : -lhdf5 -lhdf5_cpp -lz
>>>>> -L/home/asbhalla/softwares/PETSc-BitBucket/PETSc/linux-opt/lib
>>>>> -Wl,-rpath,/home/asbhalla/softwares/PETSc-BitBucket/PETSc/linux-opt/lib
>>>>> -Wl,-rpath,/opt/intel/mkl/lib/intel64 -L/opt/intel/mkl/lib/intel64
>>>>> -Wl,-rpath,/opt/mellanox/hcoll/lib -L/opt/mellanox/hcoll/lib
>>>>> -Wl,-rpath,/opt/mellanox/mxm/lib -L/opt/mellanox/mxm/lib
>>>>> -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4
>>>>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4
>>>>> -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64
>>>>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64
>>>>> -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64
>>>>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64
>>>>> -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64
>>>>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64
>>>>> -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin
>>>>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin
>>>>> -lpetsc -lHYPRE -lmkl_intel_lp64 -lmkl_sequential -lmkl_core
>>>>> -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lifport
>>>>> -lifcoremt_pic -limf -lsvml -lm -lipgo -lirc -lpthread -lgcc_s -lirc_s
>>>>> -lstdc++ -ldl -L/lib -Wl,-rpath,/lib
>>>>> -Wl,-rpath,/usr/local/mpi/intel/openmpi-4.0.0/lib64
>>>>> -L/usr/local/mpi/intel/openmpi-4.0.0/lib64
>>>>> -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.8.5
>>>>> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5
>>>>>
>>>>> Perhaps PETSc also picks up both versions (and there is a way to query
>>>>> it from PETSc?), but I can't confirm this. Is there a way to instruct make
>>>>> to select only -L/lib64? I want to rule out that 32-bit dynamic library is
>>>>> not a culprit for the random non-convergence of PETSc solvers and the
>>>>> eventual crash of the simulations. I have tried both gcc-7.3.0 and intel-18
>>>>> compilers -- but the same thing is happening.
>>>>>
>>>>>
>>>>> --
>>>>> --Amneet
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>>>
>>> --
>>> --Amneet
>>>
>>>
>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
> --
> --Amneet
>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190314/eae386ea/attachment-0001.html>

More information about the petsc-users mailing list