[petsc-users] Cross-compilation cluster

Sun Mar 17 22:41:14 CDT 2019

On Sun, Mar 17, 2019 at 11:23 PM Amneet Bhalla <mail2amneet at gmail.com>
wrote:

> Apparently the problem is with openmpi 4.0 (or perhaps latest PETSc 3.10.4
> not being compatible with it). I switched to openmpi 1.10.7 and PETSc 3.10,
> which is what I was using on my workstation, and everything works fine.
>
> ====
>
> Matt, I tried your suggestion. Apparently, Silo when compiled with mpi
> wrappers (libsilo.a) does not produce link flags, as required by
> AC_CHECK_LINKFLAGS(silo). When I compile it with serial compiler it does
> not cause that check failure.
>
> I noticed that configure help of openmpi has some options to add some
> extra link flags. Perhaps it is supposed to circumvent situations like
> above.
>

Long ago I made Silo work with MPI compilers, but we dropped support since
no one was using it. Is Silo still supported?

  Thanks,

    Matt

> On Thu, Mar 14, 2019 at 10:04 AM Amneet Bhalla <mail2amneet at gmail.com>
> wrote:
>
>> Hmmn, ok. I will try it out.
>>
>> On Thu, Mar 14, 2019 at 9:34 AM Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Thu, Mar 14, 2019 at 10:05 AM Amneet Bhalla <mail2amneet at gmail.com>
>>> wrote:
>>>
>>>> I need the serial built of Silo and HDF5 libraries. I am sure that the
>>>> MPI wrappers are coming from the serial compilers.
>>>>
>>>> i.e, If I do
>>>>
>>>> gcc —version and mpicc —version, I see the same GCC version.
>>>>
>>>
>>> That is in a perfect world. Unfortunately, flags for compilers can make
>>> incompatible libraries. Its no problem
>>> to give the MPI compilers to Silo and HDF5, even if you do not want to
>>> link the libraries.
>>>
>>>    Matt
>>>
>>>
>>>> On Thu, Mar 14, 2019 at 6:44 AM Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>>> It is very dangerous to use different compilers. I would make sure
>>>>> that all the compilers are the MPI compilers.
>>>>>
>>>>>   Thanks,
>>>>>
>>>>>     Matt
>>>>>
>>>>> On Thu, Mar 14, 2019 at 8:46 AM Amneet Bhalla <mail2amneet at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Ah, Ok. Do serial compilers look OK to you?
>>>>>>
>>>>>> Can lib-32 and lib-64 (say -lm) operate simulataneously during
>>>>>> runtime, or this is my imagination?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 14, 2019 at 5:36 AM Matthew Knepley <knepley at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> On Thu, Mar 14, 2019 at 8:28 AM Amneet Bhalla <mail2amneet at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Matt -- SAMRAI, PETSc, and libMesh configure logs are attached with
>>>>>>>> this email. Also including some other log files in case they are useful.
>>>>>>>>
>>>>>>>
>>>>>>> Okay, PETSc is not sticking in a /usr/lib (or /usr/lib64). However,
>>>>>>> I can see that you mpif90 (and perhaps other of the wrappers) are
>>>>>>> reporting both /usr/lib64 AND /usr/lib flags, and I am guessing that
>>>>>>> is where the other configures are picking it up.
>>>>>>>
>>>>>>>   Thanks,
>>>>>>>
>>>>>>>     Matt
>>>>>>>
>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> --Amneet
>>>>>>>>
>>>>>>>> On Thu, Mar 14, 2019 at 4:21 AM Matthew Knepley <knepley at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> In order to see why each flag was included, we need to see
>>>>>>>>> configure.log.
>>>>>>>>>
>>>>>>>>>   Thanks,
>>>>>>>>>
>>>>>>>>>      Matt
>>>>>>>>>
>>>>>>>>> On Thu, Mar 14, 2019 at 2:40 AM Amneet Bhalla via petsc-users <
>>>>>>>>> petsc-users at mcs.anl.gov> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Folks,
>>>>>>>>>>
>>>>>>>>>> I am on a cluster that has -L/lib dir with 32-bit libraries and
>>>>>>>>>> -L/lib64 with 64-bit libraries. During compilation of some of
>>>>>>>>>> libraries required for my code (such as SAMRAI and libMesh) both paths
>>>>>>>>>> get picked  -L/lib and -L/lib64.
>>>>>>>>>>
>>>>>>>>>> I am seeing some sporadic behavior in runtime when at some
>>>>>>>>>> timesteps PETSc does not converge. The same code with the same number of
>>>>>>>>>> processors run just fine on my workstation that has just 64-bit version of
>>>>>>>>>> libraries.
>>>>>>>>>>
>>>>>>>>>> Even during the final linking stage of the executable, the linker
>>>>>>>>>> gives warnings like
>>>>>>>>>>
>>>>>>>>>> ld: skipping incompatible //lib/libm.so when searching for -lm
>>>>>>>>>>
>>>>>>>>>> ld: skipping incompatible /lib/libm.so when searching for -lm
>>>>>>>>>>
>>>>>>>>>> ld: skipping incompatible /lib/libm.so when searching for -lm
>>>>>>>>>>
>>>>>>>>>> ld: skipping incompatible //lib/libpthread.so when searching for
>>>>>>>>>> -lpthread
>>>>>>>>>>
>>>>>>>>>> ld: skipping incompatible /lib/libpthread.so when searching for
>>>>>>>>>> -lpthread
>>>>>>>>>>
>>>>>>>>>> ld: skipping incompatible /lib/libpthread.so when searching for
>>>>>>>>>> -lpthread
>>>>>>>>>>
>>>>>>>>>> ld: skipping incompatible //lib/libdl.so when searching for -ldl
>>>>>>>>>>
>>>>>>>>>> ld: skipping incompatible //lib/libc.so when searching for -lc
>>>>>>>>>>
>>>>>>>>>> ld: skipping incompatible /lib/libc.so when searching for -lc
>>>>>>>>>>
>>>>>>>>>> ld: skipping incompatible /lib/libc.so when searching for -lc
>>>>>>>>>> but the executable runs.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> This is during config of SAMRAI when it picks both -L/lib and
>>>>>>>>>> -L/lib64:
>>>>>>>>>>
>>>>>>>>>> checking whether we are using the GNU Fortran 77 compiler... no
>>>>>>>>>>
>>>>>>>>>> checking whether ifort accepts -g... yes
>>>>>>>>>>
>>>>>>>>>> checking how to get verbose linking output from ifort... -v
>>>>>>>>>>
>>>>>>>>>> checking for Fortran 77 libraries of ifort...
>>>>>>>>>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4
>>>>>>>>>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64
>>>>>>>>>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64
>>>>>>>>>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64
>>>>>>>>>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin
>>>>>>>>>> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/
>>>>>>>>>> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64
>>>>>>>>>> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/ -L/lib/../lib64
>>>>>>>>>> -L/lib/../lib64/ -L/usr/lib/../lib64 -L/usr/lib/../lib64/
>>>>>>>>>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4/
>>>>>>>>>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64/
>>>>>>>>>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64/
>>>>>>>>>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64/
>>>>>>>>>> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../ -L/lib64 -L/lib/
>>>>>>>>>> -L/usr/lib64 -L/usr/lib -lifport -lifcoremt -limf -lsvml -lm -lipgo -lirc
>>>>>>>>>> -lpthread -lgcc -lgcc_s -lirc_s -ldl
>>>>>>>>>>
>>>>>>>>>> libMesh is also picking that path
>>>>>>>>>>
>>>>>>>>>> libmesh_optional_LIBS............ : -lhdf5 -lhdf5_cpp -lz
>>>>>>>>>> -L/home/asbhalla/softwares/PETSc-BitBucket/PETSc/linux-opt/lib
>>>>>>>>>> -Wl,-rpath,/home/asbhalla/softwares/PETSc-BitBucket/PETSc/linux-opt/lib
>>>>>>>>>> -Wl,-rpath,/opt/intel/mkl/lib/intel64 -L/opt/intel/mkl/lib/intel64
>>>>>>>>>> -Wl,-rpath,/opt/mellanox/hcoll/lib -L/opt/mellanox/hcoll/lib
>>>>>>>>>> -Wl,-rpath,/opt/mellanox/mxm/lib -L/opt/mellanox/mxm/lib
>>>>>>>>>> -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4
>>>>>>>>>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4
>>>>>>>>>> -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64
>>>>>>>>>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64
>>>>>>>>>> -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64
>>>>>>>>>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64
>>>>>>>>>> -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64
>>>>>>>>>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64
>>>>>>>>>> -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin
>>>>>>>>>> -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin
>>>>>>>>>> -lpetsc -lHYPRE -lmkl_intel_lp64 -lmkl_sequential -lmkl_core
>>>>>>>>>> -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lifport
>>>>>>>>>> -lifcoremt_pic -limf -lsvml -lm -lipgo -lirc -lpthread -lgcc_s -lirc_s
>>>>>>>>>> -lstdc++ -ldl -L/lib -Wl,-rpath,/lib
>>>>>>>>>> -Wl,-rpath,/usr/local/mpi/intel/openmpi-4.0.0/lib64
>>>>>>>>>> -L/usr/local/mpi/intel/openmpi-4.0.0/lib64
>>>>>>>>>> -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.8.5
>>>>>>>>>> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5
>>>>>>>>>>
>>>>>>>>>> Perhaps PETSc also picks up both versions (and there is a way to
>>>>>>>>>> query it from PETSc?), but I can't confirm this. Is there a way to instruct
>>>>>>>>>> make to select only -L/lib64? I want to rule out that 32-bit dynamic
>>>>>>>>>> library is not a culprit for the random non-convergence of PETSc solvers
>>>>>>>>>> and the eventual crash of the simulations. I have tried both gcc-7.3.0 and
>>>>>>>>>> intel-18 compilers -- but the same thing is happening.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> --Amneet
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> What most experimenters take for granted before they begin their
>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>> experiments lead.
>>>>>>>>> -- Norbert Wiener
>>>>>>>>>
>>>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> --Amneet
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>>>
>>>>>> --
>>>>>> --Amneet
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>
>>>> --
>>>> --Amneet
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>> --
>> --Amneet
>>
>>
>>
>> --
> --Amneet
>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190317/bcae3789/attachment-0001.html>