<div><div dir="auto">Apparently the problem is with openmpi 4.0 (or perhaps latest PETSc 3.10.4 not being compatible with it). I switched to openmpi 1.10.7 and PETSc 3.10, which is what I was using on my workstation, and everything works fine.</div><div dir="auto"><br></div><div dir="auto">====</div><div dir="auto"><br></div><div dir="auto">Matt, I tried your suggestion. Apparently, Silo when compiled with mpi wrappers (libsilo.a) does not produce link flags, as required by AC_CHECK_LINKFLAGS(silo). When I compile it with serial compiler it does not cause that check failure.</div><div dir="auto"><br></div><div dir="auto">I noticed that configure help of openmpi has some options to add some extra link flags. Perhaps it is supposed to circumvent situations like above.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 14, 2019 at 10:04 AM Amneet Bhalla <<a href="mailto:mail2amneet@gmail.com">mail2amneet@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div dir="auto">Hmmn, ok. I will try it out.</div></div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 14, 2019 at 9:34 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr">On Thu, Mar 14, 2019 at 10:05 AM Amneet Bhalla <<a href="mailto:mail2amneet@gmail.com" target="_blank">mail2amneet@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><div dir="auto">I need the serial built of Silo and HDF5 libraries. I am sure that the MPI wrappers are coming from the serial compilers.</div></div><div dir="auto"><br></div><div dir="auto">i.e, If I do</div><div dir="auto"><br></div><div dir="auto">gcc —version and mpicc —version, I see the same GCC version.</div></div></blockquote><div><br></div><div>That is in a perfect world. Unfortunately, flags for compilers can make incompatible libraries. Its no problem</div><div>to give the MPI compilers to Silo and HDF5, even if you do not want to link the libraries.</div></div></div><div dir="ltr"><div class="gmail_quote"><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 14, 2019 at 6:44 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">It is very dangerous to use different compilers. I would make sure that all the compilers are the MPI compilers.<div><br></div><div> Thanks,</div><div><br></div><div> Matt</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 14, 2019 at 8:46 AM Amneet Bhalla <<a href="mailto:mail2amneet@gmail.com" target="_blank">mail2amneet@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div dir="auto">Ah, Ok. Do serial compilers look OK to you? </div><div dir="auto"><br></div><div dir="auto">Can lib-32 and lib-64 (say -lm) operate simulataneously during runtime, or this is my imagination?</div></div><div dir="auto"><br></div><div dir="auto"><br></div><div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 14, 2019 at 5:36 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Thu, Mar 14, 2019 at 8:28 AM Amneet Bhalla <<a href="mailto:mail2amneet@gmail.com" target="_blank">mail2amneet@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Matt -- SAMRAI, PETSc, and libMesh configure logs are attached with this email. Also including some other log files in case they are useful. </div></blockquote><div><br></div><div>Okay, PETSc is not sticking in a /usr/lib (or /usr/lib64). However, I can see that you mpif90 (and perhaps other of the wrappers) are</div><div>reporting both /usr/lib64 AND /usr/lib flags, and I am guessing that is where the other configures are picking it up.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div></div></div><div dir="ltr"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Thanks,</div><div>--Amneet </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 14, 2019 at 4:21 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">In order to see why each flag was included, we need to see configure.log.<div><br></div><div> Thanks,</div><div><br></div><div> Matt</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 14, 2019 at 2:40 AM Amneet Bhalla via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi Folks, <div><br></div><div>I am on a cluster that has -L/lib dir with 32-bit libraries and -L/lib64 with 64-bit libraries. During compilation of some of libraries required for my code (such as SAMRAI and libMesh) both paths</div><div>get picked -L/lib and -L/lib64. </div><div><br></div><div>I am seeing some sporadic behavior in runtime when at some timesteps PETSc does not converge. The same code with the same number of processors run just fine on my workstation that has just 64-bit version of libraries. </div><div><br></div><div>Even during the final linking stage of the executable, the linker gives warnings like </div><div><br></div><div><p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span style="font-variant-ligatures:no-common-ligatures">ld: skipping incompatible //lib/libm.so when searching for -lm</span></p>
<p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span style="font-variant-ligatures:no-common-ligatures">ld: skipping incompatible /lib/libm.so when searching for -lm</span></p>
<p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span style="font-variant-ligatures:no-common-ligatures">ld: skipping incompatible /lib/libm.so when searching for -lm</span></p>
<p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span style="font-variant-ligatures:no-common-ligatures">ld: skipping incompatible //lib/libpthread.so when searching for -lpthread</span></p>
<p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span style="font-variant-ligatures:no-common-ligatures">ld: skipping incompatible /lib/libpthread.so when searching for -lpthread</span></p>
<p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span style="font-variant-ligatures:no-common-ligatures">ld: skipping incompatible /lib/libpthread.so when searching for -lpthread</span></p>
<p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span style="font-variant-ligatures:no-common-ligatures">ld: skipping incompatible //lib/libdl.so when searching for -ldl</span></p><p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span style="font-variant-ligatures:no-common-ligatures">ld: skipping incompatible //lib/libc.so when searching for -lc</span></p><p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span style="font-variant-ligatures:no-common-ligatures">ld: skipping incompatible /lib/libc.so when searching for -lc</span></p><p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span style="font-variant-ligatures:no-common-ligatures"> </span></p><p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span style="font-variant-ligatures:no-common-ligatures">ld: skipping incompatible /lib/libc.so when searching for -lc</span></p></div><div><span style="font-variant-ligatures:no-common-ligatures">but the executable runs. </span></div><div><br></div><div><br></div><div>This is during config of SAMRAI when it picks both -L/lib and -L/lib64:</div><div><br></div><div><p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span style="font-variant-ligatures:no-common-ligatures">checking whether we are using the GNU Fortran 77 compiler... no</span></p><p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span style="font-variant-ligatures:no-common-ligatures">checking whether ifort accepts -g... yes</span></p><p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span style="font-variant-ligatures:no-common-ligatures">checking how to get verbose linking output from ifort... -v</span></p><p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span style="font-variant-ligatures:no-common-ligatures">checking for Fortran 77 libraries of ifort... -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/ -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/ -L/lib/../lib64 -L/lib/../lib64/ -L/usr/lib/../lib64 -L/usr/lib/../lib64/ -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4/ -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64/ -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64/ -L/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64/ -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../ -L/lib64 -L/lib/ -L/usr/lib64 -L/usr/lib -lifport -lifcoremt -limf -lsvml -lm -lipgo -lirc -lpthread -lgcc -lgcc_s -lirc_s -ldl</span></p></div><div><br></div><div>libMesh is also picking that path <div><br></div><div><p style="margin:0px;font-stretch:normal;font-size:15px;line-height:normal;font-family:Monaco;color:rgb(244,244,244);background-color:rgba(0,0,0,0.85)"><span style="font-variant-ligatures:no-common-ligatures">libmesh_optional_LIBS............ : -lhdf5 -lhdf5_cpp -lz -L/home/asbhalla/softwares/PETSc-BitBucket/PETSc/linux-opt/lib -Wl,-rpath,/home/asbhalla/softwares/PETSc-BitBucket/PETSc/linux-opt/lib -Wl,-rpath,/opt/intel/mkl/lib/intel64 -L/opt/intel/mkl/lib/intel64 -Wl,-rpath,/opt/mellanox/hcoll/lib -L/opt/mellanox/hcoll/lib -Wl,-rpath,/opt/mellanox/mxm/lib -L/opt/mellanox/mxm/lib -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/tbb/lib/intel64/gcc4.4 -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl/lib/intel64 -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64 -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64 -L/opt/intel/compilers_and_libraries_2018.2.199/linux/ipp/lib/intel64 -Wl,-rpath,/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin -L/opt/intel/compilers_and_libraries_2018.2.199/linux/compiler/lib/intel64_lin -lpetsc -lHYPRE -lmkl_intel_lp64 -lmkl_sequential -lmkl_core -lmpi_usempif08 -lmpi_usempi_ignore_tkr -lmpi_mpifh -lmpi -lifport -lifcoremt_pic -limf -lsvml -lm -lipgo -lirc -lpthread -lgcc_s -lirc_s -lstdc++ -ldl -L/lib -Wl,-rpath,/lib -Wl,-rpath,/usr/local/mpi/intel/openmpi-4.0.0/lib64 -L/usr/local/mpi/intel/openmpi-4.0.0/lib64 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5</span></p></div></div><div><br></div><div>Perhaps PETSc also picks up both versions (and there is a way to query it from PETSc?), but I can't confirm this. Is there a way to instruct make to select only -L/lib64? I want to rule out that 32-bit dynamic library is not a culprit for the random non-convergence of PETSc solvers and the eventual crash of the simulations. I have tried both gcc-7.3.0 and intel-18 compilers -- but the same thing is happening. </div><div><br></div><div> </div>-- <br><div dir="ltr" class="m_4311609668935825221m_-7925870064654127423gmail-m_7866235354521722369m_1645981437486028713m_-2057193163635279886gmail-m_-3903186710666779796m_2128621372028159220m_-4139022886434475495gmail-m_8315293528280414196gmail-m_7046424159374142080gmail-m_-6837913619737073365gmail-m_-8313424874557009010gmail_signature"><div dir="ltr"><div>--Amneet <br><br></div><div><br></div><div><br></div></div></div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_4311609668935825221m_-7925870064654127423gmail-m_7866235354521722369m_1645981437486028713m_-2057193163635279886gmail-m_-3903186710666779796m_2128621372028159220m_-4139022886434475495gmail-m_8315293528280414196gmail-m_7046424159374142080gmail-m_-6837913619737073365gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_4311609668935825221m_-7925870064654127423gmail-m_7866235354521722369m_1645981437486028713m_-2057193163635279886gmail-m_-3903186710666779796m_2128621372028159220m_-4139022886434475495gmail-m_8315293528280414196gmail-m_7046424159374142080gmail_signature"><div dir="ltr"><div>--Amneet <br><br></div><div><br></div><div><br></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_4311609668935825221m_-7925870064654127423gmail-m_7866235354521722369m_1645981437486028713m_-2057193163635279886gmail-m_-3903186710666779796m_2128621372028159220m_-4139022886434475495gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div></div>-- <br><div dir="ltr" class="m_4311609668935825221m_-7925870064654127423gmail-m_7866235354521722369m_1645981437486028713m_-2057193163635279886gmail-m_-3903186710666779796m_2128621372028159220gmail_signature"><div dir="ltr"><div>--Amneet <br><br></div><div><br></div><div><br></div></div></div>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_4311609668935825221m_-7925870064654127423gmail-m_7866235354521722369m_1645981437486028713m_-2057193163635279886gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div>
</blockquote></div></div>-- <br><div dir="ltr" class="m_4311609668935825221m_-7925870064654127423gmail-m_7866235354521722369m_1645981437486028713gmail_signature"><div dir="ltr"><div>--Amneet <br><br></div><div><br></div><div><br></div></div></div>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_4311609668935825221m_-7925870064654127423gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div>
</blockquote></div></div>-- <br><div dir="ltr" class="m_4311609668935825221gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>--Amneet <br><br></div><div><br></div><div><br></div></div></div>
</blockquote></div></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>--Amneet <br><br></div><div><br></div><div><br></div></div></div>