<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Sat, Feb 10, 2018 at 12:29 PM, Éric Chamberland <span dir="ltr"><<a href="mailto:Eric.Chamberland@giref.ulaval.ca" target="_blank">Eric.Chamberland@giref.ulaval.ca</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi Matthew,</p>
<p>Yes, I heard that the MPI C++ API has been deprecated.</p>
<p>Yes, the mpi_cxx is now missing. Our link line is formed mainly
with PETSC_WITH_EXTERNAL_LIB variable that is now:</p>
<p>PETSC_WITH_EXTERNAL_LIB = -L/opt/petsc-master_debug/lib
-Wl,-rpath,/opt/petsc-master_<wbr>debug/lib
-L/opt/petsc-master_debug/lib
-L/opt/intel/composer_xe_2015.<wbr>2.164/mkl/lib/intel64
-Wl,-rpath,/opt/intel/<wbr>composer_xe_2015.2.164/mkl/<wbr>lib/intel64
-Wl,-rpath,/opt/openmpi-1.10.<wbr>2/lib -L/opt/openmpi-1.10.2/lib
-Wl,-rpath,/usr/lib/gcc/x86_<wbr>64-redhat-linux/4.8.5
-L/usr/lib/gcc/x86_64-redhat-<wbr>linux/4.8.5
-Wl,-rpath,/opt/intel/<wbr>composer_xe_2015.2.164/<wbr>compiler/lib/intel64
-L/opt/intel/composer_xe_2015.<wbr>2.164/compiler/lib/intel64
-Wl,-rpath,/opt/intel/<wbr>composer_xe_2015.2.164/ipp/<wbr>lib/intel64
-L/opt/intel/composer_xe_2015.<wbr>2.164/ipp/lib/intel64
-Wl,-rpath,/opt/intel/<wbr>composer_xe_2015.2.164/tbb/<wbr>lib/intel64/gcc4.4
-L/opt/intel/composer_xe_2015.<wbr>2.164/tbb/lib/intel64/gcc4.4 -lpetsc
-lsuperlu -lsuperlu_dist -lHYPRE -lcmumps -ldmumps -lsmumps
-lzmumps -lmumps_common -lpord -lmkl_scalapack_lp64
-lmkl_blacs_openmpi_lp64 -lml -lumfpack -lklu -lcholmod -lbtf
-lccolamd -lcolamd -lcamd -lamd -lsuitesparseconfig
-lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread
-lmkl_blacs_intelmpi_lp64 -liomp5 -ldl -lpthread -lparmetis
-lmetis -lptesmumps -lptscotch -lptscotcherr -lesmumps -lscotch
-lscotcherr -lm -lX11 -lstdc++ -ldl -lmpi_usempi -lmpi_mpifh -lmpi
-lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lpthread -lrt
-lm -lpthread -lz -lstdc++ -ldl<br>
</p>
<p>We add some stuff to this (our own compiled libs), but nothing
related to MPI or PETSc since we used to rely on
PETSC_WITH_EXTERNAL_LIB for all our diffrement environments (think
about different petsc version, MPI libs and compilers: clang, icc,
g++) and it used to work until yesterday changes...</p>
<p>As you can see in the diff, this "block" of libraries have been
removed from PETSC_WITH_EXTERNAL_LIB: (the -- is part of the diff
output):<br>
</p>
<pre style="color:rgb(0,0,0);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;word-wrap:break-word;white-space:pre-wrap">--ldl
--lmpi_cxx
--lmpi
--lstdc++
--lm
--lgcc_s
--lpthread</pre>
and into this block there was the "-lmpi_cxx" that we need...<br></div></blockquote><div><br></div><div>These libraries were removed from the definition because they were redundant if you are using the compilers that PETSc was configured with:</div><div><br></div><div><div>Compilers:</div><div> C Compiler: /opt/openmpi-1.10.2/bin/mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -g3 </div><div> C++ Compiler: /opt/openmpi-1.10.2/bin/mpicxx -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -fstack-protector -g -fPIC </div><div> Fortran Compiler: /opt/openmpi-1.10.2/bin/mpif90 -fPIC -Wall -ffree-line-length-0 -Wno-unused-dummy-argument -g </div></div><div><br></div><div>The C++ compiler <span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">/opt/openmpi-1.10.2/bin/mpicxx does link using -lmpi_cxx, but your link must not be using it. That is why I wanted to see the whole</span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">link line.</span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"> Thanks,</span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"> Matt</span></div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
I could send you our whole line of link, but the error is into this
small change introduced yesterday into master...<br>
<br>
Thanks a lot!<br>
<br>
Eric<br>
<br>
<div class="gmail-m_-1430438765440137623moz-cite-prefix">Le 18-02-10 à 10:34, Matthew Knepley a
écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Sat, Feb 10, 2018 at 9:42 AM, Éric
Chamberland <span dir="ltr"><<a href="mailto:Eric.Chamberland@giref.ulaval.ca" target="_blank">Eric.Chamberland@giref.<wbr>ulaval.ca</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi,</p>
<p>we used to link our c++ code with PETSc using
PETSC_WITH_EXTERNAL_LIB variable defined in
$PETSC_DIR/lib/petsc/conf/pets<wbr>cvariables and
everything was fine until this night.</p>
<p>It seems some libs have vanished from this variable,
see the diff here:</p>
<pre style="color:rgb(0,0,0);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;word-wrap:break-word;white-space:pre-wrap"> -lptscotcherr
-lesmumps
-lscotch
-lscotcherr
-lm
-lX11
+-lstdc++
-ldl
-lmpi_usempi
-lmpi_mpifh
-lmpi
-lgfortran
-lm
-lgfortran
-lm
-lgcc_s
-lquadmath
-lpthread
--ldl
--lmpi_cxx
--lmpi
--lstdc++
--lm
--lgcc_s
--lpthread
-lrt
-lm
-lpthread
-lz
+-lstdc++
-ldl
causing these errors at link phase for us:
<font color="blue"> /pmi/cmpbib/compilation_BIB_gc<wbr>c_redhat_petsc-master_debug/CO<wbr>MPILE_AUTO/GIREF/obj/dev/Stati<wbr>stiqueMemoire.o: In function `MPI::Op::Init(void (*)(void const*, void*, int, MPI::Datatype const&), bool)': </font>
/opt/openmpi-1.10.2/include/op<wbr>enmpi/ompi/mpi/cxx/op_inln.h:<wbr>122: undefined reference to `ompi_mpi_cxx_op_intercept'
<font color="blue"> /pmi/cmpbib/compilation_BIB_gc<wbr>c_redhat_petsc-master_debug/CO<wbr>MPILE_AUTO/GIREF/obj/dev/Stati<wbr>stiqueMemoire.o: In function `MPI::Intracomm::Create_graph(<wbr>int, int const*, int const*, bool) const': </font>
/opt/openmpi-1.10.2/include/op<wbr>enmpi/ompi/mpi/cxx/intracomm.<wbr>h:25: undefined reference to `MPI::Comm::Comm()'
<font color="blue"> /pmi/cmpbib/compilation_BIB_gc<wbr>c_redhat_petsc-master_debug/CO<wbr>MPILE_AUTO/GIREF/obj/dev/Stati<wbr>stiqueMemoire.o: In function `MPI::Intercomm::Merge(bool) const': </font>
/opt/openmpi-1.10.2/include/op<wbr>enmpi/ompi/mpi/cxx/intracomm_<wbr>inln.h:23: undefined reference to `MPI::Comm::Comm()'
<font color="blue"> /pmi/cmpbib/compilation_BIB_gc<wbr>c_redhat_petsc-master_debug/CO<wbr>MPILE_AUTO/GIREF/obj/dev/Stati<wbr>stiqueMemoire.o: In function `MPI::Intracomm::Split(int, int) const': </font>
/opt/openmpi-1.10.2/include/op<wbr>enmpi/ompi/mpi/cxx/intracomm_<wbr>inln.h:23: undefined reference to `MPI::Comm::Comm()'
<font color="blue"> /pmi/cmpbib/compilation_BIB_gc<wbr>c_redhat_petsc-master_debug/CO<wbr>MPILE_AUTO/GIREF/obj/dev/Stati<wbr>stiqueMemoire.o: In function `MPI::Intracomm::Create(MPI::G<wbr>roup const&) const': </font>
/opt/openmpi-1.10.2/include/op<wbr>enmpi/ompi/mpi/cxx/intracomm_<wbr>inln.h:23: undefined reference to `MPI::Comm::Comm()'
<font color="blue"> /pmi/cmpbib/compilation_BIB_gc<wbr>c_redhat_petsc-master_debug/CO<wbr>MPILE_AUTO/GIREF/obj/dev/Stati<wbr>stiqueMemoire.o: In function `MPI::Intracomm::Clone() const': </font></pre>
</div>
</blockquote>
<div>Hi Eric,</div>
<div><br>
</div>
<div>These symbols are all coming from -lmpi_cxx. I would
note that I believe the MPI Forum has deprecated the C++
interface, so it will</div>
<div>eventually go away. However, lets fix this. In the
configure log, I see that mpicxx -show has that library in
it, so it seems that the link</div>
<div>is not being done with the C++ compiler. Can you send
the whole link line?</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Is this a normal and definitive change or an
unwanted/unobserved bug?</p>
<p>Thanks,</p>
<p>Eric</p>
<p>ps: here are the logs:</p>
<p>this night:</p>
<p>---------</p>
<pre style="color:rgb(0,0,0);font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;word-wrap:break-word;white-space:pre-wrap"><a class="gmail-m_-1430438765440137623m_1095262550823765165moz-txt-link-freetext" href="http://www.giref.ulaval.ca/%7Ecmpgiref/petsc-master-debug/2018.02.10.02h00m01s_configure.log" target="_blank">http://www.giref.ulaval.ca/~cm<wbr>pgiref/petsc-master-debug/2018<wbr>.02.10.02h00m01s_configure.log</a>
<a class="gmail-m_-1430438765440137623m_1095262550823765165moz-txt-link-freetext" href="http://www.giref.ulaval.ca/%7Ecmpgiref/petsc-master-debug/2018.02.10.02h00m01s_make.log" target="_blank">http://www.giref.ulaval.ca/~cm<wbr>pgiref/petsc-master-debug/2018<wbr>.02.10.02h00m01s_make.log</a>
a day before:
------------
<a class="gmail-m_-1430438765440137623m_1095262550823765165moz-txt-link-freetext" href="http://www.giref.ulaval.ca/%7Ecmpgiref/petsc-master-debug/2018.02.09.02h00m02s_configure.log" target="_blank">http://www.giref.ulaval.ca/~cm<wbr>pgiref/petsc-master-debug/2018<wbr>.02.09.02h00m02s_configure.log</a>
<a class="gmail-m_-1430438765440137623m_1095262550823765165moz-txt-link-freetext" href="http://www.giref.ulaval.ca/%7Ecmpgiref/petsc-master-debug/2018.02.09.02h00m02s_make.log" target="_blank">http://www.giref.ulaval.ca/~cm<wbr>pgiref/petsc-master-debug/2018<wbr>.02.09.02h00m02s_make.log</a>
</pre>
</div>
</blockquote>
</div>
<br>
<br clear="all"><span class="gmail-HOEnZb"><font color="#888888">
<div><br>
</div>
-- <br>
<div class="gmail-m_-1430438765440137623gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before
they begin their experiments is infinitely more
interesting than any results to which their
experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="http://www.caam.rice.edu/%7Emk51/" target="_blank">https://www.cse.buffalo.edu/~<wbr>knepley/</a><br>
</div>
</div>
</div>
</div>
</div>
</font></span></div>
</div>
</blockquote>
<br>
</div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.caam.rice.edu/~mk51/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div>
</div></div>