[petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

Smith, Barry F. bsmith at mcs.anl.gov
Sat Feb 10 11:58:54 CST 2018 


> On Feb 10, 2018, at 11:51 AM, Éric Chamberland <Eric.Chamberland at giref.ulaval.ca> wrote:
> 
> Ok guys!
> 
> You found it (again...).  We are no using mpic++ to link!!!
> 
> We are not always using the same compiler that we used to compile petsc+mpi, so we do not use mpic++ to compile (excepted for intelmpi).
> 
> For example, I compile MPI+Petsc with g++, but I am working with clang++ and may switch to g++ for my day to day work.  This is perfectly working since... long long time ago (I won't reveal my age... ;) )

   WOW. You are lucky this worked for you. I don't think missing C++ compilers/linkers is ever safe?

> 
> I understand you choice, but your PETSC_WITH_EXTERNAL_LIB variable was helping me to painlessly find the good linking options.
> 
> I can work to manually add it when it is relevant into our compilation flags...
> 
> How did you managed to extract the libs from all the different flavors of MPI (mpich, openmpi, intelmpi) so I can now reproduce what you did before?
> 
> (maybe this should have been my question... )

  Eric,

   Your problem is a result of Jed's constant bitching that we always list many libraries either redundantly or unnecessarily when we create our massive link line that allowed you this flexibility of linking with the "wrong" compiler. I changed the tests to not generate some redundant and unnecessary libraries to make Jed slightly less bitchy but missed the C++ one because no one uses it (except you). It would not be trivial for me to add the test because it would fail on any newer MPI that does not provide C++ bindings.

   Jed,

      Do all MPI's still provide the C++ bindings and will they forever so I can just directly put in an MPI C++ function call in the test? 

   Barry

> 
> Thanks again,
> 
> Eric
> 
> Le 18-02-10 à 12:37, Smith, Barry F. a écrit :
>>   Eric,
>> 
>>    We need the entire link line.
>> 
>>    What linker are you using C, or C++?  This is important.
>> 
>>    Do you have dependencies on MPI C++ symbols? In other words, are you using C++ MPI bindings?
>> 
>>     I cannot explain why all the libraries you listed would disappear from PETSC_WITH_EXTERNAL_LIB but we did recently make a change to manual pass under some circumstances less system (including MPI libraries) explicitly since they are already usually passed by the linker. This may be causing your difficulties.
>> 
>>     Eagerly awaiting your reply.
>> 
>>      Barry
>> 
>> 
>>> On Feb 10, 2018, at 11:29 AM, Éric Chamberland <Eric.Chamberland at giref.ulaval.ca> wrote:
>>> 
>>> Hi Matthew,
>>> 
>>> Yes, I heard that the MPI C++ API has been deprecated.
>>> 
>>> Yes, the mpi_cxx is now missing.  Our link line is formed mainly with PETSC_WITH_EXTERNAL_LIB variable that is now:
>>> 
>>> PETSC_WITH_EXTERNAL_LIB = -L/opt/petsc-master_debug/lib -Wl,-rpath,/opt/petsc-master_debug/lib -L/opt/petsc-master_debug/lib -L/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64 -Wl,-rpath,/opt/openmpi-1.10.2/lib -L/opt/openmpi-1.10.2/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/compiler/lib/intel64 -L/opt/intel/composer_xe_2015.2.164/compiler/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/ipp/lib/intel64 -L/opt/intel/composer_xe_2015.2.164/ipp/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/tbb/lib/intel64/gcc4.4 -L/opt/intel/composer_xe_2015.2.164/tbb/lib/intel64/gcc4.4 -lpetsc -lsuperlu -lsuperlu_dist -lHYPRE -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64 -lml -lumfpack -lklu -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd -lsuitesparseconfig -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -lmkl_blacs_intelmpi_lp64 -liomp5 -ldl -lpthread -lparmetis -lmetis -lptesmumps -lptscotch -lptscotcherr -lesmumps -lscotch -lscotcherr -lm -lX11 -lstdc++ -ldl -lmpi_usempi -lmpi_mpifh -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lpthread -lrt -lm -lpthread -lz -lstdc++ -ldl
>>> We add some stuff to this (our own compiled libs), but nothing related to MPI or PETSc since we used to rely on PETSC_WITH_EXTERNAL_LIB for all our diffrement environments (think about different petsc version, MPI libs and compilers: clang, icc, g++) and it used to work until yesterday changes...
>>> 
>>> As you can see in the diff, this "block" of libraries have been removed from PETSC_WITH_EXTERNAL_LIB: (the -- is part of the diff output):
>>> --ldl
>>> --lmpi_cxx
>>> --lmpi
>>> --lstdc++
>>> --lm
>>> --lgcc_s
>>> --lpthread
>>> 
>>> and into this block there was the "-lmpi_cxx" that we need...
>>> 
>>> I could send you our whole line of link, but the error is into this small change introduced yesterday into master...
>>> 
>>> Thanks a lot!
>>> 
>>> Eric
>>> 
>>> Le 18-02-10 à 10:34, Matthew Knepley a écrit :
>>>> On Sat, Feb 10, 2018 at 9:42 AM, Éric Chamberland <Eric.Chamberland at giref.ulaval.ca> wrote:
>>>> Hi,
>>>> 
>>>> we used to link our c++ code with PETSc using PETSC_WITH_EXTERNAL_LIB variable defined in $PETSC_DIR/lib/petsc/conf/petscvariables and everything was fine until this night.
>>>> 
>>>> It seems some libs have vanished from this variable, see the diff here:
>>>> 
>>>>  -lptscotcherr
>>>>  -lesmumps
>>>>  -lscotch
>>>>  -lscotcherr
>>>>  -lm
>>>>  -lX11
>>>> +-lstdc++
>>>>  -ldl
>>>>  -lmpi_usempi
>>>>  -lmpi_mpifh
>>>>  -lmpi
>>>>  -lgfortran
>>>>  -lm
>>>>  -lgfortran
>>>>  -lm
>>>>  -lgcc_s
>>>>  -lquadmath
>>>>  -lpthread
>>>> --ldl
>>>> --lmpi_cxx
>>>> --lmpi
>>>> --lstdc++
>>>> --lm
>>>> --lgcc_s
>>>> --lpthread
>>>>  -lrt
>>>>  -lm
>>>>  -lpthread
>>>>  -lz
>>>> +-lstdc++
>>>>  -ldl
>>>> 
>>>> 
>>>> causing these errors at link phase for us:
>>>> 
>>>> 
>>>>  /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o: In function `MPI::Op::Init(void (*)(void const*, void*, int, MPI::Datatype const&), bool)':
>>>> 
>>>> /opt/openmpi-1.10.2/include/
>>>> openmpi/ompi/mpi/cxx/op_inln.
>>>> h:122: undefined reference to `ompi_mpi_cxx_op_intercept'
>>>> 
>>>>  /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o: In function `MPI::Intracomm::Create_graph(int, int const*, int const*, bool) const':
>>>> 
>>>> /opt/openmpi-1.10.2/include/
>>>> openmpi/ompi/mpi/cxx/
>>>> intracomm.h:25: undefined reference to `MPI::Comm::Comm()'
>>>> 
>>>>  /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o: In function `MPI::Intercomm::Merge(bool) const':
>>>> 
>>>> /opt/openmpi-1.10.2/include/
>>>> openmpi/ompi/mpi/cxx/
>>>> intracomm_inln.h:23: undefined reference to `MPI::Comm::Comm()'
>>>> 
>>>>  /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o: In function `MPI::Intracomm::Split(int, int) const':
>>>> 
>>>> /opt/openmpi-1.10.2/include/
>>>> openmpi/ompi/mpi/cxx/
>>>> intracomm_inln.h:23: undefined reference to `MPI::Comm::Comm()'
>>>> 
>>>>  /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o: In function `MPI::Intracomm::Create(MPI::Group const&) const':
>>>> 
>>>> /opt/openmpi-1.10.2/include/
>>>> openmpi/ompi/mpi/cxx/
>>>> intracomm_inln.h:23: undefined reference to `MPI::Comm::Comm()'
>>>> 
>>>>  /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o: In function `MPI::Intracomm::Clone() const':
>>>> Hi Eric,
>>>> 
>>>> These symbols are all coming from -lmpi_cxx. I would note that I believe the MPI Forum has deprecated the C++ interface, so it will
>>>> eventually go away. However, lets fix this. In the configure log, I see that mpicxx -show has that library in it, so it seems that the link
>>>> is not being done with the C++ compiler. Can you send the whole link line?
>>>> 
>>>>   Thanks,
>>>> 
>>>>      Matt
>>>>  Is this a normal and definitive change or an unwanted/unobserved bug?
>>>> 
>>>> Thanks,
>>>> 
>>>> Eric
>>>> 
>>>> ps: here are the logs:
>>>> 
>>>> this night:
>>>> 
>>>> ---------
>>>> 
>>>> http://www.giref.ulaval.ca/~cmpgiref/petsc-master-debug/2018.02.10.02h00m01s_configure.log
>>>> http://www.giref.ulaval.ca/~cmpgiref/petsc-master-debug/2018.02.10.02h00m01s_make.log
>>>> 
>>>> 
>>>> 
>>>> a day before:
>>>> ------------
>>>> 
>>>> 
>>>> http://www.giref.ulaval.ca/~cmpgiref/petsc-master-debug/2018.02.09.02h00m02s_configure.log
>>>> http://www.giref.ulaval.ca/~cmpgiref/petsc-master-debug/2018.02.09.02h00m02s_make.log
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> -- 
>>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>>> -- Norbert Wiener
>>>> 
>>>> https://www.cse.buffalo.edu/~knepley/
> 
>

More information about the petsc-dev mailing list