[petsc-dev] petsc/master: unable to link in C++ with last night PETSC_WITH_EXTERNAL_LIB variable changes

Sat Feb 10 11:51:22 CST 2018

Ok guys!

You found it (again...).  We are no using mpic++ to link!!!

We are not always using the same compiler that we used to compile 
petsc+mpi, so we do not use mpic++ to compile (excepted for intelmpi).

For example, I compile MPI+Petsc with g++, but I am working with clang++ 
and may switch to g++ for my day to day work.  This is perfectly working 
since... long long time ago (I won't reveal my age... ;) )

I understand you choice, but your PETSC_WITH_EXTERNAL_LIB variable was 
helping me to painlessly find the good linking options.

I can work to manually add it when it is relevant into our compilation 
flags...

How did you managed to extract the libs from all the different flavors 
of MPI (mpich, openmpi, intelmpi) so I can now reproduce what you did 
before?

(maybe this should have been my question... )

Thanks again,

Eric

Le 18-02-10 à 12:37, Smith, Barry F. a écrit :
>    Eric,
>
>     We need the entire link line.
>
>     What linker are you using C, or C++?  This is important.
>
>     Do you have dependencies on MPI C++ symbols? In other words, are you using C++ MPI bindings?
>
>      I cannot explain why all the libraries you listed would disappear from PETSC_WITH_EXTERNAL_LIB but we did recently make a change to manual pass under some circumstances less system (including MPI libraries) explicitly since they are already usually passed by the linker. This may be causing your difficulties.
>
>      Eagerly awaiting your reply.
>
>       Barry
>
>
>> On Feb 10, 2018, at 11:29 AM, Éric Chamberland <Eric.Chamberland at giref.ulaval.ca> wrote:
>>
>> Hi Matthew,
>>
>> Yes, I heard that the MPI C++ API has been deprecated.
>>
>> Yes, the mpi_cxx is now missing.  Our link line is formed mainly with PETSC_WITH_EXTERNAL_LIB variable that is now:
>>
>> PETSC_WITH_EXTERNAL_LIB = -L/opt/petsc-master_debug/lib -Wl,-rpath,/opt/petsc-master_debug/lib -L/opt/petsc-master_debug/lib -L/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/mkl/lib/intel64 -Wl,-rpath,/opt/openmpi-1.10.2/lib -L/opt/openmpi-1.10.2/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/compiler/lib/intel64 -L/opt/intel/composer_xe_2015.2.164/compiler/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/ipp/lib/intel64 -L/opt/intel/composer_xe_2015.2.164/ipp/lib/intel64 -Wl,-rpath,/opt/intel/composer_xe_2015.2.164/tbb/lib/intel64/gcc4.4 -L/opt/intel/composer_xe_2015.2.164/tbb/lib/intel64/gcc4.4 -lpetsc -lsuperlu -lsuperlu_dist -lHYPRE -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64 -lml -lumfpack -lklu -lcholmod -lbtf -lccolamd -lcolamd -lcamd -lamd -lsuitesparseconfig -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -lmkl_blacs_intelmpi_lp64 -liomp5 -ldl -lpthread -lparmetis -lmetis -lptesmumps -lptscotch -lptscotcherr -lesmumps -lscotch -lscotcherr -lm -lX11 -lstdc++ -ldl -lmpi_usempi -lmpi_mpifh -lmpi -lgfortran -lm -lgfortran -lm -lgcc_s -lquadmath -lpthread -lrt -lm -lpthread -lz -lstdc++ -ldl
>> We add some stuff to this (our own compiled libs), but nothing related to MPI or PETSc since we used to rely on PETSC_WITH_EXTERNAL_LIB for all our diffrement environments (think about different petsc version, MPI libs and compilers: clang, icc, g++) and it used to work until yesterday changes...
>>
>> As you can see in the diff, this "block" of libraries have been removed from PETSC_WITH_EXTERNAL_LIB: (the -- is part of the diff output):
>> --ldl
>> --lmpi_cxx
>> --lmpi
>> --lstdc++
>> --lm
>> --lgcc_s
>> --lpthread
>>
>> and into this block there was the "-lmpi_cxx" that we need...
>>
>> I could send you our whole line of link, but the error is into this small change introduced yesterday into master...
>>
>> Thanks a lot!
>>
>> Eric
>>
>> Le 18-02-10 à 10:34, Matthew Knepley a écrit :
>>> On Sat, Feb 10, 2018 at 9:42 AM, Éric Chamberland <Eric.Chamberland at giref.ulaval.ca> wrote:
>>> Hi,
>>>
>>> we used to link our c++ code with PETSc using PETSC_WITH_EXTERNAL_LIB variable defined in $PETSC_DIR/lib/petsc/conf/petscvariables and everything was fine until this night.
>>>
>>> It seems some libs have vanished from this variable, see the diff here:
>>>
>>>   -lptscotcherr
>>>   -lesmumps
>>>   -lscotch
>>>   -lscotcherr
>>>   -lm
>>>   -lX11
>>> +-lstdc++
>>>   -ldl
>>>   -lmpi_usempi
>>>   -lmpi_mpifh
>>>   -lmpi
>>>   -lgfortran
>>>   -lm
>>>   -lgfortran
>>>   -lm
>>>   -lgcc_s
>>>   -lquadmath
>>>   -lpthread
>>> --ldl
>>> --lmpi_cxx
>>> --lmpi
>>> --lstdc++
>>> --lm
>>> --lgcc_s
>>> --lpthread
>>>   -lrt
>>>   -lm
>>>   -lpthread
>>>   -lz
>>> +-lstdc++
>>>   -ldl
>>>
>>>
>>> causing these errors at link phase for us:
>>>
>>>
>>>   /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o: In function `MPI::Op::Init(void (*)(void const*, void*, int, MPI::Datatype const&), bool)':
>>>
>>> /opt/openmpi-1.10.2/include/
>>> openmpi/ompi/mpi/cxx/op_inln.
>>> h:122: undefined reference to `ompi_mpi_cxx_op_intercept'
>>>
>>>   /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o: In function `MPI::Intracomm::Create_graph(int, int const*, int const*, bool) const':
>>>
>>> /opt/openmpi-1.10.2/include/
>>> openmpi/ompi/mpi/cxx/
>>> intracomm.h:25: undefined reference to `MPI::Comm::Comm()'
>>>
>>>   /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o: In function `MPI::Intercomm::Merge(bool) const':
>>>
>>> /opt/openmpi-1.10.2/include/
>>> openmpi/ompi/mpi/cxx/
>>> intracomm_inln.h:23: undefined reference to `MPI::Comm::Comm()'
>>>
>>>   /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o: In function `MPI::Intracomm::Split(int, int) const':
>>>
>>> /opt/openmpi-1.10.2/include/
>>> openmpi/ompi/mpi/cxx/
>>> intracomm_inln.h:23: undefined reference to `MPI::Comm::Comm()'
>>>
>>>   /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o: In function `MPI::Intracomm::Create(MPI::Group const&) const':
>>>
>>> /opt/openmpi-1.10.2/include/
>>> openmpi/ompi/mpi/cxx/
>>> intracomm_inln.h:23: undefined reference to `MPI::Comm::Comm()'
>>>
>>>   /pmi/cmpbib/compilation_BIB_gcc_redhat_petsc-master_debug/COMPILE_AUTO/GIREF/obj/dev/StatistiqueMemoire.o: In function `MPI::Intracomm::Clone() const':
>>> Hi Eric,
>>>
>>> These symbols are all coming from -lmpi_cxx. I would note that I believe the MPI Forum has deprecated the C++ interface, so it will
>>> eventually go away. However, lets fix this. In the configure log, I see that mpicxx -show has that library in it, so it seems that the link
>>> is not being done with the C++ compiler. Can you send the whole link line?
>>>
>>>    Thanks,
>>>
>>>       Matt
>>>   
>>> Is this a normal and definitive change or an unwanted/unobserved bug?
>>>
>>> Thanks,
>>>
>>> Eric
>>>
>>> ps: here are the logs:
>>>
>>> this night:
>>>
>>> ---------
>>>
>>> http://www.giref.ulaval.ca/~cmpgiref/petsc-master-debug/2018.02.10.02h00m01s_configure.log
>>> http://www.giref.ulaval.ca/~cmpgiref/petsc-master-debug/2018.02.10.02h00m01s_make.log
>>>
>>>
>>>
>>> a day before:
>>> ------------
>>>
>>>
>>> http://www.giref.ulaval.ca/~cmpgiref/petsc-master-debug/2018.02.09.02h00m02s_configure.log
>>> http://www.giref.ulaval.ca/~cmpgiref/petsc-master-debug/2018.02.09.02h00m02s_make.log
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> -- 
>>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/