[petsc-dev] dealing with MPIUNi

[Satish Balay]

Looks like uedge build tools look for mpiuni spearately and adds in
-Impiuni.

But I think its good to keep -Impiuni in petsc makefiles for
regular users who have mpi.h [or mpif.h] in their non-petsc soures.

Satish


On Fri, 26 Feb 2010, Barry Smith wrote:

> 
>  Satish,
> 
>   Ok I will expose the -Impiuni/ so that external packages can use the MPI uni
> directly. Please let me know if problems arise.
> 
>   Barry
> 
> On Feb 26, 2010, at 2:01 PM, Satish Balay wrote:
> 
> > I should have included facets-devel in this discussion. Will cc
> > facets-devel list now [there are likely to be e-mail bounces due to
> > this - will handle the bounces on petsc-dev]
> > 
> > The discussion is at:
> > http://lists.mcs.anl.gov/pipermail/petsc-dev/2010-February/002200.html
> > http://lists.mcs.anl.gov/pipermail/petsc-dev/2010-February/002270.html
> > 
> > The one usage I know off - is petsc+mpiuni is from uedge. I'm guessing
> > there are other usages from facets [I don't really know these other
> > usages]
> > 
> > Based on the previous discussions on facets-devel list, I was
> > advocating the following: when having sequential builds with multiple
> > packages - that have their own internal seqmpi switch [with normal mpi
> > code]:
> > 
> > - the best thing to do is always use a proper mpi impl even for
> >  sequential builds - and make every package use this impl[ with
> >  perhaps an error message when parallel run is attempted]
> > 
> > - if proper mpi must to be avoided - then one way to minise conflicts
> > is use seq-mpi from only one package - and have others use this
> > aswell. However this does not work with all packages [as there are
> > some packages that do need MPI - like hypre etc..].
> > 
> > So currently mpiuni is being used as the common seq-mpi [from uedge,
> > and perhaps other codes]
> > 
> > Wrt uedge I'm yet to check how the current changes might affect
> > it. [currently uedge buildsystem does not work with petsc-dev [due to
> > the makefile changes]. will check on this later.
> > 
> > On Fri, 26 Feb 2010, Barry Smith wrote:
> > 
> > > 
> > > If we switch back slightly to the old model in the following way will that
> > > resolve the problems with FACETS?
> > > 
> > > 1) expose the mpiuni include directory by adding the mpiuni include path
> > > to
> > > the list of include search paths
> > 
> > wrt uedge - the current buildsystem picks up everything from PETSc makefiles
> > -
> > so the above change should work.
> > 
> > > 2) DO NOT set PETSC_HAVE_MPI
> > 
> > uedge does not use this - so this change should be fine.
> > 
> > > 3) DO NOT have a separate mpiuni library (which doesn't exist with single
> > > library anyways).
> > 
> > since uedge links with petsc+mpi detected from petsc makefiles - this
> > should work..
> > 
> > > Slightly related note: is there any reason not to always #if
> > > defined(MPIUNI_AVOID_MPI_NAMESPACE) so that C mpi symbols don't appear in
> > > our
> > > libraries. Not that this is important but I still like simple clean code
> > > without a bunch of confusing if not needed ifdefs.
> > 
> > Looks like when this flag is defined - the fortran mpiuni symbols are
> > also not built.
> > 
> > satish
> > 
> > 
> > > 
> > > 
> > > Barry
> > > 
> > > On Feb 26, 2010, at 1:00 PM, Barry Smith wrote:
> > > 
> > > > 
> > > > 1)  So FACETS Fortran code includes mpif.h directly AND includes PETSc
> > > > includes in the same file? Can those extra includes just be removed from
> > > > FACETS.
> > > > 2)  Some FACES codes, or other packages that FACETS use THAT DO NOT use
> > > > PETSc can/do use PETSc's mpiuni mpif.h?
> > > > 
> > > > Are these ONLY conflict with the new MPI unimodel and FACETS or are
> > > > there
> > > > other conflicts?
> > > > 
> > > > Is there any issue with not having a separate libmpiuni.a ?
> > > > 
> > > > Barry
> > > > 
> > > > On Feb 26, 2010, at 12:46 PM, Satish Balay wrote:
> > > > 
> > > > > On Fri, 26 Feb 2010, Barry Smith wrote:
> > > > > 
> > > > > > 
> > > > > > On Feb 26, 2010, at 11:22 AM, Satish Balay wrote:
> > > > > > 
> > > > > > > I think eliminating mpi.h, mpi.mod etc is not a good change. Its
> > > > > > > likely to break users codes.  I suspect it breaks FACETS. [and
> > > > > > > perhaps
> > > > > > > Flotran]
> > > > > > 
> > > > > > Suspect isn't good enough. HOW does it break FACETS? It won't break
> > > > > > pflotran
> > > > > > unless they have #include "mpi.h" directly in their code
> > > > > 
> > > > > Just for this reason [prohibiting using 'mpi.h'/mpif.h from user code
> > > > > - if one wants --with-mpi=0] the current change is bad
> > > > > 
> > > > > 
> > > > > I've explained the facets/uedge model used below.  I'll check uedge
> > > > > usage. I'll have to ask facets folks to check other usages..
> > > > > 
> > > > > > > 
> > > > > > > With this change we will continue to have MPI symbols [esp from
> > > > > > > fortran] in libpetsc.a. So this is not really clean absorbtion of
> > > > > > > mpiuni into petsc. And I don't think we can avoid having these MPI
> > > > > > > symbols in -lpetsc.
> > > > > > 
> > > > > > Yes the fake "symbols" are in libpetsc.a but what is wrong with
> > > > > > that?
> > > > > > What
> > > > > > is the disadvantage over having them in libmpiuni.a?
> > > > > > 
> > > > > > > 
> > > > > > > The fact that MPI API symbols exist in libpetsc.a makes PETSc not
> > > > > > > really pure plant. Its still a plant/animal - so the goal of this
> > > > > > > code
> > > > > > > reorganization is still not met.
> > > > > > 
> > > > > > As I said in my previous email; only Matt's plan is perfect.
> > > > > 
> > > > > My argument is - the change mode is still imperfect - but at the cost
> > > > > of breaking some user codes. [so its not worth the cost].
> > > > > 
> > > > > > > 
> > > > > > > [I might not have raised this issue earlier wrt merging mpiuni
> > > > > > > completely into petsc - thats my omission sorry about that.]
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > I suspec the following usage in FACETS/UEDGE will break with this
> > > > > > > change:
> > > > > > > 
> > > > > > > If you have package-A doing the same type of thing for its
> > > > > > > sequential
> > > > > > > implementation - it will have its own mpi_comm_size() etc
> > > > > > > internally.
> > > > > > > With this - mixing package-A with petsc-sequential will cause
> > > > > > > duplicate symbol conflict.
> > > > > > 
> > > > > > The fix is to not have any symbols in MPIUNI that are called MPI_xxx
> > > > > > This is
> > > > > > handled in mpiunis mpi.h with
> > > > > > #if defined(MPIUNI_AVOID_MPI_NAMESPACE)
> > > > > > we should just ALWAYS avoid the MPI Namespace
> > > > > > 
> > > > > > 
> > > > > > > 
> > > > > > > The way I'm currently resolving this issue is: Only one package
> > > > > > > should
> > > > > > > provide [internal] MPI - the other package is compiled with
> > > > > > > --with-mpi=enabled.
> > > > > > > 
> > > > > > > I.e PETSc compiled with MPIUNI [says MPI_ENABLED]. Package-A is
> > > > > > > now
> > > > > > > compiled with MPI-ENABLED - but links with PETSc - that provides
> > > > > > > MPIUNI - as MPI.
> > > > > > > 
> > > > > > > Also due to this - we added more stuff to MPUNI to cover all MPI
> > > > > > > symbol usage from FACETS.
> > > > > > > 
> > > > > > > 
> > > > > > > So - the previous MPIUNI implementation scheme eventhough slightly
> > > > > > > inconsistant, provided good user interface - with minimal
> > > > > > > maintainance
> > > > > > > overhead.. Matt's schem makes everything consistant - but with
> > > > > > > extra
> > > > > > > maintainance overhead.
> > > > > > > 
> > > > > > 
> > > > > > We can certainly go back to the way it was if we need to. But you
> > > > > > have
> > > > > > not
> > > > > > demonstrated that need, you've only speculated.
> > > > > > Here is how it should work ok.
> > > > > > 1) Several packages each use their own fake MPI, everything is fine
> > > > > > unless two
> > > > > > different mpi.h's get included but that should not happen.
> > > > > 
> > > > > One of the ways it can conflict is:
> > > > > 
> > > > > 'include' package1/mpif.h'
> > > > > 'include mpiuni/mpif.h
> > > > > 
> > > > > and there will be duplicate variable conflicts for constants
> > > > > like MPI_COMM_WORLD, and others.
> > > > > 
> > > > > I'll admit the old way of using mpiuni with other packages does not
> > > > > work with all packages. [only with a select few codes that facets
> > > > > folks tested with].
> > > > > 
> > > > > 
> > > > > > 2) Several packages each use PETSc's fake MPI. To support this we
> > > > > > only
> > > > > > need to
> > > > > > have them add -I$PETSC_DIR/include/mpiuni in their compiles
> > > > > 
> > > > > In this case - the user should be consious that he needs mpiuni from
> > > > > user code - and enable another petsc build option? [or create
> > > > > unportable makefile]. One more additional cost to the current
> > > > > imperfect change.
> > > > > 
> > > > > So I still prefer the previous scheme.
> > > > > 
> > > > > Satish
> > > > > 
> > > > > > 
> > > > > > > So I still prefer previous secheme. If that not acceptable - then
> > > > > > > I
> > > > > > > guess we have to go with Matt's scheme [ slplit up mpiuni into a
> > > > > > > different package, add build/make support to it - and deal with
> > > > > > > all
> > > > > > > the petsc-maint that creates..]. But the current change really
> > > > > > > breaks
> > > > > > > things - so we should not do this.
> > > > > > > 
> > > > > > > Satish
> > > > > > > 
> > > > > > > On Thu, 25 Feb 2010, Barry Smith wrote:
> > > > > > > 
> > > > > > > > 
> > > > > > > > After listening to our discussion of the half-plant/half animal
> > > > > > > > handling
> > > > > > > > of
> > > > > > > > MPIUni I have adopted the plant :-) model.
> > > > > > > > 
> > > > > > > > Background: All the packages in PETSc/packages and
> > > > > > > > BuildSystem/config/packages
> > > > > > > > have the following paradigm except MPI and BlasLapack.
> > > > > > > > 
> > > > > > > > 1) PETSc can be built --with-package or --with-package=0
> > > > > > > > 2) If --with-package=0 then there is no PETSC_HAVE_package
> > > > > > > > defined,
> > > > > > > > no
> > > > > > > > extra
> > > > > > > > libraries to link against and no extra include paths to look in
> > > > > > > > 
> > > > > > > > BlasLapack breaks this paradigm in only one way! You cannot use
> > > > > > > > --with-blaspack=0
> > > > > > > > 
> > > > > > > > MPI breaks the paradigm more completely. You can use
> > > > > > > > --with-mpi=0
> > > > > > > > BUT if
> > > > > > > > you
> > > > > > > > use --with-mpi=0 then PETSC_HAVE_MPI is STILL defined!!!!!!
> > > > > > > > There is
> > > > > > > > an
> > > > > > > > extra
> > > > > > > > library to link against and an extra include path to look in.
> > > > > > > > 
> > > > > > > > The two possible solutions to resolve this perverse beast are
> > > > > > > > 1) make mpiuni be a --download-mpiuni replacement for MPI, as we
> > > > > > > > do
> > > > > > > > --download-c-blaslapack (this would replace the current
> > > > > > > > --with-mpi=0
> > > > > > > > support).
> > > > > > > > 2) better supporting --with-mpi=0 without breaking the paradigm
> > > > > > > > 
> > > > > > > > I agree with Matt that 1 is the more elegant solution, since it
> > > > > > > > fits
> > > > > > > > the
> > > > > > > > paradigm perfectly. But having --with-mpi=0 is easier and more
> > > > > > > > understandable
> > > > > > > > to the user then explaining about downloading a dummy MPI.
> > > > > > > > 
> > > > > > > > Thus I have implemented and pushed 2). When you use --with-mpi=0
> > > > > > > > 1) the macro PETSC_HAVE_MPI is not set
> > > > > > > > 2) the list of include directories is not added to
> > > > > > > > 3) the list of linked against libraries is not added to.
> > > > > > > > 
> > > > > > > > I have implemented 2) and 3) by having in petsc.h (fortran also)
> > > > > > > > #if defined(PETSC_HAVE_MPI)
> > > > > > > > #include "mpi.h"
> > > > > > > > #else
> > > > > > > > #include "mpiuni/mpi.h"
> > > > > > > > #endif
> > > > > > > > and putting the dummy MPI stubs always into the PETSc libraries
> > > > > > > > for
> > > > > > > > both
> > > > > > > > single library and multiple library PETSc installs.
> > > > > > > > 
> > > > > > > > Note: this means one cannot have an include "mpi.h" in the uni
> > > > > > > > case
> > > > > > > > which
> > > > > > > > bothered me initially but then Lisandro convinced me it was not
> > > > > > > > a
> > > > > > > > bad
> > > > > > > > thing.
> > > > > > > > 
> > > > > > > > The actual code changes to implement this were, of course, tiny.
> > > > > > > > It
> > > > > > > > is not
> > > > > > > > perfect (only --download-mpiuni would be perfect :-), but it is
> > > > > > > > better
> > > > > > > > than
> > > > > > > > before.
> > > > > > > > 
> > > > > > > > 
> > > > > > > > Sorry Matt,
> > > > > > > > 
> > > > > > > > 
> > > > > > > > Barry
> > > > > > > > 
> > > > > > > > 
> > > > > > > > 
> > > > > > > > 
> > > > > > > 
> > > > > > 
> > > > > 
> > > > 
> > > 
> > 
>