[petsc-dev] dealing with MPIUNi

[Satish Balay]

On Fri, 26 Feb 2010, Barry Smith wrote:

> 
> On Feb 26, 2010, at 11:22 AM, Satish Balay wrote:
> 
> > I think eliminating mpi.h, mpi.mod etc is not a good change. Its
> > likely to break users codes.  I suspect it breaks FACETS. [and perhaps
> > Flotran]
> 
>   Suspect isn't good enough. HOW does it break FACETS? It won't break pflotran
> unless they have #include "mpi.h" directly in their code

Just for this reason [prohibiting using 'mpi.h'/mpif.h from user code
- if one wants --with-mpi=0] the current change is bad


I've explained the facets/uedge model used below.  I'll check uedge
usage. I'll have to ask facets folks to check other usages..

> > 
> > With this change we will continue to have MPI symbols [esp from
> > fortran] in libpetsc.a. So this is not really clean absorbtion of
> > mpiuni into petsc. And I don't think we can avoid having these MPI
> > symbols in -lpetsc.
> 
>   Yes the fake "symbols" are in libpetsc.a but what is wrong with that? What
> is the disadvantage over having them in libmpiuni.a?
> 
> > 
> > The fact that MPI API symbols exist in libpetsc.a makes PETSc not
> > really pure plant. Its still a plant/animal - so the goal of this code
> > reorganization is still not met.
> 
>   As I said in my previous email; only Matt's plan is perfect.

My argument is - the change mode is still imperfect - but at the cost
of breaking some user codes. [so its not worth the cost].

> > 
> > [I might not have raised this issue earlier wrt merging mpiuni
> > completely into petsc - thats my omission sorry about that.]
> > 
> > 
> > 
> > I suspec the following usage in FACETS/UEDGE will break with this
> > change:
> > 
> > If you have package-A doing the same type of thing for its sequential
> > implementation - it will have its own mpi_comm_size() etc internally.
> > With this - mixing package-A with petsc-sequential will cause
> > duplicate symbol conflict.
> 
>   The fix is to not have any symbols in MPIUNI that are called MPI_xxx This is
> handled in mpiunis mpi.h with
> #if defined(MPIUNI_AVOID_MPI_NAMESPACE)
> we should just ALWAYS avoid the MPI Namespace
> 
> 
> > 
> > The way I'm currently resolving this issue is: Only one package should
> > provide [internal] MPI - the other package is compiled with
> > --with-mpi=enabled.
> > 
> > I.e PETSc compiled with MPIUNI [says MPI_ENABLED]. Package-A is now
> > compiled with MPI-ENABLED - but links with PETSc - that provides
> > MPIUNI - as MPI.
> > 
> > Also due to this - we added more stuff to MPUNI to cover all MPI
> > symbol usage from FACETS.
> > 
> > 
> > So - the previous MPIUNI implementation scheme eventhough slightly
> > inconsistant, provided good user interface - with minimal maintainance
> > overhead.. Matt's schem makes everything consistant - but with extra
> > maintainance overhead.
> > 
> 
>   We can certainly go back to the way it was if we need to. But you have not
> demonstrated that need, you've only speculated.
> Here is how it should work ok.
> 1) Several packages each use their own fake MPI, everything is fine unless two
> different mpi.h's get included but that should not happen.

One of the ways it can conflict is:

'include' package1/mpif.h'
'include mpiuni/mpif.h

and there will be duplicate variable conflicts for constants
like MPI_COMM_WORLD, and others.

I'll admit the old way of using mpiuni with other packages does not
work with all packages. [only with a select few codes that facets
folks tested with].


> 2) Several packages each use PETSc's fake MPI. To support this we only need to
> have them add -I$PETSC_DIR/include/mpiuni in their compiles

In this case - the user should be consious that he needs mpiuni from
user code - and enable another petsc build option? [or create
unportable makefile]. One more additional cost to the current
imperfect change.

So I still prefer the previous scheme.

Satish

> 
> > So I still prefer previous secheme. If that not acceptable - then I
> > guess we have to go with Matt's scheme [ slplit up mpiuni into a
> > different package, add build/make support to it - and deal with all
> > the petsc-maint that creates..]. But the current change really breaks
> > things - so we should not do this.
> > 
> > Satish
> > 
> > On Thu, 25 Feb 2010, Barry Smith wrote:
> > 
> > > 
> > > After listening to our discussion of the half-plant/half animal handling
> > > of
> > > MPIUni I have adopted the plant :-) model.
> > > 
> > > Background: All the packages in PETSc/packages and
> > > BuildSystem/config/packages
> > > have the following paradigm except MPI and BlasLapack.
> > > 
> > > 1) PETSc can be built --with-package or --with-package=0
> > > 2) If --with-package=0 then there is no PETSC_HAVE_package defined, no
> > > extra
> > > libraries to link against and no extra include paths to look in
> > > 
> > > BlasLapack breaks this paradigm in only one way! You cannot use
> > > --with-blaspack=0
> > > 
> > > MPI breaks the paradigm more completely. You can use --with-mpi=0 BUT if
> > > you
> > > use --with-mpi=0 then PETSC_HAVE_MPI is STILL defined!!!!!! There is an
> > > extra
> > > library to link against and an extra include path to look in.
> > > 
> > > The two possible solutions to resolve this perverse beast are
> > > 1) make mpiuni be a --download-mpiuni replacement for MPI, as we do
> > > --download-c-blaslapack (this would replace the current --with-mpi=0
> > > support).
> > > 2) better supporting --with-mpi=0 without breaking the paradigm
> > > 
> > > I agree with Matt that 1 is the more elegant solution, since it fits the
> > > paradigm perfectly. But having --with-mpi=0 is easier and more
> > > understandable
> > > to the user then explaining about downloading a dummy MPI.
> > > 
> > > Thus I have implemented and pushed 2). When you use --with-mpi=0
> > > 1) the macro PETSC_HAVE_MPI is not set
> > > 2) the list of include directories is not added to
> > > 3) the list of linked against libraries is not added to.
> > > 
> > > I have implemented 2) and 3) by having in petsc.h (fortran also)
> > > #if defined(PETSC_HAVE_MPI)
> > > #include "mpi.h"
> > > #else
> > > #include "mpiuni/mpi.h"
> > > #endif
> > > and putting the dummy MPI stubs always into the PETSc libraries for both
> > > single library and multiple library PETSc installs.
> > > 
> > > Note: this means one cannot have an include "mpi.h" in the uni case which
> > > bothered me initially but then Lisandro convinced me it was not a bad
> > > thing.
> > > 
> > > The actual code changes to implement this were, of course, tiny. It is not
> > > perfect (only --download-mpiuni would be perfect :-), but it is better
> > > than
> > > before.
> > > 
> > > 
> > > Sorry Matt,
> > > 
> > > 
> > > Barry
> > > 
> > > 
> > > 
> > > 
> > 
>