[petsc-dev] dealing with MPIUNi

[Barry Smith]

On Feb 26, 2010, at 11:22 AM, Satish Balay wrote:

> I think eliminating mpi.h, mpi.mod etc is not a good change. Its
> likely to break users codes.  I suspect it breaks FACETS. [and perhaps
> Flotran]

    Suspect isn't good enough. HOW does it break FACETS? It won't  
break pflotran unless they have #include "mpi.h" directly in their code
>
> With this change we will continue to have MPI symbols [esp from
> fortran] in libpetsc.a. So this is not really clean absorbtion of
> mpiuni into petsc. And I don't think we can avoid having these MPI
> symbols in -lpetsc.

    Yes the fake "symbols" are in libpetsc.a but what is wrong with  
that? What is the disadvantage over having them in libmpiuni.a?

>
> The fact that MPI API symbols exist in libpetsc.a makes PETSc not
> really pure plant. Its still a plant/animal - so the goal of this code
> reorganization is still not met.

    As I said in my previous email; only Matt's plan is perfect.
>
> [I might not have raised this issue earlier wrt merging mpiuni
> completely into petsc - thats my omission sorry about that.]
>
>
>
> I suspec the following usage in FACETS/UEDGE will break with this
> change:
>
> If you have package-A doing the same type of thing for its sequential
> implementation - it will have its own mpi_comm_size() etc internally.
> With this - mixing package-A with petsc-sequential will cause
> duplicate symbol conflict.

    The fix is to not have any symbols in MPIUNI that are called  
MPI_xxx This is handled in mpiunis mpi.h with
#if defined(MPIUNI_AVOID_MPI_NAMESPACE)
we should just ALWAYS avoid the MPI Namespace


>
> The way I'm currently resolving this issue is: Only one package should
> provide [internal] MPI - the other package is compiled with
> --with-mpi=enabled.
>
> I.e PETSc compiled with MPIUNI [says MPI_ENABLED]. Package-A is now
> compiled with MPI-ENABLED - but links with PETSc - that provides
> MPIUNI - as MPI.
>
> Also due to this - we added more stuff to MPUNI to cover all MPI
> symbol usage from FACETS.
>
>
> So - the previous MPIUNI implementation scheme eventhough slightly
> inconsistant, provided good user interface - with minimal maintainance
> overhead.. Matt's schem makes everything consistant - but with extra
> maintainance overhead.
>

    We can certainly go back to the way it was if we need to. But you  
have not demonstrated that need, you've only speculated.
Here is how it should work ok.
1) Several packages each use their own fake MPI, everything is fine  
unless two different mpi.h's get included but that should not happen.
2) Several packages each use PETSc's fake MPI. To support this we only  
need to have them add -I$PETSC_DIR/include/mpiuni in their compiles


> So I still prefer previous secheme. If that not acceptable - then I
> guess we have to go with Matt's scheme [ slplit up mpiuni into a
> different package, add build/make support to it - and deal with all
> the petsc-maint that creates..]. But the current change really breaks
> things - so we should not do this.
>
> Satish
>
> On Thu, 25 Feb 2010, Barry Smith wrote:
>
>>
>> After listening to our discussion of the half-plant/half animal  
>> handling of
>> MPIUni I have adopted the plant :-) model.
>>
>> Background: All the packages in PETSc/packages and BuildSystem/ 
>> config/packages
>> have the following paradigm except MPI and BlasLapack.
>>
>> 1) PETSc can be built --with-package or --with-package=0
>> 2) If --with-package=0 then there is no PETSC_HAVE_package defined,  
>> no extra
>> libraries to link against and no extra include paths to look in
>>
>> BlasLapack breaks this paradigm in only one way! You cannot use
>> --with-blaspack=0
>>
>> MPI breaks the paradigm more completely. You can use --with-mpi=0  
>> BUT if you
>> use --with-mpi=0 then PETSC_HAVE_MPI is STILL defined!!!!!! There  
>> is an extra
>> library to link against and an extra include path to look in.
>>
>> The two possible solutions to resolve this perverse beast are
>> 1) make mpiuni be a --download-mpiuni replacement for MPI, as we do
>> --download-c-blaslapack (this would replace the current --with- 
>> mpi=0 support).
>> 2) better supporting --with-mpi=0 without breaking the paradigm
>>
>> I agree with Matt that 1 is the more elegant solution, since it  
>> fits the
>> paradigm perfectly. But having --with-mpi=0 is easier and more  
>> understandable
>> to the user then explaining about downloading a dummy MPI.
>>
>> Thus I have implemented and pushed 2). When you use --with-mpi=0
>> 1) the macro PETSC_HAVE_MPI is not set
>> 2) the list of include directories is not added to
>> 3) the list of linked against libraries is not added to.
>>
>> I have implemented 2) and 3) by having in petsc.h (fortran also)
>> #if defined(PETSC_HAVE_MPI)
>> #include "mpi.h"
>> #else
>> #include "mpiuni/mpi.h"
>> #endif
>> and putting the dummy MPI stubs always into the PETSc libraries for  
>> both
>> single library and multiple library PETSc installs.
>>
>> Note: this means one cannot have an include "mpi.h" in the uni case  
>> which
>> bothered me initially but then Lisandro convinced me it was not a  
>> bad thing.
>>
>> The actual code changes to implement this were, of course, tiny. It  
>> is not
>> perfect (only --download-mpiuni would be perfect :-), but it is  
>> better than
>> before.
>>
>>
>> Sorry Matt,
>>
>>
>>  Barry
>>
>>
>>
>>
>