[petsc-dev] dealing with MPIUNi

[Barry Smith]

   One thing that really pissed me off with the old model as that  
PETSC_HAVE_MPI was defined even though there was no real MPI. When you  
can't trust your flags what can you trust?


   Barry

On Feb 26, 2010, at 12:46 PM, Satish Balay wrote:

> On Fri, 26 Feb 2010, Barry Smith wrote:
>
>>
>> On Feb 26, 2010, at 11:22 AM, Satish Balay wrote:
>>
>>> I think eliminating mpi.h, mpi.mod etc is not a good change. Its
>>> likely to break users codes.  I suspect it breaks FACETS. [and  
>>> perhaps
>>> Flotran]
>>
>>  Suspect isn't good enough. HOW does it break FACETS? It won't  
>> break pflotran
>> unless they have #include "mpi.h" directly in their code
>
> Just for this reason [prohibiting using 'mpi.h'/mpif.h from user code
> - if one wants --with-mpi=0] the current change is bad
>
>
> I've explained the facets/uedge model used below.  I'll check uedge
> usage. I'll have to ask facets folks to check other usages..
>
>>>
>>> With this change we will continue to have MPI symbols [esp from
>>> fortran] in libpetsc.a. So this is not really clean absorbtion of
>>> mpiuni into petsc. And I don't think we can avoid having these MPI
>>> symbols in -lpetsc.
>>
>>  Yes the fake "symbols" are in libpetsc.a but what is wrong with  
>> that? What
>> is the disadvantage over having them in libmpiuni.a?
>>
>>>
>>> The fact that MPI API symbols exist in libpetsc.a makes PETSc not
>>> really pure plant. Its still a plant/animal - so the goal of this  
>>> code
>>> reorganization is still not met.
>>
>>  As I said in my previous email; only Matt's plan is perfect.
>
> My argument is - the change mode is still imperfect - but at the cost
> of breaking some user codes. [so its not worth the cost].
>
>>>
>>> [I might not have raised this issue earlier wrt merging mpiuni
>>> completely into petsc - thats my omission sorry about that.]
>>>
>>>
>>>
>>> I suspec the following usage in FACETS/UEDGE will break with this
>>> change:
>>>
>>> If you have package-A doing the same type of thing for its  
>>> sequential
>>> implementation - it will have its own mpi_comm_size() etc  
>>> internally.
>>> With this - mixing package-A with petsc-sequential will cause
>>> duplicate symbol conflict.
>>
>>  The fix is to not have any symbols in MPIUNI that are called  
>> MPI_xxx This is
>> handled in mpiunis mpi.h with
>> #if defined(MPIUNI_AVOID_MPI_NAMESPACE)
>> we should just ALWAYS avoid the MPI Namespace
>>
>>
>>>
>>> The way I'm currently resolving this issue is: Only one package  
>>> should
>>> provide [internal] MPI - the other package is compiled with
>>> --with-mpi=enabled.
>>>
>>> I.e PETSc compiled with MPIUNI [says MPI_ENABLED]. Package-A is now
>>> compiled with MPI-ENABLED - but links with PETSc - that provides
>>> MPIUNI - as MPI.
>>>
>>> Also due to this - we added more stuff to MPUNI to cover all MPI
>>> symbol usage from FACETS.
>>>
>>>
>>> So - the previous MPIUNI implementation scheme eventhough slightly
>>> inconsistant, provided good user interface - with minimal  
>>> maintainance
>>> overhead.. Matt's schem makes everything consistant - but with extra
>>> maintainance overhead.
>>>
>>
>>  We can certainly go back to the way it was if we need to. But you  
>> have not
>> demonstrated that need, you've only speculated.
>> Here is how it should work ok.
>> 1) Several packages each use their own fake MPI, everything is fine  
>> unless two
>> different mpi.h's get included but that should not happen.
>
> One of the ways it can conflict is:
>
> 'include' package1/mpif.h'
> 'include mpiuni/mpif.h
>
> and there will be duplicate variable conflicts for constants
> like MPI_COMM_WORLD, and others.
>
> I'll admit the old way of using mpiuni with other packages does not
> work with all packages. [only with a select few codes that facets
> folks tested with].
>
>
>> 2) Several packages each use PETSc's fake MPI. To support this we  
>> only need to
>> have them add -I$PETSC_DIR/include/mpiuni in their compiles
>
> In this case - the user should be consious that he needs mpiuni from
> user code - and enable another petsc build option? [or create
> unportable makefile]. One more additional cost to the current
> imperfect change.
>
> So I still prefer the previous scheme.
>
> Satish
>
>>
>>> So I still prefer previous secheme. If that not acceptable - then I
>>> guess we have to go with Matt's scheme [ slplit up mpiuni into a
>>> different package, add build/make support to it - and deal with all
>>> the petsc-maint that creates..]. But the current change really  
>>> breaks
>>> things - so we should not do this.
>>>
>>> Satish
>>>
>>> On Thu, 25 Feb 2010, Barry Smith wrote:
>>>
>>>>
>>>> After listening to our discussion of the half-plant/half animal  
>>>> handling
>>>> of
>>>> MPIUni I have adopted the plant :-) model.
>>>>
>>>> Background: All the packages in PETSc/packages and
>>>> BuildSystem/config/packages
>>>> have the following paradigm except MPI and BlasLapack.
>>>>
>>>> 1) PETSc can be built --with-package or --with-package=0
>>>> 2) If --with-package=0 then there is no PETSC_HAVE_package  
>>>> defined, no
>>>> extra
>>>> libraries to link against and no extra include paths to look in
>>>>
>>>> BlasLapack breaks this paradigm in only one way! You cannot use
>>>> --with-blaspack=0
>>>>
>>>> MPI breaks the paradigm more completely. You can use --with-mpi=0  
>>>> BUT if
>>>> you
>>>> use --with-mpi=0 then PETSC_HAVE_MPI is STILL defined!!!!!! There  
>>>> is an
>>>> extra
>>>> library to link against and an extra include path to look in.
>>>>
>>>> The two possible solutions to resolve this perverse beast are
>>>> 1) make mpiuni be a --download-mpiuni replacement for MPI, as we do
>>>> --download-c-blaslapack (this would replace the current --with- 
>>>> mpi=0
>>>> support).
>>>> 2) better supporting --with-mpi=0 without breaking the paradigm
>>>>
>>>> I agree with Matt that 1 is the more elegant solution, since it  
>>>> fits the
>>>> paradigm perfectly. But having --with-mpi=0 is easier and more
>>>> understandable
>>>> to the user then explaining about downloading a dummy MPI.
>>>>
>>>> Thus I have implemented and pushed 2). When you use --with-mpi=0
>>>> 1) the macro PETSC_HAVE_MPI is not set
>>>> 2) the list of include directories is not added to
>>>> 3) the list of linked against libraries is not added to.
>>>>
>>>> I have implemented 2) and 3) by having in petsc.h (fortran also)
>>>> #if defined(PETSC_HAVE_MPI)
>>>> #include "mpi.h"
>>>> #else
>>>> #include "mpiuni/mpi.h"
>>>> #endif
>>>> and putting the dummy MPI stubs always into the PETSc libraries  
>>>> for both
>>>> single library and multiple library PETSc installs.
>>>>
>>>> Note: this means one cannot have an include "mpi.h" in the uni  
>>>> case which
>>>> bothered me initially but then Lisandro convinced me it was not a  
>>>> bad
>>>> thing.
>>>>
>>>> The actual code changes to implement this were, of course, tiny.  
>>>> It is not
>>>> perfect (only --download-mpiuni would be perfect :-), but it is  
>>>> better
>>>> than
>>>> before.
>>>>
>>>>
>>>> Sorry Matt,
>>>>
>>>>
>>>> Barry
>>>>
>>>>
>>>>
>>>>
>>>
>>
>