[petsc-dev] MPI_IN_PLACE

[Barry Smith]

On May 19, 2010, at 12:48 AM, Jed Brown wrote:

> On Tue, 18 May 2010 21:03:53 -0500, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> 
>>  Jed,
>> 
>>    We've only been using MPI_IN_PLACE within 
>> #if defined(PETSC_HAVE_MPI_IN_PLACE) 
>> to make PETSc remain portable to older MPI implementations. You have violated this in ex48.c
>> 
>>   Should we change PETSc to only use MPI that supports MPI_IN_PLACE or change ex48.c?
> 
> Somehow I thought it was included in an early MPI-1.x revision, but I
> don't see it in my copy of 1.3 so it must be MPI-2 only.  So it looks
> like I should change ex48.c unless MPI_IN_PLACE is consistently
> supported across all "real" implementations (anything but MPIUNI).
> Presumably if you asked Bill, he would point out that MPI-2 was
> finalized 13 years ago and everyone should be using it by now.


  In general any MPI routine that is post 1.1 requires an ifdef check and should whenever possible have suitable code that works without the MPI routine. Yes, this is not documented, I will put it in developers.tex right now. And yes 1.1 is ancient but there is no really harm in remaining portable to such an old MPI.

   I have removed the unneeded "out of PETSc style guide" use of MPI_IN_PLACE in the examples.

> 
> A related issue is calling PetscGlobal(Max|Min|Sum) with the same
> address for local and result.  The standard expressly disallows this,
> but the docs never say this and I think a lot of users do it anyway
> (including Dave's example, for instance).  Since PetscGlobal* only ever
> reduces a single value, I would be in favor of just passing it by value
> so that it never aliases (instead of forcing the user to create a
> temporary variable which is ugly).

   Thanks for pointing this out. This is wrong, all this code SHOULD NOT BE USING PetscGlobal(MaxMinSum) AT ALL. These functions are reserved for use ONLY with code that will be run through Adic. I have updated their manual pages to clearly indicate this and fixed all the offending code to properly not use PetscGlobalSum. Also changed the calling sequence of the PetscGlobalXXX() to use a const for the first argument.

   I have added the following to the developers.tex style guide

There shall be no PETSc routines introduced that provide essentially the same functionality as an available MPI routine. For example, one should not write a routine PetscGlobalSum() that takes a scalar value and performs an MPI_Allreduce() on it. One should use MPI_Allreduce() directly in the code. (Note: PetscGlobalSum() does exist but is ONLY to be used in code that is to be run through ADIC (since ADIC cannot handle MPI calls)). This has always been implicit in PETSc design but never written down.

   Barry


> 
> Jed