[petsc-dev] PETSC_COMM_WORLD not a PETSc communicator

[Junchao Zhang]

On Thu, Aug 12, 2021 at 11:22 AM Barry Smith <bsmith at petsc.dev> wrote:

>
>  User visible communicators generally do not have a keyval attached.
> Rather the keyval is attached to the inner communicator; because we don't
> want both PETSc and the user doing MPI operations on the same communicator
> (to prevent mixing up tags).
>
>   I think PetscShmCommGet() is wrong. I think it should not call
> MPI_Comm_get_attr(globcomm,Petsc_Counter_keyval,&counter,&flg); but should
> call PetscCommDuplicate() and use that communicator to stash the pshmcomm;
> then you would not have the problem you are having.
>
Barry, I think it over and find the problem is:  if I PetscCommDuplicate()
the outer comm in PetscShmCommGet(), then I can not find a place to destroy
the inner communicator (note usually petsc inner communciators are
destroyed with petsc objects. Doing what you said breaks the rule)

PetscShmCommGet() was designed to help doing OpenMP multithreading on a
communicator that some petsc objects live in. So requiring the input
communicator to be petsc comm is not totally nonsense.

I tried another approach: Don't check whether the input comm in
PetscShmCommGet(globcomm,&pshmcomm) is an outer comm or an inner comm. We
instead check its Petsc_ShmComm_keyval. If it does not have one, we just
create one for it (along with a new shared memory communicator)
With that, one is able to call PetscShmCommGet(PETSC_COMM_WORLD, ...).  The
problem is we attached an attribute to PETSC_COMM_WORLD. It is deleted
inside MPI_Finalize().   PETSc -malloc_dump complains of unfreed memory
(since I used PetscMalloc inside PetscShmCommGet). I could
bypass PetscMalloc and directly use malloc() to avoid this situation. Is it
worthy?


>
>
> Barry
>
>
>
> > On Aug 12, 2021, at 11:05 AM, Pierre Jolivet <pierre at joliv.et> wrote:
> >
> > Hello,
> > Is there a specific reason why PETSC_COMM_WORLD is not a PETSc
> communicator, i.e., has no Petsc_Counter_keyval attached?
> > ierr = PetscOmpCtrlCreate(PETSC_COMM_WORLD,nthreads,&ctrl);CHKERRQ(ierr);
> > yields
> > [0]PETSC ERROR: Bad MPI communicator supplied must be a PETSc
> communicator
> > [0]PETSC ERROR: #1 PetscShmCommGet() at src/sys/utils/mpishm.c:60
> > [0]PETSC ERROR: #2 PetscOmpCtrlCreate() at src/sys/utils/mpishm.c:340
> >
> > Thanks,
> > Pierre
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20210814/d6b9e3b6/attachment.html>