[petsc-dev] PETSC_COMM_WORLD not a PETSc communicator

[Junchao Zhang]

Barry,
  Thanks for the PetscRegisterFinalize() suggestion. I made an MR at
https://gitlab.com/petsc/petsc/-/merge_requests/4238
   In rare cases, if I do need to duplicate communicators, I now free them
through PetscRegisterFinalize().

--Junchao Zhang


On Sun, Aug 15, 2021 at 12:50 PM Barry Smith <bsmith at petsc.dev> wrote:

>
>    Junchao,
>
>     Thanks for checking this.
>
>     Could you use PetscRegisterFinalize()?
>
>  Barry
>
>
> On Aug 14, 2021, at 10:46 PM, Junchao Zhang <junchao.zhang at gmail.com>
> wrote:
>
>
>
> On Thu, Aug 12, 2021 at 11:22 AM Barry Smith <bsmith at petsc.dev> wrote:
>
>>
>>  User visible communicators generally do not have a keyval attached.
>> Rather the keyval is attached to the inner communicator; because we don't
>> want both PETSc and the user doing MPI operations on the same communicator
>> (to prevent mixing up tags).
>>
>>   I think PetscShmCommGet() is wrong. I think it should not call
>> MPI_Comm_get_attr(globcomm,Petsc_Counter_keyval,&counter,&flg); but should
>> call PetscCommDuplicate() and use that communicator to stash the pshmcomm;
>> then you would not have the problem you are having.
>>
> Barry, I think it over and find the problem is:  if I PetscCommDuplicate()
> the outer comm in PetscShmCommGet(), then I can not find a place to destroy
> the inner communicator (note usually petsc inner communciators are
> destroyed with petsc objects. Doing what you said breaks the rule)
>
> PetscShmCommGet() was designed to help doing OpenMP multithreading on a
> communicator that some petsc objects live in. So requiring the input
> communicator to be petsc comm is not totally nonsense.
>
> I tried another approach: Don't check whether the input comm in
> PetscShmCommGet(globcomm,&pshmcomm) is an outer comm or an inner comm. We
> instead check its Petsc_ShmComm_keyval. If it does not have one, we just
> create one for it (along with a new shared memory communicator)
> With that, one is able to call PetscShmCommGet(PETSC_COMM_WORLD, ...).
> The problem is we attached an attribute to PETSC_COMM_WORLD. It is deleted
> inside MPI_Finalize().   PETSc -malloc_dump complains of unfreed memory
> (since I used PetscMalloc inside PetscShmCommGet). I could
> bypass PetscMalloc and directly use malloc() to avoid this situation. Is
> it worthy?
>
>
>>
>>
>> Barry
>>
>>
>>
>> > On Aug 12, 2021, at 11:05 AM, Pierre Jolivet <pierre at joliv.et> wrote:
>> >
>> > Hello,
>> > Is there a specific reason why PETSC_COMM_WORLD is not a PETSc
>> communicator, i.e., has no Petsc_Counter_keyval attached?
>> > ierr =
>> PetscOmpCtrlCreate(PETSC_COMM_WORLD,nthreads,&ctrl);CHKERRQ(ierr);
>> > yields
>> > [0]PETSC ERROR: Bad MPI communicator supplied must be a PETSc
>> communicator
>> > [0]PETSC ERROR: #1 PetscShmCommGet() at src/sys/utils/mpishm.c:60
>> > [0]PETSC ERROR: #2 PetscOmpCtrlCreate() at src/sys/utils/mpishm.c:340
>> >
>> > Thanks,
>> > Pierre
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20210816/f2fd4e50/attachment.html>