<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">What about<div class=""><br class=""></div><div class="">PetscCommGetPkgComm(MPI_Comm comm ,const char* package, MPI_Comm* pkgcomm)<div class=""><br class=""></div><div class="">with a key for each of the external packages PETSc can use?</div><div class=""><br class=""></div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Apr 3, 2018, at 10:56 PM, Kong, Fande <<a href="mailto:fande.kong@inl.gov" class="">fande.kong@inl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class=""><div class="">I think we could add an inner comm for external package. If the same comm is passed in again, we just retrieve the same communicator, instead of <span class="gmail-im">MPI_Comm_dup()</span>, for that external package (at least HYPRE team claimed this will be fine). I did not see any issue with this idea so far. <br class=""><br class=""></div>I might be missing something here <br class=""><br class=""><br class=""></div>Fande,<br class=""></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Tue, Apr 3, 2018 at 1:45 PM, Satish Balay <span dir="ltr" class=""><<a href="mailto:balay@mcs.anl.gov" target="_blank" class="">balay@mcs.anl.gov</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Tue, 3 Apr 2018, Smith, Barry F. wrote:<br class="">
<br class="">
><br class="">
><br class="">
> > On Apr 3, 2018, at 11:59 AM, Balay, Satish <<a href="mailto:balay@mcs.anl.gov" class="">balay@mcs.anl.gov</a>> wrote:<br class="">
> ><br class="">
> > On Tue, 3 Apr 2018, Smith, Barry F. wrote:<br class="">
> ><br class="">
> >> Note that PETSc does one MPI_Comm_dup() for each hypre matrix. Internally hypre does at least one MPI_Comm_create() per hypre boomerAMG solver. So even if PETSc does not do the MPI_Comm_dup() you will still be limited due to hypre's MPI_Comm_create.<br class="">
> >><br class="">
> >> I will compose an email to hypre cc:ing everyone to get information from them.<br class="">
> ><br class="">
> > Actually I don't see any calls to MPI_Comm_dup() in hypre sources [there are stubs for it for non-mpi build]<br class="">
> ><br class="">
> > There was that call to MPI_Comm_create() in the stack trace [via hypre_BoomerAMGSetup]<br class="">
><br class="">
> This is what I said. The MPI_Comm_create() is called for each solver and hence uses a slot for each solver.<br class="">
<br class="">
</span>Ops sorry - misread the text..<br class="">
<span class="HOEnZb"><font color="#888888" class=""><br class="">
Satish<br class="">
</font></span></blockquote></div><br class=""></div>
</div></blockquote></div><br class=""></div></div></body></html>