<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 3, 2018 at 1:17 AM, Smith, Barry F. <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
Each external package definitely needs its own duplicated communicator; cannot share between packages.<br>
<br>
The only problem with the dups below is if they are in a loop and get called many times.<br></blockquote><div><br><br></div><div>The "standard test" that has this issue actually has 1K fields. MOOSE creates its own field-split preconditioner (not based on the PETSc fieldsplit), and each filed is associated with one PC HYPRE. If PETSc duplicates communicators, we should easily reach the limit 2048.<br><br></div><div>I also want to confirm what extra communicators are introduced in the bad commit. <br></div><div><br><br></div><div>Fande,<br></div><div><br> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
To debug the hypre/duplication issue in MOOSE I would run in the debugger with a break point in MPI_Comm_dup() and see<br>
who keeps calling it an unreasonable amount of times. (My guess is this is a new "feature" in hypre that they will need to fix but only debugging will tell)<br>
<span class="HOEnZb"><font color="#888888"><br>
Barry<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
<br>
> On Apr 2, 2018, at 7:44 PM, Balay, Satish <<a href="mailto:balay@mcs.anl.gov">balay@mcs.anl.gov</a>> wrote:<br>
><br>
> We do a MPI_Comm_dup() for objects related to externalpackages.<br>
><br>
> Looks like we added a new mat type MATHYPRE - in 3.8 that PCHYPRE is<br>
> using. Previously there was one MPI_Comm_dup() PCHYPRE - now I think<br>
> is one more for MATHYPRE - so more calls to MPI_Comm_dup in 3.8 vs 3.7<br>
><br>
> src/dm/impls/da/hypre/mhyp.c: ierr = MPI_Comm_dup(PetscObjectComm((<wbr>PetscObject)B),&(ex->hcomm));<wbr>CHKERRQ(ierr);<br>
> src/dm/impls/da/hypre/mhyp.c: ierr = MPI_Comm_dup(PetscObjectComm((<wbr>PetscObject)B),&(ex->hcomm));<wbr>CHKERRQ(ierr);<br>
> src/dm/impls/swarm/data_ex.c: ierr = MPI_Comm_dup(comm,&d->comm);<wbr>CHKERRQ(ierr);<br>
> src/ksp/pc/impls/hypre/hypre.<wbr>c: ierr = MPI_Comm_dup(PetscObjectComm((<wbr>PetscObject)pc),&(jac->comm_<wbr>hypre));CHKERRQ(ierr);<br>
> src/ksp/pc/impls/hypre/hypre.<wbr>c: ierr = MPI_Comm_dup(PetscObjectComm((<wbr>PetscObject)pc),&(ex->hcomm));<wbr>CHKERRQ(ierr);<br>
> src/ksp/pc/impls/hypre/hypre.<wbr>c: ierr = MPI_Comm_dup(PetscObjectComm((<wbr>PetscObject)pc),&(ex->hcomm));<wbr>CHKERRQ(ierr);<br>
> src/ksp/pc/impls/spai/ispai.c: ierr = MPI_Comm_dup(PetscObjectComm((<wbr>PetscObject)pc),&(ispai->comm_<wbr>spai));CHKERRQ(ierr);<br>
> src/mat/examples/tests/ex152.<wbr>c: ierr = MPI_Comm_dup(MPI_COMM_WORLD, &comm);CHKERRQ(ierr);<br>
> src/mat/impls/aij/mpi/mkl_<wbr>cpardiso/mkl_cpardiso.c: ierr = MPI_Comm_dup(PetscObjectComm((<wbr>PetscObject)A),&(mat_mkl_<wbr>cpardiso->comm_mkl_cpardiso));<wbr>CHKERRQ(ierr);<br>
> src/mat/impls/aij/mpi/mumps/<wbr>mumps.c: ierr = MPI_Comm_dup(PetscObjectComm((<wbr>PetscObject)A),&(mumps->comm_<wbr>mumps));CHKERRQ(ierr);<br>
> src/mat/impls/aij/mpi/pastix/<wbr>pastix.c: ierr = MPI_Comm_dup(PetscObjectComm((<wbr>PetscObject)A),&(lu->pastix_<wbr>comm));CHKERRQ(ierr);<br>
> src/mat/impls/aij/mpi/superlu_<wbr>dist/superlu_dist.c: ierr = MPI_Comm_dup(PetscObjectComm((<wbr>PetscObject)A),&(lu->comm_<wbr>superlu));CHKERRQ(ierr);<br>
> src/mat/impls/hypre/mhypre.c: ierr = MPI_Comm_dup(PetscObjectComm((<wbr>PetscObject)B),&hB->comm);<wbr>CHKERRQ(ierr);<br>
> src/mat/partition/impls/<wbr>pmetis/pmetis.c: ierr = MPI_Comm_dup(pcomm,&comm);<wbr>CHKERRQ(ierr);<br>
> src/sys/mpiuni/mpi.c: MPI_COMM_SELF, MPI_COMM_WORLD, and a MPI_Comm_dup() of each of these (duplicates of duplicates return the same communictor)<br>
> src/sys/mpiuni/mpi.c:int MPI_Comm_dup(MPI_Comm comm,MPI_Comm *out)<br>
> src/sys/objects/pinit.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD,&<wbr>local_comm);CHKERRQ(ierr);<br>
> src/sys/objects/pinit.c: ierr = MPI_Comm_dup(MPI_COMM_WORLD,&<wbr>local_comm);CHKERRQ(ierr);<br>
> src/sys/objects/tagm.c: ierr = MPI_Comm_dup(comm_in,comm_out)<wbr>;CHKERRQ(ierr);<br>
> src/sys/utils/mpiu.c: ierr = MPI_Comm_dup(comm,&local_comm)<wbr>;CHKERRQ(ierr);<br>
> src/ts/impls/implicit/<wbr>sundials/sundials.c: ierr = MPI_Comm_dup(PetscObjectComm((<wbr>PetscObject)ts),&(cvode->comm_<wbr>sundials));CHKERRQ(ierr);<br>
><br>
> Perhaps we need a PetscCommDuplicateExternalPkg(<wbr>) to somehow avoid these MPI_Comm_dup() calls?<br>
><br>
> Satish<br>
><br>
> On Tue, 3 Apr 2018, Smith, Barry F. wrote:<br>
><br>
>><br>
>> Are we sure this is a PETSc comm issue and not a hypre comm duplication issue<br>
>><br>
>> frame #6: 0x00000001061345d9 libpetsc.3.07.dylib`hypre_<wbr>GenerateSubComm(comm=-<wbr>1006627852, participate=<unavailable>, new_comm_ptr=<unavailable>) + 409 at gen_redcs_mat.c:531 [opt]<br>
>><br>
>> Looks like hypre is needed to generate subcomms, perhaps it generates too many?<br>
>><br>
>> Barry<br>
>><br>
>><br>
>>> On Apr 2, 2018, at 7:07 PM, Derek Gaston <<a href="mailto:friedmud@gmail.com">friedmud@gmail.com</a>> wrote:<br>
>>><br>
>>> I’m working with Fande on this and I would like to add a bit more. There are many circumstances where we aren’t working on COMM_WORLD at all (e.g. working on a sub-communicator) but PETSc was initialized using MPI_COMM_WORLD (think multi-level solves)… and we need to create arbitrarily many PETSc vecs/mats/solvers/<wbr>preconditioners and solve. We definitely can’t rely on using PETSC_COMM_WORLD to avoid triggering duplication.<br>
>>><br>
>>> Can you explain why PETSc needs to duplicate the communicator so much?<br>
>>><br>
>>> Thanks for your help in tracking this down!<br>
>>><br>
>>> Derek<br>
>>><br>
>>> On Mon, Apr 2, 2018 at 5:44 PM Kong, Fande <<a href="mailto:fande.kong@inl.gov">fande.kong@inl.gov</a>> wrote:<br>
>>> Why we do not use user-level MPI communicators directly? What are potential risks here?<br>
>>><br>
>>><br>
>>> Fande,<br>
>>><br>
>>> On Mon, Apr 2, 2018 at 5:08 PM, Satish Balay <<a href="mailto:balay@mcs.anl.gov">balay@mcs.anl.gov</a>> wrote:<br>
>>> PETSC_COMM_WORLD [via PetscCommDuplicate()] attempts to minimize calls to MPI_Comm_dup() - thus potentially avoiding such errors<br>
>>><br>
>>> <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.mcs.anl.gov_petsc_petsc-2Dcurrent_docs_manualpages_Sys_PetscCommDuplicate.html&d=DwIBAg&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=DUUt3SRGI0_JgtNaS3udV68GRkgV4ts7XKfj2opmiCY&m=jgv7gpZ3K52d_FWMgkK9yEScbLA7pkrWydFuJnYflsU&s=_zpWRcyk3kHuEHoq02NDqYExnXIohXpNnjyabYnnDjU&e=" rel="noreferrer" target="_blank">https://urldefense.proofpoint.<wbr>com/v2/url?u=http-3A__www.mcs.<wbr>anl.gov_petsc_petsc-2Dcurrent_<wbr>docs_manualpages_Sys_<wbr>PetscCommDuplicate.html&d=<wbr>DwIBAg&c=<wbr>54IZrppPQZKX9mLzcGdPfFD1hxrcB_<wbr>_aEkJFOKJFd00&r=DUUt3SRGI0_<wbr>JgtNaS3udV68GRkgV4ts7XKfj2opmi<wbr>CY&m=jgv7gpZ3K52d_<wbr>FWMgkK9yEScbLA7pkrWydFuJnYflsU<wbr>&s=_<wbr>zpWRcyk3kHuEHoq02NDqYExnXIohXp<wbr>NnjyabYnnDjU&e=</a><br>
>>><br>
>>><br>
>>> Satish<br>
>>><br>
>>> On Mon, 2 Apr 2018, Kong, Fande wrote:<br>
>>><br>
>>>> On Mon, Apr 2, 2018 at 4:23 PM, Satish Balay <<a href="mailto:balay@mcs.anl.gov">balay@mcs.anl.gov</a>> wrote:<br>
>>>><br>
>>>>> Does this 'standard test' use MPI_COMM_WORLD' to crate PETSc objects?<br>
>>>>><br>
>>>>> If so - you could try changing to PETSC_COMM_WORLD<br>
>>>>><br>
>>>><br>
>>>><br>
>>>> I do not think we are using PETSC_COMM_WORLD when creating PETSc objects.<br>
>>>> Why we can not use MPI_COMM_WORLD?<br>
>>>><br>
>>>><br>
>>>> Fande,<br>
>>>><br>
>>>><br>
>>>>><br>
>>>>> Satish<br>
>>>>><br>
>>>>><br>
>>>>> On Mon, 2 Apr 2018, Kong, Fande wrote:<br>
>>>>><br>
>>>>>> Hi All,<br>
>>>>>><br>
>>>>>> I am trying to upgrade PETSc from 3.7.6 to 3.8.3 for MOOSE and its<br>
>>>>>> applications. I have a error message for a standard test:<br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>> *preconditioners/pbp.lots_of_<wbr>variables: MPI had an<br>
>>>>>> errorpreconditioners/pbp.lots_<wbr>of_variables:<br>
>>>>>> ------------------------------<wbr>------------------<br>
>>>>> preconditioners/pbp.lots_of_<wbr>variables:<br>
>>>>>> Other MPI error, error stack:preconditioners/pbp.<wbr>lots_of_variables:<br>
>>>>>> PMPI_Comm_dup(177)............<wbr>......: MPI_Comm_dup(comm=0x84000001,<br>
>>>>>> new_comm=0x97d1068) failedpreconditioners/pbp.<wbr>lots_of_variables:<br>
>>>>>> PMPI_Comm_dup(162)............<wbr>......:<br>
>>>>>> preconditioners/pbp.lots_of_<wbr>variables:<br>
>>>>>> MPIR_Comm_dup_impl(57)........<wbr>......:<br>
>>>>>> preconditioners/pbp.lots_of_<wbr>variables:<br>
>>>>>> MPIR_Comm_copy(739)...........<wbr>......:<br>
>>>>>> preconditioners/pbp.lots_of_<wbr>variables:<br>
>>>>>> MPIR_Get_contextid_sparse_<wbr>group(614): Too many communicators (0/2048<br>
>>>>> free<br>
>>>>>> on this process; ignore_id=0)*<br>
>>>>>><br>
>>>>>><br>
>>>>>> I did "git bisect', and the following commit introduces this issue:<br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>><br>
>>>>>> *commit 49a781f5cee36db85e8d5b951eec29<wbr>f10ac13593Author: Stefano Zampini<br>
>>>>>> <<a href="mailto:stefano.zampini@gmail.com">stefano.zampini@gmail.com</a> <<a href="mailto:stefano.zampini@gmail.com">stefano.zampini@gmail.com</a>>><wbr>Date: Sat Nov 5<br>
>>>>>> 20:15:19 2016 +0300 PCHYPRE: use internal Mat of type MatHYPRE<br>
>>>>>> hpmat already stores two HYPRE vectors*<br>
>>>>>><br>
>>>>>> Before I debug line-by-line, anyone has a clue on this?<br>
>>>>>><br>
>>>>>><br>
>>>>>> Fande,<br>
>>>>>><br>
>>>>><br>
>>>>><br>
>>>><br>
>>><br>
>><br>
<br>
</div></div></blockquote></div><br></div></div>