<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>My previous message may sound misleading. This problem happens
despite the fact that the old matrices are destroyed.</p>
<p>Feimi<br>
</p>
<div class="moz-cite-prefix">On 8/18/21 4:31 PM, Feimi Yu wrote:<br>
</div>
<blockquote type="cite"
cite="mid:71f57283-93b4-2470-7f34-0b2af7309e00@rpi.edu">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<p>Hi Junchao,</p>
<p>Thank you for the suggestion! I'm using the deal.ii wrapper
dealii::PETScWrappers::PreconditionBase to handle the PETSc
preconditioners, and the wrappers does the destroy when the
preconditioner is reinitialized or gets out of scope. I just
double-checked, this is called to make sure the old matrices are
destroyed:<br>
</p>
<p> void<br>
PreconditionBase::clear()<br>
{<br>
matrix = nullptr;<br>
<br>
if (pc != nullptr)<br>
{<br>
PetscErrorCode ierr = PCDestroy(&pc);<br>
pc = nullptr;<br>
AssertThrow(ierr == 0, ExcPETScError(ierr));<br>
}<br>
}</p>
<p>Thanks!</p>
<p>Feimi<br>
</p>
<div class="moz-cite-prefix">On 8/18/21 4:23 PM, Junchao Zhang
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CA+MQGp_EVwiHF1zaRF5o4qvWpk4yByq-Qrrx3OO2hrrKHnSvZA@mail.gmail.com">
<meta http-equiv="content-type" content="text/html;
charset=UTF-8">
<div dir="ltr">
<div dir="ltr"><br>
<br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Aug 18, 2021 at
12:52 PM Feimi Yu <<a href="mailto:yuf2@rpi.edu"
target="_blank" moz-do-not-send="true">yuf2@rpi.edu</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<p>Hi,</p>
<p>I was trying to run a simulation with a PETSc-wrapped
Hypre preconditioner, and encountered this problem:</p>
<p>[dcs122:133012] Out of resources: all 4095
communicator IDs have been used.<br>
[19]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------<br>
[19]PETSC ERROR: General MPI error <br>
[19]PETSC ERROR: MPI error 17 MPI_ERR_INTERN: internal
error<br>
[19]PETSC ERROR: See <a
href="https://www.mcs.anl.gov/petsc/documentation/faq.html"
rel="noreferrer" target="_blank"
moz-do-not-send="true">https://www.mcs.anl.gov/petsc/documentation/faq.html</a>
for trouble shooting.<br>
[19]PETSC ERROR: Petsc Release Version 3.15.2, unknown
<br>
[19]PETSC ERROR: ./main on a arch-linux-c-opt named
dcs122 by CFSIfmyu Wed Aug 11 19:51:47 2021 <br>
[19]PETSC ERROR: [dcs122:133010] Out of resources: all
4095 communicator IDs have been used.<br>
[18]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------<br>
[18]PETSC ERROR: General MPI error <br>
[18]PETSC ERROR: MPI error 17 MPI_ERR_INTERN: internal
error<br>
[18]PETSC ERROR: See <a
href="https://www.mcs.anl.gov/petsc/documentation/faq.html"
rel="noreferrer" target="_blank"
moz-do-not-send="true">https://www.mcs.anl.gov/petsc/documentation/faq.html</a>
for trouble shooting.<br>
[18]PETSC ERROR: Petsc Release Version 3.15.2, unknown
<br>
[18]PETSC ERROR: ./main on a arch-linux-c-opt named
dcs122 by CFSIfmyu Wed Aug 11 19:51:47 2021 <br>
[18]PETSC ERROR: Configure options
--download-scalapack --download-mumps --download-hypre
--with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90
--with-cudac=0 --with-debugging=0
--with-blaslapack-dir=/gpfs/u/home/CFSI/CFSIfmyu/barn-shared/dcs-rh8/lapack-build/<br>
[18]PETSC ERROR: <a
href="https://itssc.rpi.edu/hc/requests/1"
rel="ticket" target="_blank" moz-do-not-send="true">#1</a>
MatCreate_HYPRE() at
/gpfs/u/barn/CFSI/shared/dcs-rh8/petsc/src/mat/impls/hypre/mhypre.c:2120<br>
[18]PETSC ERROR: <a
href="https://itssc.rpi.edu/hc/requests/2"
rel="ticket" target="_blank" moz-do-not-send="true">#2</a>
MatSetType() at
/gpfs/u/barn/CFSI/shared/dcs-rh8/petsc/src/mat/interface/matreg.c:91<br>
[18]PETSC ERROR: <a
href="https://itssc.rpi.edu/hc/requests/3"
rel="ticket" target="_blank" moz-do-not-send="true">#3</a>
MatConvert_AIJ_HYPRE() at
/gpfs/u/barn/CFSI/shared/dcs-rh8/petsc/src/mat/impls/hypre/mhypre.c:392<br>
[18]PETSC ERROR: <a
href="https://itssc.rpi.edu/hc/requests/4"
rel="ticket" target="_blank" moz-do-not-send="true">#4</a>
MatConvert() at
/gpfs/u/barn/CFSI/shared/dcs-rh8/petsc/src/mat/interface/matrix.c:4439<br>
[18]PETSC ERROR: <a
href="https://itssc.rpi.edu/hc/requests/5"
rel="ticket" target="_blank" moz-do-not-send="true">#5</a>
PCSetUp_HYPRE() at
/gpfs/u/barn/CFSI/shared/dcs-rh8/petsc/src/ksp/pc/impls/hypre/hypre.c:240<br>
[18]PETSC ERROR: <a
href="https://itssc.rpi.edu/hc/requests/6"
rel="ticket" target="_blank" moz-do-not-send="true">#6</a>
PCSetUp() at
/gpfs/u/barn/CFSI/shared/dcs-rh8/petsc/src/ksp/pc/interface/precon.c:1015<br>
Configure options --download-scalapack
--download-mumps --download-hypre --with-cc=mpicc
--with-cxx=mpicxx --with-fc=mpif90 --with-cudac=0
--with-debugging=0
--with-blaslapack-dir=/gpfs/u/home/CFSI/CFSIfmyu/barn-shared/dcs-rh8/lapack-build/<br>
[19]PETSC ERROR: <a
href="https://itssc.rpi.edu/hc/requests/1"
rel="ticket" target="_blank" moz-do-not-send="true">#1</a>
MatCreate_HYPRE() at
/gpfs/u/barn/CFSI/shared/dcs-rh8/petsc/src/mat/impls/hypre/mhypre.c:2120<br>
[19]PETSC ERROR: <a
href="https://itssc.rpi.edu/hc/requests/2"
rel="ticket" target="_blank" moz-do-not-send="true">#2</a>
MatSetType() at
/gpfs/u/barn/CFSI/shared/dcs-rh8/petsc/src/mat/interface/matreg.c:91<br>
[19]PETSC ERROR: <a
href="https://itssc.rpi.edu/hc/requests/3"
rel="ticket" target="_blank" moz-do-not-send="true">#3</a>
MatConvert_AIJ_HYPRE() at
/gpfs/u/barn/CFSI/shared/dcs-rh8/petsc/src/mat/impls/hypre/mhypre.c:392<br>
[19]PETSC ERROR: <a
href="https://itssc.rpi.edu/hc/requests/4"
rel="ticket" target="_blank" moz-do-not-send="true">#4</a>
MatConvert() at
/gpfs/u/barn/CFSI/shared/dcs-rh8/petsc/src/mat/interface/matrix.c:4439<br>
[19]PETSC ERROR: <a
href="https://itssc.rpi.edu/hc/requests/5"
rel="ticket" target="_blank" moz-do-not-send="true">#5</a>
PCSetUp_HYPRE() at
/gpfs/u/barn/CFSI/shared/dcs-rh8/petsc/src/ksp/pc/impls/hypre/hypre.c:240<br>
[19]PETSC ERROR: <a
href="https://itssc.rpi.edu/hc/requests/6"
rel="ticket" target="_blank" moz-do-not-send="true">#6</a>
PCSetUp() at
/gpfs/u/barn/CFSI/shared/dcs-rh8/petsc/src/ksp/pc/interface/precon.c:1015</p>
<p>It seems that MPI_Comm_dup() at
petsc/src/mat/impls/hypre/mhypre.c:2120 caused the
problem. Since mine is a time-dependent problem,
MatCreate_HYPRE() is called every time the new system
matrix is assembled. The above error message is
reported after ~4095 calls of MatCreate_HYPRE(), which
is around 455 time steps in my code. Here is some
basic compiler information:</p>
</div>
</blockquote>
<div>Can you destroy old matrices to free MPI
communicators? Otherwise, you run into a limitation we
knew before.</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<p> </p>
</div>
</blockquote>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div>
<p>IBM Spectrum MPI 10.4.0</p>
<p>GCC 8.4.1</p>
<p>I've never had this problem before with OpenMPI or
MPICH implementation, so I was wondering if this can
be resolved from my end, or it's an implementation
specific problem.</p>
<p>Thanks!</p>
<p>Feimi<br>
</p>
</div>
</blockquote>
</div>
</div>
</blockquote>
</blockquote>
</body>
</html>