<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hi,</p>
<p>I was trying to run a simulation with a PETSc-wrapped Hypre
preconditioner, and encountered this problem:</p>
<p>[dcs122:133012] Out of resources: all 4095 communicator IDs have
been used.<br>
[19]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------<br>
[19]PETSC ERROR: General MPI error <br>
[19]PETSC ERROR: MPI error 17 MPI_ERR_INTERN: internal error<br>
[19]PETSC ERROR: See <a
href="https://www.mcs.anl.gov/petsc/documentation/faq.html"
rel="noreferrer">https://www.mcs.anl.gov/petsc/documentation/faq.html</a>
for trouble shooting.<br>
[19]PETSC ERROR: Petsc Release Version 3.15.2, unknown <br>
[19]PETSC ERROR: ./main on a arch-linux-c-opt named dcs122 by
CFSIfmyu Wed Aug 11 19:51:47 2021 <br>
[19]PETSC ERROR: [dcs122:133010] Out of resources: all 4095
communicator IDs have been used.<br>
[18]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------<br>
[18]PETSC ERROR: General MPI error <br>
[18]PETSC ERROR: MPI error 17 MPI_ERR_INTERN: internal error<br>
[18]PETSC ERROR: See <a
href="https://www.mcs.anl.gov/petsc/documentation/faq.html"
rel="noreferrer">https://www.mcs.anl.gov/petsc/documentation/faq.html</a>
for trouble shooting.<br>
[18]PETSC ERROR: Petsc Release Version 3.15.2, unknown <br>
[18]PETSC ERROR: ./main on a arch-linux-c-opt named dcs122 by
CFSIfmyu Wed Aug 11 19:51:47 2021 <br>
[18]PETSC ERROR: Configure options --download-scalapack
--download-mumps --download-hypre --with-cc=mpicc
--with-cxx=mpicxx --with-fc=mpif90 --with-cudac=0
--with-debugging=0
--with-blaslapack-dir=/gpfs/u/home/CFSI/CFSIfmyu/barn-shared/dcs-rh8/lapack-build/<br>
[18]PETSC ERROR: <a href="https://itssc.rpi.edu/hc/requests/1"
target="_blank" rel="ticket">#1</a> MatCreate_HYPRE() at
/gpfs/u/barn/CFSI/shared/dcs-rh8/petsc/src/mat/impls/hypre/mhypre.c:2120<br>
[18]PETSC ERROR: <a href="https://itssc.rpi.edu/hc/requests/2"
target="_blank" rel="ticket">#2</a> MatSetType() at
/gpfs/u/barn/CFSI/shared/dcs-rh8/petsc/src/mat/interface/matreg.c:91<br>
[18]PETSC ERROR: <a href="https://itssc.rpi.edu/hc/requests/3"
target="_blank" rel="ticket">#3</a> MatConvert_AIJ_HYPRE() at
/gpfs/u/barn/CFSI/shared/dcs-rh8/petsc/src/mat/impls/hypre/mhypre.c:392<br>
[18]PETSC ERROR: <a href="https://itssc.rpi.edu/hc/requests/4"
target="_blank" rel="ticket">#4</a> MatConvert() at
/gpfs/u/barn/CFSI/shared/dcs-rh8/petsc/src/mat/interface/matrix.c:4439<br>
[18]PETSC ERROR: <a href="https://itssc.rpi.edu/hc/requests/5"
target="_blank" rel="ticket">#5</a> PCSetUp_HYPRE() at
/gpfs/u/barn/CFSI/shared/dcs-rh8/petsc/src/ksp/pc/impls/hypre/hypre.c:240<br>
[18]PETSC ERROR: <a href="https://itssc.rpi.edu/hc/requests/6"
target="_blank" rel="ticket">#6</a> PCSetUp() at
/gpfs/u/barn/CFSI/shared/dcs-rh8/petsc/src/ksp/pc/interface/precon.c:1015<br>
Configure options --download-scalapack --download-mumps
--download-hypre --with-cc=mpicc --with-cxx=mpicxx
--with-fc=mpif90 --with-cudac=0 --with-debugging=0
--with-blaslapack-dir=/gpfs/u/home/CFSI/CFSIfmyu/barn-shared/dcs-rh8/lapack-build/<br>
[19]PETSC ERROR: <a href="https://itssc.rpi.edu/hc/requests/1"
target="_blank" rel="ticket">#1</a> MatCreate_HYPRE() at
/gpfs/u/barn/CFSI/shared/dcs-rh8/petsc/src/mat/impls/hypre/mhypre.c:2120<br>
[19]PETSC ERROR: <a href="https://itssc.rpi.edu/hc/requests/2"
target="_blank" rel="ticket">#2</a> MatSetType() at
/gpfs/u/barn/CFSI/shared/dcs-rh8/petsc/src/mat/interface/matreg.c:91<br>
[19]PETSC ERROR: <a href="https://itssc.rpi.edu/hc/requests/3"
target="_blank" rel="ticket">#3</a> MatConvert_AIJ_HYPRE() at
/gpfs/u/barn/CFSI/shared/dcs-rh8/petsc/src/mat/impls/hypre/mhypre.c:392<br>
[19]PETSC ERROR: <a href="https://itssc.rpi.edu/hc/requests/4"
target="_blank" rel="ticket">#4</a> MatConvert() at
/gpfs/u/barn/CFSI/shared/dcs-rh8/petsc/src/mat/interface/matrix.c:4439<br>
[19]PETSC ERROR: <a href="https://itssc.rpi.edu/hc/requests/5"
target="_blank" rel="ticket">#5</a> PCSetUp_HYPRE() at
/gpfs/u/barn/CFSI/shared/dcs-rh8/petsc/src/ksp/pc/impls/hypre/hypre.c:240<br>
[19]PETSC ERROR: <a href="https://itssc.rpi.edu/hc/requests/6"
target="_blank" rel="ticket">#6</a> PCSetUp() at
/gpfs/u/barn/CFSI/shared/dcs-rh8/petsc/src/ksp/pc/interface/precon.c:1015</p>
<p>It seems that MPI_Comm_dup() at
petsc/src/mat/impls/hypre/mhypre.c:2120 caused the problem. Since
mine is a time-dependent problem, MatCreate_HYPRE() is called
every time the new system matrix is assembled. The above error
message is reported after ~4095 calls of MatCreate_HYPRE(), which
is around 455 time steps in my code. Here is some basic compiler
information:</p>
<p>IBM Spectrum MPI 10.4.0</p>
<p>GCC 8.4.1</p>
<p>I've never had this problem before with OpenMPI or MPICH
implementation, so I was wondering if this can be resolved from my
end, or it's an implementation specific problem.</p>
<p>Thanks!</p>
<p>Feimi<br>
</p>
</body>
</html>