[petsc-users] how to use petsc4py with mpi subcommunicators?

[Gaetan Kenway]

Hi Rodrigo

I just ran your example on Nasa's Pleiades system. Here's what I got:

PBS r459i4n11:~> time mpiexec -n 5 python3.5 another_split_ex.py
number of subcomms = 2.5
petsc rank=2, petsc size=5
sub rank 1/3, color:0
petsc rank=4, petsc size=5
sub rank 2/3, color:0
petsc rank=0, petsc size=5
sub rank 0/3, color:0
KSP Object: 2 MPI processes
  type: cg
  maximum iterations=10000, initial guess is zero
  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
  left preconditioning
  using DEFAULT norm type for convergence test
PC Object: 2 MPI processes
  type: none
  PC has not been set up so information may be incomplete
  linear system matrix = precond matrix:
  Mat Object:   2 MPI processes
    type: mpidense
    rows=100, cols=100
    total: nonzeros=10000, allocated nonzeros=10000
    total number of mallocs used during MatSetValues calls =0
petsc rank=1, petsc size=5
sub rank 0/2, color:1
creating A in subcomm 1= 2, 0
petsc rank=3, petsc size=5
sub rank 1/2, color:1
creating A in subcomm 1= 2, 1

real    0m1.236s
user    0m0.088s
sys     0m0.008s

So everything looks like it went through fine. I know this doesn't help you
directly, but we can confirm at least the python code itself is fine.

Gaetan

On Wed, Apr 12, 2017 at 10:10 AM, Rodrigo Felicio <
Rodrigo.Felicio at iongeo.com> wrote:

> Going over my older codes I found out that I have already tried the
> approach of splitting PETSc.COMM_WORLD, but whenever I try to create a
> matrix using a subcommuicator, the program fails. For example, executing
> the following python code attached to this msg,  I get the following output
>
> time mpirun -n 5 python another_split_ex.py
> petsc rank=2, petsc size=5
> petsc rank=3, petsc size=5
> petsc rank=0, petsc size=5
> petsc rank=1, petsc size=5
> petsc rank=4, petsc size=5
> number of subcomms = 2
> sub rank 0/3, color:0
> sub rank 0/2, color:1
> sub rank 1/3, color:0
> sub rank 1/2, color:1
> sub rank 2/3, color:0
> creating A in subcomm 1= 2, 1
> creating A in subcomm 1= 2, 0
> Traceback (most recent call last):
>   File "another_split_ex.py", line 43, in <module>
> Traceback (most recent call last):
>   File "another_split_ex.py", line 43, in <module>
>     A = PETSc.Mat().createDense([n,n], comm=subcomm)
>   File "PETSc/Mat.pyx", line 390, in petsc4py.PETSc.Mat.createDense
> (src/petsc4py.PETSc.c:113792)
>     A = PETSc.Mat().createDense([n,n], comm=subcomm)
>   File "PETSc/Mat.pyx", line 390, in petsc4py.PETSc.Mat.createDense
> (src/petsc4py.PETSc.c:113792)
>   File "PETSc/petscmat.pxi", line 602, in petsc4py.PETSc.Mat_Create
> (src/petsc4py.PETSc.c:25274)
>   File "PETSc/petscmat.pxi", line 602, in petsc4py.PETSc.Mat_Create
> (src/petsc4py.PETSc.c:25274)
>   File "PETSc/petscsys.pxi", line 104, in petsc4py.PETSc.Sys_Layout
> (src/petsc4py.PETSc.c:13666)
>   File "PETSc/petscsys.pxi", line 104, in petsc4py.PETSc.Sys_Layout
> (src/petsc4py.PETSc.c:13666)
> petsc4py.PETSc.Error: petsc4py.PETSc.Errorerror code 608517
> [1] PetscSplitOwnership() line 86 in ~/mylocal/petsc/src/sys/utils/
> psplit.c
> : error code 134826245
> [3] PetscSplitOwnership() line 86 in ~/mylocal/petsc/src/sys/utils/
> psplit.c
>
>
> Checking the traceback, all I can say is that when the subcommunicator
> object reaches psplit.c code it gets somehow corrupted, because
> PetscSplitOwnership() fails to retrieve the size of the subcommunicator ...
> :-(
>
> regards
> Rodrigo
>
> ________________________________
>
>
> This email and any files transmitted with it are confidential and are
> intended solely for the use of the individual or entity to whom they are
> addressed. If you are not the original recipient or the person responsible
> for delivering the email to the intended recipient, be advised that you
> have received this email in error, and that any use, dissemination,
> forwarding, printing, or copying of this email is strictly prohibited. If
> you received this email in error, please immediately notify the sender and
> delete the original.
>
>
> ________________________________
>
>
> This email and any files transmitted with it are confidential and are
> intended solely for the use of the individual or entity to whom they are
> addressed. If you are not the original recipient or the person responsible
> for delivering the email to the intended recipient, be advised that you
> have received this email in error, and that any use, dissemination,
> forwarding, printing, or copying of this email is strictly prohibited. If
> you received this email in error, please immediately notify the sender and
> delete the original.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20170412/ea5e1bc7/attachment.html>