[petsc-users] how to use petsc4py with mpi subcommunicators?

[Rodrigo Felicio]

Thanks, Gaetan, for your suggestions and for running the code…Now I know for sure there is something wrong with my installation!
Cheers
Rodrigo

From: Gaetan Kenway [mailto:gaetank at gmail.com]
Sent: Wednesday, April 12, 2017 12:20 PM
To: Rodrigo Felicio
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] how to use petsc4py with mpi subcommunicators?

Maybe try doing
pComm=MPI.COMM_WORLD instead of the PETSc.COMM_WORLD. I know it shouldn't matter, but it's worth a shot. Also then you won't need the tompi4py() i guess.

Gaetan

On Wed, Apr 12, 2017 at 10:17 AM, Gaetan Kenway <gaetank at gmail.com<mailto:gaetank at gmail.com>> wrote:
Hi Rodrigo

I just ran your example on Nasa's Pleiades system. Here's what I got:

PBS r459i4n11:~> time mpiexec -n 5 python3.5 another_split_ex.py
number of subcomms = 2.5
petsc rank=2, petsc size=5
sub rank 1/3, color:0
petsc rank=4, petsc size=5
sub rank 2/3, color:0
petsc rank=0, petsc size=5
sub rank 0/3, color:0
KSP Object: 2 MPI processes
  type: cg
  maximum iterations=10000, initial guess is zero
  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
  left preconditioning
  using DEFAULT norm type for convergence test
PC Object: 2 MPI processes
  type: none
  PC has not been set up so information may be incomplete
  linear system matrix = precond matrix:
  Mat Object:   2 MPI processes
    type: mpidense
    rows=100, cols=100
    total: nonzeros=10000, allocated nonzeros=10000
    total number of mallocs used during MatSetValues calls =0
petsc rank=1, petsc size=5
sub rank 0/2, color:1
creating A in subcomm 1= 2, 0
petsc rank=3, petsc size=5
sub rank 1/2, color:1
creating A in subcomm 1= 2, 1

real    0m1.236s
user    0m0.088s
sys     0m0.008s

So everything looks like it went through fine. I know this doesn't help you directly, but we can confirm at least the python code itself is fine.

Gaetan

On Wed, Apr 12, 2017 at 10:10 AM, Rodrigo Felicio <Rodrigo.Felicio at iongeo.com<mailto:Rodrigo.Felicio at iongeo.com>> wrote:
Going over my older codes I found out that I have already tried the approach of splitting PETSc.COMM_WORLD, but whenever I try to create a matrix using a subcommuicator, the program fails. For example, executing the following python code attached to this msg,  I get the following output

time mpirun -n 5 python another_split_ex.py
petsc rank=2, petsc size=5
petsc rank=3, petsc size=5
petsc rank=0, petsc size=5
petsc rank=1, petsc size=5
petsc rank=4, petsc size=5
number of subcomms = 2
sub rank 0/3, color:0
sub rank 0/2, color:1
sub rank 1/3, color:0
sub rank 1/2, color:1
sub rank 2/3, color:0
creating A in subcomm 1= 2, 1
creating A in subcomm 1= 2, 0
Traceback (most recent call last):
  File "another_split_ex.py", line 43, in <module>
Traceback (most recent call last):
  File "another_split_ex.py", line 43, in <module>
    A = PETSc.Mat().createDense([n,n], comm=subcomm)
  File "PETSc/Mat.pyx", line 390, in petsc4py.PETSc.Mat.createDense (src/petsc4py.PETSc.c:113792)
    A = PETSc.Mat().createDense([n,n], comm=subcomm)
  File "PETSc/Mat.pyx", line 390, in petsc4py.PETSc.Mat.createDense (src/petsc4py.PETSc.c:113792)
  File "PETSc/petscmat.pxi", line 602, in petsc4py.PETSc.Mat_Create (src/petsc4py.PETSc.c:25274)
  File "PETSc/petscmat.pxi", line 602, in petsc4py.PETSc.Mat_Create (src/petsc4py.PETSc.c:25274)
  File "PETSc/petscsys.pxi", line 104, in petsc4py.PETSc.Sys_Layout (src/petsc4py.PETSc.c:13666)
  File "PETSc/petscsys.pxi", line 104, in petsc4py.PETSc.Sys_Layout (src/petsc4py.PETSc.c:13666)
petsc4py.PETSc.Error: petsc4py.PETSc.Errorerror code 608517
[1] PetscSplitOwnership() line 86 in ~/mylocal/petsc/src/sys/utils/psplit.c
: error code 134826245
[3] PetscSplitOwnership() line 86 in ~/mylocal/petsc/src/sys/utils/psplit.c


Checking the traceback, all I can say is that when the subcommunicator object reaches psplit.c code it gets somehow corrupted, because PetscSplitOwnership() fails to retrieve the size of the subcommunicator ... :-(

regards
Rodrigo

________________________________


This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the original recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender and delete the original.


________________________________


This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the original recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender and delete the original.



________________________________


This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the original recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender and delete the original.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20170412/08f18a66/attachment.html>