[petsc-users] how to use petsc4py with mpi subcommunicators?

[Rodrigo Felicio]

Thanks, Jed, but using color == 0 lead to the same error msg. Is there no way to set PETSc.COMM_WORLD to a subcomm instead of MPI.COMM_WORLD in python?

Cheers
Rodrigo

-----Original Message-----
From: Jed Brown [mailto:jed at jedbrown.org]
Sent: Monday, April 10, 2017 8:45 PM
To: Rodrigo Felicio; petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] how to use petsc4py with mpi subcommunicators?

Rodrigo Felicio <Rodrigo.Felicio at iongeo.com> writes:

> Hello all,
>
> Sorry for the newbie question, but is there a way of making petsc4py work with an MPI group or subcommunicator?  I  saw a solution posted back in 2010 (http://lists.mcs.anl.gov/pipermail/petsc-users/2010-May/006382.html), but it does not work for me. Indeed, if I try to use petsc4py.init(comm=newcom), then my sample code prints a msg "Attempting to use an MPI routine before initializing MPI".  Below I attach both the output and the source of the python code.
>
> kind regards
> Rodrigo
>
>
> time mpirun -n 5 python split_comm_ex2.py
> Global: rank 0 of 5. New comm : rank  0 of  3
> Global: rank 1 of 5. New comm : rank  0 of  2
> Global: rank 2 of 5. New comm : rank  1 of  3
> Global: rank 3 of 5. New comm : rank  1 of  2
> Global: rank 4 of 5. New comm : rank  2 of  3 Attempting to use an MPI
> routine before initializing MPI Attempting to use an MPI routine
> before initializing MPI
>
> real    0m0.655s
> user    0m1.122s
> sys     0m1.047s
>
> And the python code:
>
> from mpi4py import MPI
>
> comm = MPI.COMM_WORLD
> world_rank = comm.rank
> world_size = comm.size
>
> color = world_rank % 2
>
> newcomm = comm.Split(color)
> newcomm_rank = newcomm.rank
> newcomm_size = newcomm.size
>
> for i in range(world_size):
>     comm.Barrier()
>     if (world_rank == i):
>         print ("Global: rank %d of %d. New comm : rank % d of  %d" %
>                (world_rank, world_size, newcomm_rank, newcomm_size))
>
> if newcomm.rank == 0:
>     import petsc4py
>     petsc4py.init(comm=newcomm)

I don't know if it fixes your problem, but you definitely need to call this collectively on newcomm.  For example, using "if color == 0" above would have the right collective semantics.

>     from petsc4py import PETSc
>
>     pcomm = PETSc.COMM_WORLD
>     print('pcomm size is {}/{}'.format(pcomm.rank, pcomm.size))
>
> newcomm.Free()
>
> ________________________________
>
>
> This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the original recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender and delete the original.

________________________________


This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the original recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender and delete the original.