[petsc-users] how to use petsc4py with mpi subcommunicators?

[Rodrigo Felicio]

I think they are linked to the same MPI, but I am not sure how to confirm that. Looking into mpi4py/mpi.cfg I see the expected mpicc. On petsc4py/lib/petsc.cfg points to the petsc install directory and in there on the configure.log file I see the same path for the same intel mpi compiler...

I noticed that if I instead of
petsc4py.init(comm=newcomm)

using
petsc4py.init()

the works, but then PETSc.COMM_WORLD = MPI.COMM_WORLD

Further searching on the web suggest you are right and that the error msg that I am getting points to mismatch between mpirun and linked libraries, but why does it only happens if trying to initiate petsc with petsc4py.init(comm=newcomm)?
Anyway, thanks Jed, really appreciate the help.

Cheers
Rodrigo


-----Original Message-----
From: Jed Brown [mailto:jed at jedbrown.org]
Sent: Tuesday, April 11, 2017 11:36 AM
To: Rodrigo Felicio; petsc-users at mcs.anl.gov
Cc: Lisandro Dalcin
Subject: Re: [petsc-users] how to use petsc4py with mpi subcommunicators?

Looks like a question for Lisandro.  I believe the code you have (with appropriate collective semantics) was intended to work, but I'm not in a position to debug right now.  Have you confirmed that mpi4py is linked to the same MPI as petsc4py/PETSc?

Rodrigo Felicio <Rodrigo.Felicio at iongeo.com> writes:

> Hello all,
>
> Sorry for the newbie question, but is there a way of making petsc4py work with an MPI group or subcommunicator?  I  saw a solution posted back in 2010 (http://lists.mcs.anl.gov/pipermail/petsc-users/2010-May/006382.html), but it does not work for me. Indeed, if I try to use petsc4py.init(comm=newcom), then my sample code prints a msg "Attempting to use an MPI routine before initializing MPI".  Below I attach both the output and the source of the python code.
>
> kind regards
> Rodrigo
>
>
> time mpirun -n 5 python split_comm_ex2.py
> Global: rank 0 of 5. New comm : rank  0 of  3
> Global: rank 1 of 5. New comm : rank  0 of  2
> Global: rank 2 of 5. New comm : rank  1 of  3
> Global: rank 3 of 5. New comm : rank  1 of  2
> Global: rank 4 of 5. New comm : rank  2 of  3 Attempting to use an MPI
> routine before initializing MPI Attempting to use an MPI routine
> before initializing MPI
>
> real    0m0.655s
> user    0m1.122s
> sys     0m1.047s
>
> And the python code:
>
> from mpi4py import MPI
>
> comm = MPI.COMM_WORLD
> world_rank = comm.rank
> world_size = comm.size
>
> color = world_rank % 2
>
> newcomm = comm.Split(color)
> newcomm_rank = newcomm.rank
> newcomm_size = newcomm.size
>
> for i in range(world_size):
>     comm.Barrier()
>     if (world_rank == i):
>         print ("Global: rank %d of %d. New comm : rank % d of  %d" %
>                (world_rank, world_size, newcomm_rank, newcomm_size))
>
> if newcomm.rank == 0:
>     import petsc4py
>     petsc4py.init(comm=newcomm)
>     from petsc4py import PETSc
>
>     pcomm = PETSc.COMM_WORLD
>     print('pcomm size is {}/{}'.format(pcomm.rank, pcomm.size))
>
> newcomm.Free()
>
> ________________________________
>
>
> This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the original recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender and delete the original.

________________________________


This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the original recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender and delete the original.