[petsc-users] how to use petsc4py with mpi subcommunicators?

[Gaetan Kenway]

Hi all

I think I remember having this or a similar issue at some point as well.
The issue was we had two (python wrapped) codes on the MPI_COMM_WORLD but
only one of them used PETSc. We never did figure out how to just get petsc
initialized on the one sub-comm. The workaround was just to do from petscpy
import PETSc at the very top of the highest level python execution script
which has worked for us.
Gaetan

On Tue, Apr 11, 2017 at 10:14 AM, Rodrigo Felicio <
Rodrigo.Felicio at iongeo.com> wrote:

> I think they are linked to the same MPI, but I am not sure how to confirm
> that. Looking into mpi4py/mpi.cfg I see the expected mpicc. On
> petsc4py/lib/petsc.cfg points to the petsc install directory and in there
> on the configure.log file I see the same path for the same intel mpi
> compiler...
>
> I noticed that if I instead of
> petsc4py.init(comm=newcomm)
>
> using
> petsc4py.init()
>
> the works, but then PETSc.COMM_WORLD = MPI.COMM_WORLD
>
> Further searching on the web suggest you are right and that the error msg
> that I am getting points to mismatch between mpirun and linked libraries,
> but why does it only happens if trying to initiate petsc with
> petsc4py.init(comm=newcomm)?
> Anyway, thanks Jed, really appreciate the help.
>
> Cheers
> Rodrigo
>
>
> -----Original Message-----
> From: Jed Brown [mailto:jed at jedbrown.org]
> Sent: Tuesday, April 11, 2017 11:36 AM
> To: Rodrigo Felicio; petsc-users at mcs.anl.gov
> Cc: Lisandro Dalcin
> Subject: Re: [petsc-users] how to use petsc4py with mpi subcommunicators?
>
> Looks like a question for Lisandro.  I believe the code you have (with
> appropriate collective semantics) was intended to work, but I'm not in a
> position to debug right now.  Have you confirmed that mpi4py is linked to
> the same MPI as petsc4py/PETSc?
>
> Rodrigo Felicio <Rodrigo.Felicio at iongeo.com> writes:
>
> > Hello all,
> >
> > Sorry for the newbie question, but is there a way of making petsc4py
> work with an MPI group or subcommunicator?  I  saw a solution posted back
> in 2010 (http://lists.mcs.anl.gov/pipermail/petsc-users/2010-
> May/006382.html), but it does not work for me. Indeed, if I try to use
> petsc4py.init(comm=newcom), then my sample code prints a msg "Attempting to
> use an MPI routine before initializing MPI".  Below I attach both the
> output and the source of the python code.
> >
> > kind regards
> > Rodrigo
> >
> >
> > time mpirun -n 5 python split_comm_ex2.py
> > Global: rank 0 of 5. New comm : rank  0 of  3
> > Global: rank 1 of 5. New comm : rank  0 of  2
> > Global: rank 2 of 5. New comm : rank  1 of  3
> > Global: rank 3 of 5. New comm : rank  1 of  2
> > Global: rank 4 of 5. New comm : rank  2 of  3 Attempting to use an MPI
> > routine before initializing MPI Attempting to use an MPI routine
> > before initializing MPI
> >
> > real    0m0.655s
> > user    0m1.122s
> > sys     0m1.047s
> >
> > And the python code:
> >
> > from mpi4py import MPI
> >
> > comm = MPI.COMM_WORLD
> > world_rank = comm.rank
> > world_size = comm.size
> >
> > color = world_rank % 2
> >
> > newcomm = comm.Split(color)
> > newcomm_rank = newcomm.rank
> > newcomm_size = newcomm.size
> >
> > for i in range(world_size):
> >     comm.Barrier()
> >     if (world_rank == i):
> >         print ("Global: rank %d of %d. New comm : rank % d of  %d" %
> >                (world_rank, world_size, newcomm_rank, newcomm_size))
> >
> > if newcomm.rank == 0:
> >     import petsc4py
> >     petsc4py.init(comm=newcomm)
> >     from petsc4py import PETSc
> >
> >     pcomm = PETSc.COMM_WORLD
> >     print('pcomm size is {}/{}'.format(pcomm.rank, pcomm.size))
> >
> > newcomm.Free()
> >
> > ________________________________
> >
> >
> > This email and any files transmitted with it are confidential and are
> intended solely for the use of the individual or entity to whom they are
> addressed. If you are not the original recipient or the person responsible
> for delivering the email to the intended recipient, be advised that you
> have received this email in error, and that any use, dissemination,
> forwarding, printing, or copying of this email is strictly prohibited. If
> you received this email in error, please immediately notify the sender and
> delete the original.
>
> ________________________________
>
>
> This email and any files transmitted with it are confidential and are
> intended solely for the use of the individual or entity to whom they are
> addressed. If you are not the original recipient or the person responsible
> for delivering the email to the intended recipient, be advised that you
> have received this email in error, and that any use, dissemination,
> forwarding, printing, or copying of this email is strictly prohibited. If
> you received this email in error, please immediately notify the sender and
> delete the original.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20170411/98413199/attachment.html>