[petsc-users] how to use petsc4py with mpi subcommunicators?

[Gaetan Kenway]

Maybe try doing
pComm=MPI.COMM_WORLD instead of the PETSc.COMM_WORLD. I know it shouldn't
matter, but it's worth a shot. Also then you won't need the tompi4py() i
guess.

Gaetan

On Wed, Apr 12, 2017 at 10:17 AM, Gaetan Kenway <gaetank at gmail.com> wrote:

> Hi Rodrigo
>
> I just ran your example on Nasa's Pleiades system. Here's what I got:
>
> PBS r459i4n11:~> time mpiexec -n 5 python3.5 another_split_ex.py
> number of subcomms = 2.5
> petsc rank=2, petsc size=5
> sub rank 1/3, color:0
> petsc rank=4, petsc size=5
> sub rank 2/3, color:0
> petsc rank=0, petsc size=5
> sub rank 0/3, color:0
> KSP Object: 2 MPI processes
>   type: cg
>   maximum iterations=10000, initial guess is zero
>   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>   left preconditioning
>   using DEFAULT norm type for convergence test
> PC Object: 2 MPI processes
>   type: none
>   PC has not been set up so information may be incomplete
>   linear system matrix = precond matrix:
>   Mat Object:   2 MPI processes
>     type: mpidense
>     rows=100, cols=100
>     total: nonzeros=10000, allocated nonzeros=10000
>     total number of mallocs used during MatSetValues calls =0
> petsc rank=1, petsc size=5
> sub rank 0/2, color:1
> creating A in subcomm 1= 2, 0
> petsc rank=3, petsc size=5
> sub rank 1/2, color:1
> creating A in subcomm 1= 2, 1
>
> real    0m1.236s
> user    0m0.088s
> sys     0m0.008s
>
> So everything looks like it went through fine. I know this doesn't help
> you directly, but we can confirm at least the python code itself is fine.
>
> Gaetan
>
> On Wed, Apr 12, 2017 at 10:10 AM, Rodrigo Felicio <
> Rodrigo.Felicio at iongeo.com> wrote:
>
>> Going over my older codes I found out that I have already tried the
>> approach of splitting PETSc.COMM_WORLD, but whenever I try to create a
>> matrix using a subcommuicator, the program fails. For example, executing
>> the following python code attached to this msg,  I get the following output
>>
>> time mpirun -n 5 python another_split_ex.py
>> petsc rank=2, petsc size=5
>> petsc rank=3, petsc size=5
>> petsc rank=0, petsc size=5
>> petsc rank=1, petsc size=5
>> petsc rank=4, petsc size=5
>> number of subcomms = 2
>> sub rank 0/3, color:0
>> sub rank 0/2, color:1
>> sub rank 1/3, color:0
>> sub rank 1/2, color:1
>> sub rank 2/3, color:0
>> creating A in subcomm 1= 2, 1
>> creating A in subcomm 1= 2, 0
>> Traceback (most recent call last):
>>   File "another_split_ex.py", line 43, in <module>
>> Traceback (most recent call last):
>>   File "another_split_ex.py", line 43, in <module>
>>     A = PETSc.Mat().createDense([n,n], comm=subcomm)
>>   File "PETSc/Mat.pyx", line 390, in petsc4py.PETSc.Mat.createDense
>> (src/petsc4py.PETSc.c:113792)
>>     A = PETSc.Mat().createDense([n,n], comm=subcomm)
>>   File "PETSc/Mat.pyx", line 390, in petsc4py.PETSc.Mat.createDense
>> (src/petsc4py.PETSc.c:113792)
>>   File "PETSc/petscmat.pxi", line 602, in petsc4py.PETSc.Mat_Create
>> (src/petsc4py.PETSc.c:25274)
>>   File "PETSc/petscmat.pxi", line 602, in petsc4py.PETSc.Mat_Create
>> (src/petsc4py.PETSc.c:25274)
>>   File "PETSc/petscsys.pxi", line 104, in petsc4py.PETSc.Sys_Layout
>> (src/petsc4py.PETSc.c:13666)
>>   File "PETSc/petscsys.pxi", line 104, in petsc4py.PETSc.Sys_Layout
>> (src/petsc4py.PETSc.c:13666)
>> petsc4py.PETSc.Error: petsc4py.PETSc.Errorerror code 608517
>> [1] PetscSplitOwnership() line 86 in ~/mylocal/petsc/src/sys/utils/
>> psplit.c
>> : error code 134826245
>> [3] PetscSplitOwnership() line 86 in ~/mylocal/petsc/src/sys/utils/
>> psplit.c
>>
>>
>> Checking the traceback, all I can say is that when the subcommunicator
>> object reaches psplit.c code it gets somehow corrupted, because
>> PetscSplitOwnership() fails to retrieve the size of the subcommunicator ...
>> :-(
>>
>> regards
>> Rodrigo
>>
>> ________________________________
>>
>>
>> This email and any files transmitted with it are confidential and are
>> intended solely for the use of the individual or entity to whom they are
>> addressed. If you are not the original recipient or the person responsible
>> for delivering the email to the intended recipient, be advised that you
>> have received this email in error, and that any use, dissemination,
>> forwarding, printing, or copying of this email is strictly prohibited. If
>> you received this email in error, please immediately notify the sender and
>> delete the original.
>>
>>
>> ________________________________
>>
>>
>> This email and any files transmitted with it are confidential and are
>> intended solely for the use of the individual or entity to whom they are
>> addressed. If you are not the original recipient or the person responsible
>> for delivering the email to the intended recipient, be advised that you
>> have received this email in error, and that any use, dissemination,
>> forwarding, printing, or copying of this email is strictly prohibited. If
>> you received this email in error, please immediately notify the sender and
>> delete the original.
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20170412/4298a2dd/attachment-0001.html>