<div dir="ltr">Hi Rodrigo<div><br></div><div>I just ran your example on Nasa's Pleiades system. Here's what I got:</div><div><br></div><div><div>PBS r459i4n11:~> time mpiexec -n 5 python3.5 another_split_ex.py</div><div>number of subcomms = 2.5</div><div>petsc rank=2, petsc size=5</div><div>sub rank 1/3, color:0</div><div>petsc rank=4, petsc size=5</div><div>sub rank 2/3, color:0</div><div>petsc rank=0, petsc size=5</div><div>sub rank 0/3, color:0</div><div>KSP Object: 2 MPI processes</div><div> type: cg</div><div> maximum iterations=10000, initial guess is zero</div><div> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</div><div> left preconditioning</div><div> using DEFAULT norm type for convergence test</div><div>PC Object: 2 MPI processes</div><div> type: none</div><div> PC has not been set up so information may be incomplete</div><div> linear system matrix = precond matrix:</div><div> Mat Object: 2 MPI processes</div><div> type: mpidense</div><div> rows=100, cols=100</div><div> total: nonzeros=10000, allocated nonzeros=10000</div><div> total number of mallocs used during MatSetValues calls =0</div><div>petsc rank=1, petsc size=5</div><div>sub rank 0/2, color:1</div><div>creating A in subcomm 1= 2, 0</div><div>petsc rank=3, petsc size=5</div><div>sub rank 1/2, color:1</div><div>creating A in subcomm 1= 2, 1</div><div><br></div><div>real 0m1.236s</div><div>user 0m0.088s</div><div>sys 0m0.008s</div></div><div><br></div><div>So everything looks like it went through fine. I know this doesn't help you directly, but we can confirm at least the python code itself is fine.</div><div><br></div><div>Gaetan</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Apr 12, 2017 at 10:10 AM, Rodrigo Felicio <span dir="ltr"><<a href="mailto:Rodrigo.Felicio@iongeo.com" target="_blank">Rodrigo.Felicio@iongeo.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Going over my older codes I found out that I have already tried the approach of splitting PETSc.COMM_WORLD, but whenever I try to create a matrix using a subcommuicator, the program fails. For example, executing the following python code attached to this msg, I get the following output<br>
<br>
time mpirun -n 5 python another_split_ex.py<br>
petsc rank=2, petsc size=5<br>
petsc rank=3, petsc size=5<br>
petsc rank=0, petsc size=5<br>
petsc rank=1, petsc size=5<br>
petsc rank=4, petsc size=5<br>
number of subcomms = 2<br>
sub rank 0/3, color:0<br>
sub rank 0/2, color:1<br>
sub rank 1/3, color:0<br>
sub rank 1/2, color:1<br>
sub rank 2/3, color:0<br>
creating A in subcomm 1= 2, 1<br>
creating A in subcomm 1= 2, 0<br>
Traceback (most recent call last):<br>
File "another_split_ex.py", line 43, in <module><br>
Traceback (most recent call last):<br>
File "another_split_ex.py", line 43, in <module><br>
A = PETSc.Mat().createDense([n,n], comm=subcomm)<br>
File "PETSc/Mat.pyx", line 390, in petsc4py.PETSc.Mat.createDense (src/petsc4py.PETSc.c:113792)<br>
A = PETSc.Mat().createDense([n,n], comm=subcomm)<br>
File "PETSc/Mat.pyx", line 390, in petsc4py.PETSc.Mat.createDense (src/petsc4py.PETSc.c:113792)<br>
File "PETSc/petscmat.pxi", line 602, in petsc4py.PETSc.Mat_Create (src/petsc4py.PETSc.c:25274)<br>
File "PETSc/petscmat.pxi", line 602, in petsc4py.PETSc.Mat_Create (src/petsc4py.PETSc.c:25274)<br>
File "PETSc/petscsys.pxi", line 104, in petsc4py.PETSc.Sys_Layout (src/petsc4py.PETSc.c:13666)<br>
File "PETSc/petscsys.pxi", line 104, in petsc4py.PETSc.Sys_Layout (src/petsc4py.PETSc.c:13666)<br>
petsc4py.PETSc.Error: petsc4py.PETSc.Errorerror code 608517<br>
[1] PetscSplitOwnership() line 86 in ~/mylocal/petsc/src/sys/utils/<wbr>psplit.c<br>
: error code 134826245<br>
[3] PetscSplitOwnership() line 86 in ~/mylocal/petsc/src/sys/utils/<wbr>psplit.c<br>
<br>
<br>
Checking the traceback, all I can say is that when the subcommunicator object reaches psplit.c code it gets somehow corrupted, because PetscSplitOwnership() fails to retrieve the size of the subcommunicator ... :-(<br>
<div class="HOEnZb"><div class="h5"><br>
regards<br>
Rodrigo<br>
<br>
______________________________<wbr>__<br>
<br>
<br>
This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the original recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender and delete the original.<br>
<br>
<br>
______________________________<wbr>__<br>
<br>
<br>
This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the original recipient or the person responsible for delivering the email to the intended recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender and delete the original.<br>
<br>
</div></div></blockquote></div><br></div>