<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.hoenzb
{mso-style-name:hoenzb;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks, Gaetan, for your suggestions and for running the code…Now I know for sure there is something wrong with my installation!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Cheers<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Rodrigo<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Gaetan Kenway [mailto:gaetank@gmail.com]
<br>
<b>Sent:</b> Wednesday, April 12, 2017 12:20 PM<br>
<b>To:</b> Rodrigo Felicio<br>
<b>Cc:</b> petsc-users@mcs.anl.gov<br>
<b>Subject:</b> Re: [petsc-users] how to use petsc4py with mpi subcommunicators?<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Maybe try doing<o:p></o:p></p>
<div>
<p class="MsoNormal">pComm=MPI.COMM_WORLD instead of the PETSc.COMM_WORLD. I know it shouldn't matter, but it's worth a shot. Also then you won't need the tompi4py() i guess.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Gaetan<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Wed, Apr 12, 2017 at 10:17 AM, Gaetan Kenway <<a href="mailto:gaetank@gmail.com" target="_blank">gaetank@gmail.com</a>> wrote:<o:p></o:p></p>
<div>
<p class="MsoNormal">Hi Rodrigo<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I just ran your example on Nasa's Pleiades system. Here's what I got:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">PBS r459i4n11:~> time mpiexec -n 5 python3.5 another_split_ex.py<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">number of subcomms = 2.5<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">petsc rank=2, petsc size=5<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">sub rank 1/3, color:0<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">petsc rank=4, petsc size=5<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">sub rank 2/3, color:0<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">petsc rank=0, petsc size=5<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">sub rank 0/3, color:0<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">KSP Object: 2 MPI processes<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> type: cg<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> maximum iterations=10000, initial guess is zero<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> left preconditioning<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> using DEFAULT norm type for convergence test<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">PC Object: 2 MPI processes<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> type: none<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> PC has not been set up so information may be incomplete<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> linear system matrix = precond matrix:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> Mat Object: 2 MPI processes<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> type: mpidense<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> rows=100, cols=100<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> total: nonzeros=10000, allocated nonzeros=10000<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> total number of mallocs used during MatSetValues calls =0<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">petsc rank=1, petsc size=5<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">sub rank 0/2, color:1<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">creating A in subcomm 1= 2, 0<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">petsc rank=3, petsc size=5<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">sub rank 1/2, color:1<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">creating A in subcomm 1= 2, 1<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">real 0m1.236s<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">user 0m0.088s<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">sys 0m0.008s<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">So everything looks like it went through fine. I know this doesn't help you directly, but we can confirm at least the python code itself is fine.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#888888"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:#888888">Gaetan<o:p></o:p></span></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Wed, Apr 12, 2017 at 10:10 AM, Rodrigo Felicio <<a href="mailto:Rodrigo.Felicio@iongeo.com" target="_blank">Rodrigo.Felicio@iongeo.com</a>> wrote:<o:p></o:p></p>
<p class="MsoNormal">Going over my older codes I found out that I have already tried the approach of splitting PETSc.COMM_WORLD, but whenever I try to create a matrix using a subcommuicator, the program fails. For example, executing the following python code
attached to this msg, I get the following output<br>
<br>
time mpirun -n 5 python another_split_ex.py<br>
petsc rank=2, petsc size=5<br>
petsc rank=3, petsc size=5<br>
petsc rank=0, petsc size=5<br>
petsc rank=1, petsc size=5<br>
petsc rank=4, petsc size=5<br>
number of subcomms = 2<br>
sub rank 0/3, color:0<br>
sub rank 0/2, color:1<br>
sub rank 1/3, color:0<br>
sub rank 1/2, color:1<br>
sub rank 2/3, color:0<br>
creating A in subcomm 1= 2, 1<br>
creating A in subcomm 1= 2, 0<br>
Traceback (most recent call last):<br>
File "another_split_ex.py", line 43, in <module><br>
Traceback (most recent call last):<br>
File "another_split_ex.py", line 43, in <module><br>
A = PETSc.Mat().createDense([n,n], comm=subcomm)<br>
File "PETSc/Mat.pyx", line 390, in petsc4py.PETSc.Mat.createDense (src/petsc4py.PETSc.c:113792)<br>
A = PETSc.Mat().createDense([n,n], comm=subcomm)<br>
File "PETSc/Mat.pyx", line 390, in petsc4py.PETSc.Mat.createDense (src/petsc4py.PETSc.c:113792)<br>
File "PETSc/petscmat.pxi", line 602, in petsc4py.PETSc.Mat_Create (src/petsc4py.PETSc.c:25274)<br>
File "PETSc/petscmat.pxi", line 602, in petsc4py.PETSc.Mat_Create (src/petsc4py.PETSc.c:25274)<br>
File "PETSc/petscsys.pxi", line 104, in petsc4py.PETSc.Sys_Layout (src/petsc4py.PETSc.c:13666)<br>
File "PETSc/petscsys.pxi", line 104, in petsc4py.PETSc.Sys_Layout (src/petsc4py.PETSc.c:13666)<br>
petsc4py.PETSc.Error: petsc4py.PETSc.Errorerror code 608517<br>
[1] PetscSplitOwnership() line 86 in ~/mylocal/petsc/src/sys/utils/psplit.c<br>
: error code 134826245<br>
[3] PetscSplitOwnership() line 86 in ~/mylocal/petsc/src/sys/utils/psplit.c<br>
<br>
<br>
Checking the traceback, all I can say is that when the subcommunicator object reaches psplit.c code it gets somehow corrupted, because PetscSplitOwnership() fails to retrieve the size of the subcommunicator ... :-(<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
regards<br>
Rodrigo<br>
<br>
________________________________<br>
<br>
<br>
This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the original recipient or the person responsible for delivering the email to the intended
recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender and delete the
original.<br>
<br>
<br>
________________________________<br>
<br>
<br>
This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the original recipient or the person responsible for delivering the email to the intended
recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender and delete the
original.<o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<br>
<hr>
<font face="Arial" color="Gray" size="1"><br>
<br>
This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. If you are not the original recipient or the person responsible for delivering the email to the intended
recipient, be advised that you have received this email in error, and that any use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately notify the sender and delete the
original.<br>
<br>
</font>
</body>
</html>