<html><head></head><body><div style="font-family: Verdana;font-size: 12.0px;"><div>
<p><span style="font-size:11pt"><span style="line-height:normal"><span style="font-family:Calibri,sans-serif"><span style="font-size:12.0pt"><span style="font-family:"Times New Roman",serif">thanks everyone for the very thorough answers. </span></span></span></span></span></p>
<p> </p>
<p>marius</p>
<div>
<div name="quote" style="margin:10px 5px 5px 10px; padding: 10px 0 10px 10px; border-left:2px solid #C3D9E5; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div style="margin:0 0 10px 0;"> </div>
<div name="quoted-content">
<div>MPI_Comm are opaque handles in C and integers in Fortran, which is required by MPI standard. The same applies to other types, like MPI_Op, MPI_Win etc. MPICH and OpenMPI have different implements for the handles. In MPICH handles are integer/bitfield, with some bits being offset to an array of objects. This makes it easy to do things like MPI_Comm_f2c(). In OpenMPI handles are pointers. OpenMPI has to transform pointers to integer offsets in MPI_Comm_c2f().
<div> </div>
<div>Running your tests with OpenMPI, you can see different pointers but same offsets</div>
<div>test_comms.c:</div>
<div><font face="monospace">0 4 -1419258464 909680992 909680992<br/>
1 4 1152255392 -2144517440 -2144517440<br/>
2 4 -306719328 768197312 768197312<br/>
3 4 -1766709856 715374384 715374384</font></div>
<div> </div>
<div>test_comms.f90:</div>
<div> 0 0 3</div>
<div> 1 0 3<br/>
2 0 3<br/>
3 0 3<br/>
</div>
<div>Running with MPICH, you can see C/Fortran MPI_Comm's are the same. But why ranks do not have the same integer/bitfield, I don't know. You need to dig into mpich code.</div>
<div>test_comms.c:</div>
<div>0 4 1140850688 -2080374780 -2080374780<br/>
1 4 1140850688 -2080374780 -2080374780<br/>
2 4 1140850688 -2080374782 -2080374782<br/>
3 4 1140850688 -2080374782 -2080374782</div>
<div> </div>
test_comms.f90:
<div> 0 1140850688 -2080374780<br/>
1 1140850688 -2080374780<br/>
2 1140850688 -2080374782<br/>
3 1140850688 -2080374782</div>
<div> </div>
<div>In summary, users should not expect MPI_Comm variables are equal across ranks, and MPI_Send an MPI_Comm variable to remote ranks.</div>
<div>
<div>
<div>
<div>--Junchao Zhang</div>
</div>
</div>
</div>
</div>
<div class="gmail_quote">
<div class="gmail_attr">On Wed, Apr 22, 2020 at 8:56 AM Patrick Sanan <<a href="mailto:patrick.sanan@gmail.com" onclick="parent.window.location.href='mailto:patrick.sanan@gmail.com'; return false;" target="_blank">patrick.sanan@gmail.com</a>> wrote:</div>
<blockquote class="gmail_quote" style="margin: 0.0px 0.0px 0.0px 0.8ex;border-left: 1.0px solid rgb(204,204,204);padding-left: 1.0ex;">
<div>Perhaps the confusion here is related to the fact that an MPI_Comm is not an integer identifying the communicator. Rather,
<div>it's a pointer to a data structure which contains information about the communicator (I'm not positive but probably something like <a href="https://github.com/pmodels/mpich/blob/master/src/include/mpir_comm.h#L150" target="_blank">this</a>). </div>
<div> </div>
<div>You're converting that pointer to an int and printing it out. The value happens to be the same on all ranks except 0, but this</div>
<div>doesn't directly tell you anything about equality of the MPI_comm objects that those pointers point to.</div>
</div>
<div class="gmail_quote">
<div class="gmail_attr">Am Mi., 22. Apr. 2020 um 15:28 Uhr schrieb Matthew Knepley <<a href="mailto:knepley@gmail.com" onclick="parent.window.location.href='mailto:knepley@gmail.com'; return false;" target="_blank">knepley@gmail.com</a>>:</div>
<blockquote class="gmail_quote" style="margin: 0.0px 0.0px 0.0px 0.8ex;border-left: 1.0px solid rgb(204,204,204);padding-left: 1.0ex;">
<div>
<div>On Wed, Apr 22, 2020 at 3:07 AM Marius Buerkle <<a href="mailto:mbuerkle@web.de" onclick="parent.window.location.href='mailto:mbuerkle@web.de'; return false;" target="_blank">mbuerkle@web.de</a>> wrote:</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin: 0.0px 0.0px 0.0px 0.8ex;border-left: 1.0px solid rgb(204,204,204);padding-left: 1.0ex;">
<div>
<div style="font-family: Verdana;font-size: 12.0px;">
<div>I see, but I am still puzzeled, why are the communicators different on different notes eventhough it is the same object.</div>
</div>
</div>
</blockquote>
<div> </div>
<div>This is the output of MPI_Comm_dup() on line 126 of tagm.c. Therefore, dup comms are not guaranteed to have the same id</div>
<div>across multiple processes.</div>
<div> </div>
<div> Thanks,</div>
<div> </div>
<div> Matt</div>
<div> </div>
<blockquote class="gmail_quote" style="margin: 0.0px 0.0px 0.0px 0.8ex;border-left: 1.0px solid rgb(204,204,204);padding-left: 1.0ex;">
<div>
<div style="font-family: Verdana;font-size: 12.0px;">
<div> </div>
<div>
<div>
<div style="margin: 10.0px 5.0px 5.0px 10.0px;padding: 10.0px 0.0px 10.0px 10.0px;border-left: 2.0px solid rgb(195,217,229);">
<div style="margin: 0.0px 0.0px 10.0px;"> </div>
<div>PETSc creates a duplicate of the communicator during object creation.<br/>
<a href="https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscCommDuplicate.html" target="_blank">https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscCommDuplicate.html</a><br/>
<br/>
Jose<br/>
<br/>
<br/>
> El 22 abr 2020, a las 8:40, Marius Buerkle <<a href="mailto:mbuerkle@web.de" onclick="parent.window.location.href='mailto:mbuerkle@web.de'; return false;" target="_blank">mbuerkle@web.de</a>> escribió:<br/>
><br/>
> Hi Dave,<br/>
><br/>
> I want to use it in Fortran if possible. But I tried both C and Fortran just to see if it works in general. I am using MPICH 3.3.2. I attached the MWE for C and Fortran with the output I get.<br/>
><br/>
> Marius<br/>
><br/>
><br/>
><br/>
><br/>
><br/>
> Hi,<br/>
><br/>
> What is PetscObjectGetComm expected to return?<br/>
><br/>
> As Patrick said, it returns the communicator associated with the petsc object.<br/>
><br/>
> I thought it would give the MPI communicator the object lives on. So if I create A matrix on PETSC_COMM_WORLD a call of PetscObjectGetComm for A it would return PETSC_COMM_WORLD? But it seems to return something else, and while most of the nodes return a similar communicator some are giving a different one.<br/>
><br/>
> How are you actually comparing the communicators (send code snippet)? Which MPI implementation are you using? And when are comparing comms is the comparison code written in C it FORTRAN?<br/>
><br/>
><br/>
> That said, is there a way to get the MPI communicator a matrix lives on?<br/>
><br/>
> You are using the correct function. There is a macro as well but it’s best to use the function.<br/>
><br/>
> Thanks,<br/>
> Dave<br/>
><br/>
><br/>
><br/>
><br/>
> Best,<br/>
> Marius<br/>
> <test_comm.tar.gz><br/>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<div> </div>
--
<div>
<div>
<div>
<div>
<div>
<div>
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br/>
-- Norbert Wiener</div>
<div> </div>
<div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div></div></body></html>